Sample reliability reportSimulation
A full batch delivery, worked end to end.
Synthetic vignettes, scripted outputs, simulated raters — shows the shape of a delivery, not clinical evidence.
Download the sample report (PDF)Methodology
How the reliability score is computed.
Not an assertion — a published formula, versioned per protocol, with a sample delivery you can inspect before you ever get on a call.
01 — The score, defined
One composite. Three measured components.
reliability_score = 100 × ( 0.40 × gold_pass_rate
+ 0.35 × disposition_agreement
+ 0.25 × (1 − critical_miss_rate) )- gold_pass_rate
- Share of blinded gold-standard (the protocol term for reference cases) checks the panel passed. Gold items are authored, independently confirmed by ≥2 physicians, and injected into the queue indistinguishable from live cases.
- disposition_agreement
- Pairwise inter-rater agreement on the primary judgment (reported as % agreement, with Cohen's κ alongside for chance correction), measured before any adjudication.
- critical_miss_rate
- Rate of dangerous omissions/commissions on gold items where the safe answer is known.
Weights are protocol-versioned per task family — this composite is uc1-rubric-v0 under protocol v0.1. Every delivery states the protocol version and ships component metrics, not just the composite.
02 — See a full delivery
Download the artifacts. Read the numbers yourself.
Rubric excerpt
The grading axes behind the score.
Disposition, red-flag coverage, and dangerous omission/commission — the scales raters actually use.
Download the rubric excerpt (PDF)03 — Delivery formats
What ships with every batch.
JSONL rating records
Validating against published JSON Schemas — vignette and rating.
Batch manifest
Gold IDs and provenance, so every score card traces back to its source cases.
Component metrics JSON
gold_pass_rate, disposition_agreement, and critical_miss_rate shipped alongside the composite — not just the single number.
Schemas: uc1-vignette.schema.json and uc1-rating.schema.json. Publishing the schema files themselves is a planned follow-up.
Walk the methodology on your task.
Bring your task family and we'll show you exactly how the score would be computed — protocol, gold set, and weights.