Methodology · public and versioned
A score you can defend.
Every number in a Bench report traces to this page: the rubric that produced it, the calibration that validated the raters, and the formulas that computed it. If a measurement can't be defended, it shouldn't be sold.
The rubric
Five dimensions, 0–5 each, 25 points total.
Each evaluated output is scored on 5 dimensions with written anchors. Evaluators also assign a severity (critical / high / medium / low / none), classify failures in a fixed error taxonomy, and write a gold-standard correction for every failed item.
| Dimension | Score 5 anchor | Score 0 anchor |
|---|---|---|
| Jurisdictional accuracy | Correct legal framework for the target jurisdiction, including free-zone / federal splits | Answers under another jurisdiction’s law, or transplants foreign doctrine wholesale |
| Legal correctness | The rule is stated as the authority states it, with material conditions intact | The core rule is misstated in a way that would mislead a competent reader |
| Completeness | All legally required elements, exceptions, and thresholds a practitioner would flag | A required element is missing and its absence changes the answer |
| Reasoning quality | Sound application of the right authorities to the facts, in the right order | Right sources, unsound application — or conclusion asserted without analysis |
| Hallucination risk | Every cited authority exists, is current, and supports the proposition | Fabricated statute, case, or guidance presented as real |
Derived metrics
Deterministic by construction.
Report metrics are pure functions of the evaluation rows — no model in the loop, no discretion at report time:
| Accuracy score | mean(total / 25) × 100 |
| Hallucination rate | % items tagged hallucinated-authority, or scoring ≤ 2 on the hallucination dimension |
| Critical-error rate | % items with severity = critical |
| Citation reliability | 100 − % items tagged unreliable-citation or hallucinated-authority |
| Missing-law rate | % items tagged omission |
Error taxonomy
Six failure classes.
| Hallucinated authority | Fabricated statute, case, or guidance |
| Misstatement | Real law, wrongly stated |
| Omission | Legally required element missing |
| Wrong jurisdiction | Answered under another regime's law |
| Bad reasoning | Right sources, unsound application |
| Unreliable citation | Real authority, wrong proposition |
Calibration
Raters are measured before they measure.
Before an expert's scores count, they complete a paid assessment and a calibration round in which a sample of items is independently double-scored by two experts. We compute exact and adjacent (±1) agreement per dimension in code. Items with any dimension differing by more than one point go to a disagreement-resolution step: a third senior reviewer resolves, and the rationale is recorded with the item.
Ongoing QA follows a published sampling curve: 100% of a new expert's first tasks are reviewed, dropping to 20% after sustained quality, then to exception-based review. Admission and retention decisions are made by a human, always.
Inter-rater agreementfirst cycle pending
Published here after the first calibration cycle completes. We do not invent this number; an accuracy score from uncalibrated raters is noise.
Versioning & integrity
Immutable versions. Held-out sets.
Benchmark versions are immutable once released; every report names the exact version measured against. The majority of each benchmark is held out and never published, and released sample items are excluded from paid measurement — a system that memorised our public pages gains nothing.
Named results are private to each client. Published research is aggregated or anonymized, and reports state the measured test set, date, and version — a Bench score is a point-in-time measurement of a defined set, not a warranty of production safety.
Position in the landscape
Where Bench differs.
| Alternative | Gap Bench covers |
|---|---|
| Open academic benchmarks (e.g. LegalBench) | US-centric and static; public sets can be trained on; no corrections or remediation |
| General eval platforms | Measure fluency and generic hallucination, not jurisdiction-specific legal correctness |
| Frontier-lab data vendors | Serve model labs at lab scale; Bench serves the vendor, the compliance team, and the firm — with EU/UK/UAE regulatory depth and managed corrections |