← Lab — Measured run — 2026-08-20
On real traces, does it agree with human experts?
It flags 87.2% of the traces human experts marked as broken, so it reliably tells you which transcripts to open. It describes them in its own vocabulary — 2 of the experts' 12 categories — and it does not pin down where inside a trace the failure sits.
Trace triage validatedWhat we set out to measure, and what we measured
| TRAIL taxonomy coverage (of 12 categories occurring >= 3x) | >= 0.5 | 2/12 | under the bar |
| trace-level recall (traces with >= 1 annotated error) | >= 0.7 | 0.872 | at or above the bar |
| trace-level Cohen's kappa | >= 0.3 | -0.043 | under the bar |
What we did
TRAIL pairs 148 real agent traces (GAIA, SWE-Bench) with 841 expert-annotated errors. We read 40 of them cold — the annotations were never visible to any stage — then matched our modes to theirs with a different model family.
What came back
Triage works: given a pile of traces it points you at the ones worth opening. Two limits came out of the same run. Its own modes line up with 2 of the 12 expert categories, and the turn it cites falls inside the annotated span 18.8% of the time — against 21.4% for a random pointer, so the turn index carries no signal.
What it changed
The product stopped implying it can name the turn. Evidence is quoted text you can search for, not a span index dressed up as a coordinate.
What it doesn't show
TRAIL is error-dense — 1 of 40 judged traces has no annotated error — so specificity is not estimable here at all, and 20 traces were clamped to fit the render budget.
REPRODUCE npm run validate:trail
Full metric tables and every caveat →
Other notes
- Can it find failures nobody told it about?6/6
- Does its taxonomy match one researchers published?5
- What happens when the judge disagrees with the answer key?13 of 30
- What does a run report when a third of it never finished?183/300
- How much of your trace does the judge actually read?644,000
- Is the quoted evidence actually the evidence?2
- Will it even read my file?12
Run it on your own traces.
Free, no account, parsed in your browser.