← Lab — Measured run — 2026-08-20
Can it find failures nobody told it about?
We hid 60 failures of 6 kinds in 100 traces and handed them over with no answer key. It named all 6 kinds itself and caught every planted trace.
5 of 6 pre-set bars metWhat we set out to measure, and what we measured
| discovery (of 6 planted types) | >= 5 | 6/6 | at or above the bar |
| judge precision (pooled) | >= 0.85 | 0.923 | at or above the bar |
| judge recall (pooled) | >= 0.8 | 1 | at or above the bar |
| trace-level TPR | >= 0.85 | 1 | at or above the bar |
| trace-level TNR | >= 0.9 | 0.875 | under the bar |
| Cohen's kappa | >= 0.7 | 0.894 | at or above the bar |
What we did
100 support-agent traces we wrote. 60 carry one planted failure of 6 kinds; 40 are clean. The pipeline got the traces and nothing else — no labels, no hint anything was planted.
What came back
It arrived at all 6 planted types in its own words, caught 60 of 60 broken traces and missed none, and flagged 5 of 40 clean ones. All 600 judge cells returned a verdict, for $0.66.
What it changed
This corpus is the gate every change runs through. Twice it has failed in a way that turned out to be our labels' fault rather than the judge's.
What it doesn't show
The corpus is synthetic and we wrote the failures, so they are findable by construction. Specificity landed at 87.5% against a 0.90 bar we set beforehand — 5 clean traces drew a flag, and each is worth reading rather than assuming.
REPRODUCE npm run validate:planted
Full metric tables and every caveat →
Other notes
- On real traces, does it agree with human experts?87.2%
- Does its taxonomy match one researchers published?5
- What happens when the judge disagrees with the answer key?13 of 30
- What does a run report when a third of it never finished?183/300
- How much of your trace does the judge actually read?644,000
- Is the quoted evidence actually the evidence?2
- Will it even read my file?12
Run it on your own traces.
Free, no account, parsed in your browser.