← Lab — Found while building — 2026-08-20
What happens when the judge disagrees with the answer key?
Twice we checked by hand, and twice the judge was right and our labels were wrong.
Corpus fixed, test addedJudge looks trigger-happy → tune the prompt down
Read the flagged traces → find the labels were wrong
What we did
One mode — stopping before doing what the user asked — fired on 13 of 30 traces when only 3 had it planted. The tempting move is to tune the judge down. We read the ten instead.
What came back
Four traces labelled clean asked the agent to cancel or refund an order, and it only reported status. A second run caught the same class again: clean traces that answered half of a compound request.
What it changed
The corpus was fixed and a test asserts it at both sizes, so a mislabelled ground truth can never make a correct judge look wrong again. It also made the planted numbers slightly worse — better labels, honester score.
What it doesn't show
Our corpus, our mistake. It doesn't mean every flag on your traces is right; it means every flag ships with the line it rests on, so you can do what we did.
Other notes
- Can it find failures nobody told it about?6/6
- On real traces, does it agree with human experts?87.2%
- Does its taxonomy match one researchers published?5
- What does a run report when a third of it never finished?183/300
- How much of your trace does the judge actually read?644,000
- Is the quoted evidence actually the evidence?2
- Will it even read my file?12
Run it on your own traces.
Free, no account, parsed in your browser.