← Lab — Found while building — 2026-08-20
How much of your trace does the judge actually read?
For a while, at most 20,000 characters — about 3% of the longest transcripts we tested — and nothing said so.
Budget per run, truncation disclosedEvery prompt sees 20,000 chars, silently
Budget set per run, clamping reported in the report
What we did
The rendered-trace budget was compiled into the reading and judging stages, so no caller could raise it. Invisible on short chats; not on multi-agent transcripts.
What came back
The first multi-agent run was scoring a judge on an excerpt. Re-run at full length the finding held — but it could have been a truncation artifact and nothing in the report would have told us.
What it changed
The budget is per-run state now. The product keeps a conservative default, validation runs raise it, and the report names which traces were clamped and how much survived.
What it doesn't show
Disclosed, not solved. A defect living only in the middle of a very long trace can still be missed.
Other notes
- Can it find failures nobody told it about?6/6
- On real traces, does it agree with human experts?87.2%
- Does its taxonomy match one researchers published?5
- What happens when the judge disagrees with the answer key?13 of 30
- What does a run report when a third of it never finished?183/300
- Is the quoted evidence actually the evidence?2
- Will it even read my file?12
Run it on your own traces.
Free, no account, parsed in your browser.