← Lab — Found while building — 2026-08-20
Is the quoted evidence actually the evidence?
For things the agent did, yes. For things it failed to do, the quote pointed at the wrong half.
Prompt and tests changeduser: "cancel my order and refund me"
user: "cancel my order and refund me" + agent's reply, with no cancellation
What we did
We read a live run's verdicts the way a sceptic would — not asking whether the verdict was right, but whether the quote under it proved anything.
What came back
Verdicts about what the agent did cited it perfectly. Verdicts about what it omitted cited only the user's request, which makes the reader open the trace to see the gap.
What it changed
The judge prompt now requires both halves of the gap. A test asserts every quote appears verbatim and isn't lifted from the user's turn, and unverified quotes are badged rather than highlighted.
What it doesn't show
Verbatim-checking catches an invented quote. It can't catch a real quote that doesn't support the verdict — that still needs a human.
Other notes
- Can it find failures nobody told it about?6/6
- On real traces, does it agree with human experts?87.2%
- Does its taxonomy match one researchers published?5
- What happens when the judge disagrees with the answer key?13 of 30
- What does a run report when a third of it never finished?183/300
- How much of your trace does the judge actually read?644,000
- Will it even read my file?12
Run it on your own traces.
Free, no account, parsed in your browser.