← Lab — Found while building — 2026-08-20
What does a run report when a third of it never finished?
It used to report success. Now it refuses to score, and says why.
Guard added, bad run discardedstatus: "ok" — 117 of 300 cells scored
status: "unpublishable" — 61% of the matrix is missing
What we did
Our model key hit its spend limit mid-run and every later call errored. Per cell the pipeline did the right thing — a dead call becomes an explicit ungraded, never a silent pass. The scoring layer above it did not.
What came back
The run reported ok with 183 of 300 cells missing, every rate computed over the cells that happened to survive. It looked completely plausible. That is the dangerous part.
What it changed
Any run above 10% ungraded refuses to publish and prints the cause. The invalid artifact was deleted rather than quietly kept, which is why τ²-bench shows as unmeasured instead of showing numbers.
What it doesn't show
A threshold is still a threshold. A run with 9% missing publishes — with the ungraded count printed beside every rate.
Other notes
- Can it find failures nobody told it about?6/6
- On real traces, does it agree with human experts?87.2%
- Does its taxonomy match one researchers published?5
- What happens when the judge disagrees with the answer key?13 of 30
- How much of your trace does the judge actually read?644,000
- Is the quoted evidence actually the evidence?2
- Will it even read my file?12
Run it on your own traces.
Free, no account, parsed in your browser.