Lab

What we tested, and what came back.

A tool that grades your agent should be measured itself. Every run below is published with its numbers, its limits and the one command that reproduces it.

Measured runs — every figure read from the run's own artifact

6/6failure types found unaidedCan it find failures nobody told it about?A corpus where we knew the answer and the pipeline did not.5 of 6 pre-set bars met87.2%broken traces flagged (34/39)On real traces, does it agree with human experts?Triage validated against 841 annotations made by human experts.Trace triage validated5modes discovered, all task-levelDoes its taxonomy match one researchers published?Same transcripts, two vocabularies — ours and a published one.A different level of description

Found while building — what we learned and hardened

13 of 30traces flagged, where 3 were plantedWhat happens when the judge disagrees with the answer key?The judge failed our own labels, and it was right.Corpus fixed, test added183/300judge cells that never returned a verdictWhat does a run report when a third of it never finished?A run that lost half its cells used to look healthy.Guard added, bad run discarded644,000characters in the longest transcript testedHow much of your trace does the judge actually read?How much of a long transcript the judge actually sees.Budget per run, truncation disclosed2classes of weak citation found by reading verdictsIs the quoted evidence actually the evidence?What a verdict about something missing has to quote.Prompt and tests changed12formats with dedicated parsersWill it even read my file?Twelve formats, hostile files, and a parse you see first.224 tests, hostile files included

Not measured yet

  • τ²-bench. Wired up; the first run lost part of its judge matrix to an upstream limit, so it was discarded rather than scored. It appears here when it completes cleanly.
  • Whether the modes it finds are the ones you'd prioritise. Needs teams, not benchmarks.
  • Whether any of it changes what a team ships. The number we most want and least have.

Prefer raw tables? Every metric, threshold and caveat →