Does it actually work?

A tool that grades your agent should be willing to be graded itself. Every number below was produced by a command you can run, against a corpus you can rebuild, and is read straight out of the run's artifact file — nothing here is typed by hand. The limitations are in the body, not a footnote.

The short version. On a corpus where the failures were planted and therefore known, it found everything and was almost never wrong. On real traces carrying expert annotations, it reliably picks out which traces are worth your attention — and it does not reproduce an expert's taxonomy, nor point reliably at where in a trace the failure is. Those last two missed their pre-registered bars. They are on this page with the same prominence as the win, because a tool that only publishes its good runs is not evidence of anything.

Can it find failures it was never told about?

A corpus where the answer is known exactly: 100 customer-support agent traces, 60 of them carrying one deliberately planted failure of six known kinds, 40 clean. The pipeline was given the traces and nothing else — no labels, no hint that anything was planted, no list of what to look for.

6/6
planted failure types discovered unaided
100.0%
of failing traces caught
92.3%
of flagged traces really failed
0.89
Cohen's κ vs ground truth (95% CI 0.79–0.98)

It independently discovered all 6 of the 6 planted failure types and named them in its own words. Judging every trace against every discovered mode, it flagged 60 of the 60 traces that carried a planted failure and 5 of the 40 clean ones — 100.0% recall at 92.3% precision, Cohen's κ of 0.894 against the ground truth. Every one of the 600 judge cells returned a verdict; none were dropped.

Threshold set before the runRequiredMeasuredResult
discovery (of 6 planted types)>= 56/6pass
judge precision (pooled)>= 0.850.923pass
judge recall (pooled)>= 0.81pass
trace-level TPR>= 0.851pass
trace-level TNR>= 0.90.875fail
Cohen's kappa>= 0.70.894pass

What this does not show

  • This corpus is synthetic. It proves the machinery finds failures that are genuinely there and does not invent ones that aren't — it does not prove anything about how messy production traces behave.
  • The failures were written to be findable by a careful reader. Real defects are often subtler, and a real corpus will contain kinds nobody thought to plant.
  • 5 clean traces were flagged, which is what puts specificity below its bar. They are worth reading rather than dismissing: an earlier run flagged clean traces too, and on inspection the corpus was mislabeled and the judge was right. The labels have since been corrected, so these 5 are the residual — read them before assuming either way.
  • One number to read carefully: the mode 'Contradicts returned tool output' fires more often than its matching planted type, because it also catches fabrication after a tool error. Discovered modes overlap; the trace-level figures above are the honest summary.

Run it yourself

npm run validate:planted

100 traces · seed 42 · judged by google/gemini-3.7-flash · 664 model calls · $0.66 · run 2026-08-20 · prompts 7fd23382

On real traces, does it find the same things human experts found?

TRAIL pairs 148 real agent traces (GAIA and SWE-Bench scaffolds) with 841 errors annotated by human experts — each with a category from their 21-category taxonomy, and the exact span where it happens. We read 40 of those traces cold. The annotations were never visible to any stage of the pipeline.

87.2%
of traces containing a human-annotated error were flagged
2/12
of the expert error categories independently surfaced
18.8%
of error locations hit — chance is 21.4%
40
real traces read, of 148 available

It flagged 34 of the 39 traces that contain a human-annotated error — 87.2% recall. That is the good news, and it is the part that matters most for the product: given a corpus, it finds the traces worth looking at. The rest is not good news. It independently surfaced only 2 of the 12 error categories humans catalogued, and when it pointed at where in a trace a failure happened, it landed on the annotated span 18.8% of the time — against a 21.4% rate for pointing at random. Localization here is at chance.

Threshold set before the runRequiredMeasuredResult
TRAIL taxonomy coverage (of 12 categories occurring >= 3x)>= 0.52/12fail
trace-level recall (traces with >= 1 annotated error)>= 0.70.872pass
trace-level Cohen's kappa>= 0.3-0.043fail

What this does not show

  • n = 40 traces sampled from the 148 paired traces available (TRAIL publishes 148: 117 GAIA + 31 SWE-Bench). 40 went through open coding and 40 through judging. Every interval on this page is wide; read it as directional.
  • TRAIL is error-dense: zero-error traces in the judged set number 1 of 40. With fewer than 5 negatives, TNR/specificity is NOT meaningfully estimable on this dataset, and trace-level precision is nearly as fragile. Specificity is measured on the planted corpus, which has 40 known-clean traces by construction.
  • 20 of 40 traces exceeded the 60,000-character envelope budget and kept a median 40.2% of their original text (worst 16.0%); their longest parts were head/tail clamped so that no turn was dropped (the turn->span map localization is scored on would be invalid otherwise). The pipeline then clamps each prompt again to 20,000 characters, head and tail, so on the largest traces the judge reads the beginning and the end and not the middle. A defect living only in a removed middle could not have been found.
  • GAIA and SWE-Bench are different agent scaffolds solving different tasks, and their error profiles differ. Every table above breaks out by split; a pooled headline number mixes two populations and should be quoted with the split table beside it.
  • 4 of 6 judged modes matched no TRAIL category. That is not an error: TRAIL's taxonomy is one team's list, and a mode outside it may be a real defect they did not enumerate. Unmatched modes are neither credited nor charged, and they need hand-reading before anyone calls them finds or slop.
  • Coverage is scored only against the 12 categories occurring at least 3 times in the traces we read. 6 rarer categories (Incorrect Memory Usage, Tool Definition Issues, Environment Setup Errors, Authentication Errors, Resource Not Found, Resource Exhaustion) are excluded from the denominator rather than counted as misses — you cannot discover what is barely there.
  • A localization hit means the evidence turn fell within 1 turn of a turn built from the annotated span, and one annotated span can own several turns — the median acceptance window was 6 turns (max 7). A single uniformly-random turn pointer would land inside that window 21.4% of the time, and a judge citing up to 4 spans does better than that by chance alone. Quote localization recall against that baseline, never on its own.
  • Localization is the weakest measurement of the three. The rendered trace the judge reads is not turn-numbered, so its turn index is an eyeball count of role blocks, and when it omits the index the pipeline records 0 — 0 of 75 evidence spans point at turn 0. Treat the localization figure as a floor with a wide error bar, not a measurement of pointing accuracy.
  • Localization excluded 0 annotated error(s) whose span produced no turn in our flattening, 211 whose category was not blind-matched to any of our modes, and 1 whose category string matches no taxonomy leaf. Excluded errors are reported, not scored as misses; scoring them as misses would double-charge failures the coverage number already reflects.
  • Localization precision is quoted two ways because there is no single honest denominator: over evidence spans belonging to blind-matched modes (6/28), which is the like-for-like comparison, and over every evidence span we produced (6/75), which is a lower bound because a mode outside TRAIL's taxonomy can never register a hit.
  • Our flattener rebuilds the conversation from OpenInference span attributes and emits each message only the first time it appears, because LLM spans re-send the whole history on every call. A genuinely repeated identical message therefore shows up once instead of twice. CHAIN and AGENT spans are skipped as duplicates of their children; no annotated error in the published corpus points at one.
  • The blind matcher calls a different model family directly, outside the metered client, so its spend is NOT included in the cost reported for this run.
  • Open coding never reached saturation, so the discovered taxonomy may be incomplete and coverage may be understated.

Run it yourself

npm run validate:trail

40 traces · seed 42 · judged by google/gemini-3.7-flash · 292 model calls · $2.14 · run 2026-08-20 · prompts 7fd23382

Does its taxonomy line up with one researchers wrote?

The MAD corpus carries 19 real multi-agent traces (AppWorld, HyperAgent, AG2, ChatDev, MetaGPT, GAIA) in which three human annotators independently marked which MAST failure modes are present. We ran the pipeline over the full transcripts with no sight of those labels, then matched the modes it invented against MAST's using a different model family, in both presentation orders.

0/5
of our discovered modes matched any MAST mode
0/11
of the MAST modes present were surfaced
n/a
judge-vs-human κ — no matched pair to compute it on
19
human-annotated multi-agent traces

None of them matched. 0 of the 5 modes it discovered corresponded to any of the 16 MAST modes present in these traces, so judge-versus-human agreement could not be computed at all. The modes it found are concrete and task-level — "Hallucination of Code Artifacts and Outputs", "Premature Task Abort or Incomplete Generation". MAST's are process-level properties of multi-agent systems — "Lack of result verification", "Fail to elicit clarification", "Ignoring suggestions from agents". Both are reasonable descriptions of the same transcripts; they are answers to different questions.

Threshold set before the runRequiredMeasuredResult
MAST coverage (of 11 modes occurring >= 3x)>= 0.50/11fail
judge-vs-human kappa (pooled)>= 0.4NaNfail

What this does not show

  • n = 19 traces. Every interval here is wide; read this as directional, not as a benchmark result.
  • Only 11 MAST modes occur in at least 3 of these traces and are scoreable. Modes that never occur are excluded rather than counted as misses — you cannot discover what is not there.
  • These transcripts are plain text from six different agent frameworks, so they are evaluated as text. Tool calls are not linked to their results, which puts tool-correctness failure modes out of reach here by construction.
  • 1 of 19 transcripts exceeded the 350,000-character judging budget and were truncated head-and-tail. A defect living only in a removed middle section could not have been found.
  • Human annotators agree with each other at κ NaN on the scored modes. That is the ceiling: a judge scored against noisy labels cannot exceed the labels' own consistency.
  • No matched pair had enough overlapping cells to score, so the agreement figure is absent rather than estimated.

Run it yourself

npm run validate:mad

19 traces · seed 42 · judged by google/gemini-3.7-flash · 143 model calls · $0.92 · run 2026-08-20 · prompts 7fd23382

What isn't measured yet

τ²-bench — how much benchmark failure is recoverable from the transcript alone — is wired up and has not produced a publishable run. The first attempt lost a third of its judge matrix part-way through when the model key hit its spend limit, and the harness now refuses to score a run with that many holes rather than quoting rates computed over whatever survived. It will appear here when it completes cleanly.

Two things nobody has measured at all: whether the failure modes it discovers are the ones a team would actually prioritise, and whether the insights change what anyone does. Those need users, not benchmarks.