Turn your Langfuse traces into evals

Langfuse shows you what happened. This tells you what keeps going wrong — by reading your traces, working out the failure modes they actually contain, and grading every trace against them. Both the JSON and CSV exports work, including the JSON-string input/output columns the CSV export produces.

Drop your traces here

JSONL · JSON · NDJSON · CSV · TSV · TXT · GZ · ZIP

Getting the export out of Langfuse

  1. 1Open your project and go to Tracing → Traces.
  2. 2Filter to the slice you care about — a time window, a tag, a user, a low score. Every filter you apply is respected by the export.
  3. 3Click Export and choose JSON (preferred — it keeps nested observations) or CSV.
  4. 4Drop the file above. Scores attached to your traces are read as ground-truth labels, which unlocks κ and true/false-positive rates in the report.
  5. 5Prefer the SDK? langfuse.api.trace.list() and write the array to a .json file — that works too.

Observations are grouped back under their parent trace automatically, and generation spans with message-array inputs are reconstructed into real turns rather than flattened into text.

What you get back

A self-contained HTML report: the failure modes discovered in your own traces (not a generic checklist), a heatmap of which traces hit which mode, the exact quoted evidence behind every verdict, and the judge's own reliability numbers. Plus downloadable evals — a promptfoo config, an OpenAI Evals data file, and a portable judge suite — so the analysis turns into a regression suite you keep.

Related