Turn your LangSmith runs into evals

LangSmith runs come out as a flat list of nested runs with LangChain's serialization wrapped around every message. GetEvals unwraps that — including the doubly-nested message arrays and `lc: 1` constructor objects — regroups runs into whole traces, and evaluates the trajectory rather than the individual LLM calls.

Drop your traces here

JSONL · JSON · NDJSON · CSV · TSV · TXT · GZ · ZIP

Getting the export out of LangSmith

  1. 1Open the project and select the runs you want — filters and date ranges apply.
  2. 2Use the runs table's Download button (JSONL or CSV), or the SDK: client.list_runs(project_name=...) written out as JSON.
  3. 3Drop the file above. Runs are grouped by trace_id and ordered by dotted_order, so the trajectory comes back in the order it happened.
  4. 4Tool runs are reconstructed as real tool calls with their results attached, so tool-correctness failure modes can be found.

Failed runs (status: error) are surfaced as errors inside the trace rather than dropped — an agent's behaviour after a failure is often where the interesting defect is.

What you get back

A self-contained HTML report: the failure modes discovered in your own traces (not a generic checklist), a heatmap of which traces hit which mode, the exact quoted evidence behind every verdict, and the judge's own reliability numbers. Plus downloadable evals — a promptfoo config, an OpenAI Evals data file, and a portable judge suite — so the analysis turns into a regression suite you keep.

Related