Evals: testing what an LLM says

Datasets, graders and regression gates for output that's never exactly the same twice.

0/7
☰
Evals · architecture

You can't assert one exact answer when the same input can give different outputs. So you build a set of real cases, grade every run with code, a model or a person — and let the scores decide what ships.

+ The whole system in one paragraph

A unit test checks that the same input gives the same output. An LLM gives different outputs for the same input, so you test it differently: a dataset of real tasks, a runner that uses the code you ship, graders that score each answer, and a gate that compares the scores with the last version. After shipping, you keep grading samples of real traffic, and every failure you find becomes a new test case.

In this lesson

7 chapters · ~9 min · a quick check in each

Everything is on this page — just scroll down ↓

New to this? Warm up first+ optional
1
At a glance

Words you'll meet

You'll meet these words on every diagram of this topic. Skim them now and come back whenever one shows up.

ⓘ Details & sources

The terms, in plain words.

EvalsTests for AI answers: a set of real cases,graded automatically or by people, before eachchange ships.Golden setReal questions with the answers that are knownto be right: your eval dataset.TaskOne test case: an input plus what counts assuccess.TrialOne attempt at a task. Outputs vary, so youoften run several.TranscriptThe full record of a trial: every message, toolcall and result.OutcomeThe final state a trial leaves behind (is therecord really saved?), not just what the modelsaid.GraderThe code, model or person that scores a trial.RubricWritten criteria a grader scores against: whata good answer must contain.LLM judgeA model that grades another model's answersagainst a rubric or a reference.AgentA model that chooses its own steps, callingtools in a loop until the task is done.Tool callThe model asks your code to run a function;your code runs it and replies.Capability evalHard tasks that measure what it can't do yet. Alow pass rate is expected.Regression evalTasks it already gets right, to catch a changethat breaks them. Should pass near 100%.pass@kThe chance that at least one of k triessucceeds.pass^kThe chance that all k tries succeed.Cohen's kappaHow often two graders agree, corrected forluck: 1 = always, 0 = no better than chance.BenchmarkA public test set that everyone reports scoreson.ContaminationBenchmark tasks (or their answers) leaked intoa model's training data, so a score partlymeasures memory.Offline / online evalTesting a fixed dataset before you ship /grading samples of real traffic after.CanaryShipping a change to a small group first andwatching it before everyone gets it.
Chapter 1 of 7

The whole system

2
Side by side

Why unit tests aren't enough

Models can give different outputs for the same input (OpenAI), so an exact assert fails good answers or passes on luck. You grade over many cases instead.

ⓘ Details & sources

Keep your unit tests. Add evals for what the model says.[1]

Unit testSame input → same outputassert equals the expected valueOne run proves itPass or fail, checked by codevsEvalSame input → different outputsGraded against a rubric or a referenceMany cases, sometimes many trials eachScores compared across versionsYour code still gets unit tests. Themodel's answers get evals.
3
Architecture

Evals, on one page

Cases flow into a runner that uses the code you ship; graders score each answer; a gate decides. After shipping, sampled production failures loop back into the dataset.

ⓘ Details & sources

Top half: before you ship. Bottom row: after. Click a box, play a flow.[1][2][3][7]

Watch it happen:
every change123344compare5ship6flagged failuresEdge cases: Cases people write on purpose: the weird, the rare and the hostile.Edge casesadversarial · rareProduction logs: What real users actually asked, and the answers that went wrong.Production logsreal conversationsDataset: Your test cases, each with what counts as a right answer.Datasetgolden set · labelsYour change: A new prompt, a new model or new code: anything that can change an answer.Your changeprompt · model · codeRunner: Runs every case through your app, exactly as production would.Runnersame code you shipCode checks: Plain code that checks an answer: exact match, valid JSON, tests pass.Code checksexact · schema · testsLLM judge: Another model that grades the answer against written criteria.LLM judgerubric · pass/failHuman review: People grade the hardest cases and label the examples the judge learns from.Human reviewexperts labelScores: The results, case by case, next to the current version's.Scoresvs last versionCI gate: The check that lets a change through only if the scores hold.CI gateship or blockCanary: The change goes to a small group first, before everyone gets it.Canarya few usersProduction: Everyone using the app for real.Productionreal usersSampling: A slice of real traffic is graded after shipping, to catch what tests missed.Samplinggraded live
youyour system dashed = our simplification
123456
Step 1 of 6

A developer opens a pull request that edits the prompt. CI starts the eval runner.

4
Pipeline

Build it up, one piece at a time

You don't need the whole picture on day one. Start with a small set of real failures, grade what code can grade, add a judge only after checking it, then put it in CI and feed it from production.

ⓘ Details & sources

Twenty real cases graded by code beat a perfect plan you never run.[2][3][10]

1 · 20–50 real tasksfrom real failures1grade2 · Code checksautomate what you can2nuance3 · An LLM judgechecked against humans3every PR4 · A gate in CIon every change4after ship5 · Production samplesfailures become tests5

5 steps. Follow the flow one step at a time — what happens first, and what comes next.

Quick check

Why can't a plain unit test (assert equals) check what an LLM says?

Pick one — or just keep scrolling, it's not graded.

Keep going — it's free

Create your free account to unlock the rest of this lesson, keep your progress and pick up right where you left off.

  1. 2The dataset
  2. 3Graders
  3. 4LLM as judge
  4. 5Agents
  5. 6Production
  6. 7Benchmarks vs your evals
Create my free account →I already have one — log in