Building LLM evaluation sets that actually predict production

1. Source the questions from reality

• Use real traffic, anonymised, wherever you can. Users ask things no one on the team would think of.
• Weight by importance, not frequency. A rare question about a refund policy or a drug interaction can matter more than a hundred greetings.
• Include the hard cases on purpose: exact identifiers, ambiguous questions, questions the system should refuse, and questions whose answer is "I don't know".
• Add every production failure to the set once it is fixed, so it can never quietly come back.

2. Label what "good" means

A reference answer alone is rarely enough. For each example I record the expected answer, the sources it should come from, and the behaviour that matters most, such as citing a source, refusing, or asking a clarifying question. For RAG systems I evaluate retrieval and generation separately, because a right answer from the wrong document is luck, not quality. The RAG production checklist goes deeper on the retrieval side.

3. Choose metrics that match decisions

Track cost and latency in the same run. A prompt change that lifts quality by two points and doubles cost is a business decision, not an automatic win.

4. Use LLM judges, but calibrate them

Grading thousands of answers by hand doesn't scale, so most teams use a strong model as a judge. The research supports this with caveats. Zheng and colleagues found strong LLM judges can match human preferences at levels comparable to agreement between humans, and also documented position, verbosity and self-enhancement biases¹⁴.

What that means in practice:

• Calibrate first. Have humans label a sample, run the judge on it, and measure agreement before trusting the judge on the rest.
• Swap positions when comparing two answers, and count a win only if it holds both ways.
• Don't reward length. Tell the judge explicitly that longer is not better, and check that it listens.
• Avoid self-grading where you can: use a different model family as judge from the one being evaluated.
• Grade against rubrics, with specific criteria, rather than asking "is this good?".

5. Make evaluation a release gate

1. Version the evaluation set alongside the prompts, retrieval config and model choice.
2. Run the full suite on every change to any of them.
3. Set thresholds per metric, including a maximum regression on critical subsets.
4. Block the release automatically if a threshold fails, the same way a failing unit test would.
5. Record the results with the release, so you can show what was tested and when.

That last point matters beyond engineering. NIST's Generative AI Profile, a companion to its AI Risk Management Framework, emphasises measurement and pre-deployment testing as part of managing generative AI risk². A versioned, gated evaluation suite is the practical evidence for that.

6. Close the loop with production

Offline evaluation predicts production only if you keep checking the prediction. I sample live traffic, run the same judges and metrics on it, and compare against the offline scores. When they drift apart, the evaluation set needs new examples. Linking evaluation results to production traces, using standard telemetry such as the OpenTelemetry GenAI conventions³, makes that comparison much easier. I cover the tracing side in LLM observability with OpenTelemetry.

How I can help

I design evaluation harnesses and release gates for RAG and agent systems, and review existing ones. See my AI architecture work, or analyse your challenge and bring the result to a call.