- Every AI feature has an owned evaluation set that runs before release
- Model and prompt changes ship behind gates with a clear pass or fail
- Traces show each model call, tool call and token count in one view
- Cost and quality are reported per feature, not just per vendor invoice
Anonymised composite. Client names, identifying details and confidential figures are withheld; public facts are cited in the references.
The situation
A software company had moved quickly on AI. Support had an answer assistant, sales had an email drafter, the core product had a summariser and a natural-language search box. Each team had picked prompts, vendors and settings independently. When a model provider released a new version, nobody knew which features would get better and which would quietly break. Quality problems surfaced as customer tickets. Finance saw one large model invoice and could not split it by feature.
Leadership did not want to slow the teams down. They wanted the same confidence in AI releases that they already had in ordinary code releases.
What had to be true
- Every AI feature has an evaluation set that reflects real use, owned by the team that ships it.
- Changes ship through gates: a prompt, model or retrieval change runs the evals before it reaches customers.
- Every model and tool call is traceable, with latency, tokens and errors, using an open standard rather than a vendor format.
- Sensitive content is not logged by default.
- Cost and quality are visible per feature, so product managers can make trade-offs.
Architecture
The data flow, end to end
- ProposeTeam changes a prompt, model or retrieval setting in a branch
- EvaluateCI runs the feature's eval set through the gateway and graders
- GateScores compared with the current version; regressions block the merge
- Roll outChange enabled for a slice of traffic behind a flag
- ObserveTraces and metrics compared between old and new versions
- LearnProduction failures become new eval cases
Tracing followed the OpenTelemetry semantic conventions for generative AI, which standardise how model calls are recorded, including the model, input and output token counts and, when explicitly enabled, prompt and completion content12. Standard metrics for token usage and operation duration made cost and latency comparable across features and vendors3. The detail is in LLM observability with OpenTelemetry.
The hard part: evaluation sets that predict production
Building the pipeline was straightforward. Building evaluation sets that actually predicted how features behaved for customers was not. Generic benchmarks said little about whether the support assistant would answer a specific customer's question correctly.
- Cases drawn from real, anonymised usage, grouped by intent, with a slice of known hard and sensitive cases.
- Deterministic checks wherever possible: format, required fields, citations present, forbidden content absent.
- Rubric-based model graders only where judgement was needed, calibrated against human labels. Research on LLM-as-a-judge found strong models can match human preference agreement, but also documented position and verbosity biases that need controlling4.
- Regular refresh from production misses, so the set evolves with the product.
More on this in evaluation sets that predict production and, for agentic features, evaluating AI agents.
An eval set is a product artefact, not a test fixture. The team that owns the feature owns its definition of good.
How it ran
- Gateway and tracing
- Eval harness in CI
- Dashboards
- Build eval sets
- Define pass criteria
- Ship behind gates
- Content capture policy
- Sensitive-case review
- Map to risk framework
- Route calls via gateway
- GenAI tracing
- Cost per feature
- First eval sets
- Graders and calibration
- Baseline scores
- CI release gates
- Flagged rollouts
- Version comparison
- Production misses to evals
- Model upgrade playbook
- Quarterly review
Key decisions and trade-offs
| Decision | Alternative | Why |
|---|---|---|
| OpenTelemetry GenAI conventions | A vendor-specific tracing SDK | Portable across vendors and backends, fits existing observability |
| Content capture off by default | Log everything for debugging | Prompts and tool arguments can hold sensitive data |
| Deterministic checks before judges | Judge every output with a model | Cheaper, faster and free of judge bias |
| Eval sets owned by feature teams | Central QA writes all evals | Teams know what good looks like for their users |
| Gateway for all model calls | Each team calls vendors directly | One place for keys, routing, limits and cost attribution |
The programme was mapped to the NIST Generative AI Profile, which gave the risk team a recognised structure for issues such as confabulation, information security and data privacy5.
What I'd do again, and differently
Again: start with tracing and cost per feature, because visibility creates demand for evaluation. Differently: write a model upgrade playbook in the first phase. Provider version changes arrive on their schedule, not yours, and a rehearsed path from new model to gated rollout takes the drama out of them.
If your AI features are multiplying faster than your confidence in them, my AI architecture work covers evaluation and observability platforms, or analyse your challenge and bring the result to a call.
- 1Inside the LLM Call: GenAI Observability with OpenTelemetryOpenTelemetry blog · 14 May 2026
- 2Semantic conventions for generative client AI spansOpenTelemetry
- 3Semantic conventions for generative AI metricsOpenTelemetry
- 4Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaZheng et al. (arXiv; NeurIPS 2023) · 2023
- 5