Retrieval-augmented generation is the most common pattern in enterprise AI, and the most common place for AI projects to stall. Building a demo that answers ten hand-picked questions from a folder of PDFs takes days. Building a system that answers thousands of unpredictable questions, correctly, within budget, without leaking data, takes deliberate engineering.
I have built production RAG systems with LangChain and LangGraph, and agent tooling over MCP in a self-hosted AI platform I built. This is the checklist I use. It is not exhaustive, but if you can tick every item, you are in better shape than most.
1. Define the job before the architecture
- Which decisions does the system support, and who makes them?
- What does a wrong answer cost? Annoyance, a bad customer experience, a compliance breach?
- Must every answer be cited? Can the system say "I don't know"?
- What is the acceptable latency and cost per query?
The answers decide almost everything else. An internal search helper and a clinical decision aid are both "RAG", and they share very little beyond the name.
2. Build an evaluation set first
This is the item teams skip most, and the one I insist on. Before tuning anything, collect a set of real questions with known good answers and the source passages that support them. A few hundred is a good start; even fifty is far better than none.
- Use real user questions where possible, including the awkward ones.
- Include questions the system should refuse or answer with "not enough information".
- Measure retrieval separately from generation: did the right passages come back, and was the answer faithful to them?
- Run the set on every change to prompts, models, chunking or indexes. Treat a drop in scores like a failing test.
If you cannot say whether last week's change made answers better or worse, you do not have a production system. You have a demo with users.
3. Get retrieval right
Most bad answers are retrieval failures, not model failures. If the right context is not in the prompt, no model will save you.
Chunking
- Chunk on document structure (sections, headings, table rows), not fixed character counts.
- Keep metadata with every chunk: source, date, owner, access level, document type.
- Test a few chunk sizes against your evaluation set rather than guessing.
Hybrid search
- Combine vector similarity with keyword search. Product codes, names and acronyms are often missed by embeddings alone.
- Add a reranking step on the top candidates; it is usually the cheapest quality gain available.
- Filter by metadata before ranking: date ranges, document types, and above all permissions.
4. Ground answers in a knowledge graph where it matters
Similarity search is good at "find me passages about X". It is poor at precise questions that depend on relationships: "which of my contracts renew this quarter?" or "which patients on this drug had this lab result?"
For those, I ground retrieval in a knowledge graph built on a domain ontology, an approach often called GraphRAG. The graph resolves entities, traverses relationships exactly, and enforces permissions, then documents are retrieved only for the entities that matter. I explain the building blocks in Ontology vs knowledge graph vs semantic layer.
You do not need a graph for every use case. You need one when wrong facts are expensive.
5. Orchestrate explicitly
Simple question-answering is a pipeline. Anything with branches, tools, retries or approvals is a workflow, and should be modelled as one. This is where LangGraph earns its place: state is explicit, transitions are explicit, and each step can be tested on its own.
- Keep each node small: classify the question, retrieve, check sufficiency, answer, verify.
- Expose tools through a consistent interface. The Model Context Protocol (MCP) is now a sensible default for giving agents access to systems, with clear schemas and permission boundaries.
- Put human approval steps where actions have consequences: sending an email, updating a record, placing an order.
- Cap loops and tool calls. An agent without limits will eventually find an expensive way to fail.
6. Guardrails and permissions
- Access control at retrieval time. Filter by the user's permissions before anything reaches the model. Never rely on the prompt to hide data.
- PII handling. Redact or mask sensitive fields where they are not needed, and log what was sent to which model.
- Input checks. Detect prompt injection patterns, especially in retrieved documents and tool outputs, not just user input.
- Output checks. Verify citations point to retrieved sources, and block answers that fail policy.
- Data residency. Know which model endpoints process which data, and where. For highly sensitive data, consider running models on your own hardware.
7. Observability, cost and latency
Treat the LLM system like any other production service, with a few extra measurements.
| Measure | Why |
|---|---|
| Full trace per request (retrieval, prompts, tool calls, outputs) | Debugging a bad answer is impossible without it |
| Cost per query and per use case | Surprise invoices kill AI programmes faster than bad answers |
| Latency by step | Usually retrieval or reranking, not generation, is the slow part you can fix |
| User feedback and escalation rate | The cheapest source of new evaluation cases |
| Evaluation scores over time | Catches silent regressions after model or data changes |
To control cost: cache stable answers and embeddings, keep context windows tight, and route simple requests to smaller models. Not every question needs your largest model.
8. Roll out like a product, not a launch
- Start with one workflow and one group of users who will give honest feedback.
- Ship behind a feature flag, with a clear way for users to report bad answers.
- Review traces and failures weekly; turn each failure into an evaluation case.
- Expand to more users and use cases only when evaluation scores are stable.
- Write down the operating model: who owns quality, who approves model changes, who is on call.
The checklist, condensed
- Job, cost of error and success criteria written down
- Evaluation set built and run on every change
- Structure-aware chunking with metadata
- Hybrid search plus reranking
- Knowledge graph grounding where facts and relationships matter
- Explicit orchestration, tools over MCP, capped loops
- Permissions enforced at retrieval, PII controls, injection checks
- Tracing, cost per query, latency by step
- Staged rollout with a feedback loop and a named owner
The failure modes I see most
When a RAG system disappoints, the cause is usually one of a short list. Recognising the pattern saves weeks.
The right document is never retrieved
Symptoms: confident answers that ignore an obvious source. Causes: the document was never indexed, chunking split the key passage, or the user's wording does not resemble the document's. Fixes: index coverage checks, structure-aware chunking, hybrid search and query rewriting.
Retrieved, but drowned out
The right passage is in the context, but surrounded by twenty near-misses, and the model picks the wrong one. Fixes: rerank and cut the context down. More context is not better context.
Stale or conflicting sources
Last year's policy and this year's policy both exist, and the system quotes the old one. Fixes: dates and status in metadata, filters for current documents, and an owner for each content source. This is a content governance problem dressed up as an AI problem.
Answers that are right for someone else
The system answers using data the user should not see, or data about the wrong customer. Fixes: permission filtering at retrieval time, and entity resolution through a knowledge graph so "my account" means the right thing.
Agents that wander
Multi-step agents loop, call tools repeatedly or take actions nobody intended. Fixes: explicit state machines, capped iterations, narrow tools with clear schemas, and approval steps before side effects.
Who owns it after launch
The last item on any checklist should be ownership. An LLM system degrades if nobody tends it: documents change, models are updated, users find new ways to ask questions. Someone needs to own the evaluation set, review failures, approve model changes and decide when a use case is ready to expand. In smaller organisations that is often a product owner paired with an engineer; in larger ones, a small AI platform team. Write the names down before launch, not after the first incident.
A note on frameworks
LangChain, LangGraph and similar libraries save a lot of time, and I use them. They do not replace design. Keep your retrieval logic, prompts and evaluation in code you own and understand, wrap framework components behind small interfaces, and resist adding layers you cannot explain. When something breaks at two in the morning, you want to be reading your own code, not a stack trace through five abstractions. Frameworks also change quickly, so keeping them at the edges makes upgrades a routine job rather than a rewrite.
How I can help
As an AI architect, I take LLM systems from pilot to production: retrieval, evaluation, LangGraph orchestration, guardrails and cost control, grounded in a knowledge graph where accuracy matters.
If your pilot has stalled, run the challenge analyser for a quick diagnosis, or look at the case studies in my work.