Who this is for
You have seen the demos. Perhaps your team built one. It answered the easy questions well and then, in front of a real user, confidently said something wrong. Now legal is nervous, finance wants to know the cost per query and nobody can say whether the next release is better or worse than the last.
I work with leadership teams and engineering leads who are in that position, or who want to avoid it. Typical starting points:
- An LLM pilot that never made it past the steering committee.
- A knowledge assistant over internal documents that users stopped trusting.
- A plan to add agents to a workflow, with no clear view of risk or failure modes.
- Sensitive data that cannot be sent to a public API, and a mandate to use AI anyway.
The problems I solve
Ungrounded retrieval
Chunking documents into a vector store is a starting point, not an architecture. I combine vector search with structured data and, where relationships matter, a knowledge graph (GraphRAG), so answers cite their sources and respect permissions. The ontology and knowledge graph page explains why the semantic layer matters so much.
No way to tell good from bad
Without evaluations, every change is a guess. I build an evaluation harness from real questions early, and gate releases on it. My RAG to production checklist sets out what I check before anything goes live.
Agents without guardrails
Multi-step agents are powerful and easy to get wrong. I use LangGraph to make state and control flow explicit, expose tools through the Model Context Protocol, and put human checkpoints where a wrong action would be expensive.
Data that cannot leave the building
Private, on-prem inference is now practical. I design AI gateways that classify requests by sensitivity, send regulated data to local open-weight models and route the rest to hosted APIs. More on this in running private LLMs on-prem.
My approach
- Start from the decision. What will the system help someone decide or do, and what does a wrong answer cost? That sets the quality bar.
- Model the domain. Capture the key entities and relationships so retrieval and agents work from the same meaning as the business.
- Build the evaluation set before the features. Real questions, expected answers and edge cases, versioned with the code.
- Ship one thin workflow to production. With tracing, cost tracking and guardrails, behind a feature flag.
- Expand on evidence. Add sources, tools and agents only when evaluations show they help.
- Hand over the platform. Your team owns the pipelines, prompts, evaluations and runbooks.
What you get
- An AI use-case portfolio ranked by value, risk and feasibility.
- A reference architecture covering retrieval, orchestration, model routing, security and observability.
- A working production slice: RAG or GraphRAG, LangGraph agents and MCP tools as needed.
- An evaluation harness with quality, latency and cost targets.
- An AI gateway pattern for authentication, PII redaction, logging and spend control.
- Clear guidance on which workloads run on local models and which use hosted APIs.
How an engagement runs
| Phase | Typical length | Outcome |
|---|---|---|
| AI readiness review | 2 to 4 weeks | Use-case ranking, architecture, risks and a roadmap |
| Production pilot | 6 to 10 weeks | One workflow live, measured and governed |
| Scale-out | Ongoing | More use cases on a shared platform, team trained |
Every engagement starts with a free 30-minute call. If you want to frame the problem first, try the challenge analyser.
Common mistakes I see
Choosing the model before the problem
Teams often spend weeks comparing models and almost no time defining what a correct answer looks like. The model choice matters far less than the retrieval, the data and the evaluation set. With those in place, swapping models is a configuration change.
Treating prompts as the product
A clever prompt is easy to copy and hard to maintain. The durable value sits in your data, your domain model and the workflow around the AI: who reviews what, which actions need approval, how errors are caught.
Ignoring cost until the invoice arrives
Token costs scale with usage, context length and retries. I set a cost budget per query from the start, cache what can be cached, and route simple requests to smaller or local models. A system that is accurate but unaffordable at scale is not finished.
Letting agents act without limits
An agent that can send email, change records or call payment APIs needs the same controls as a new employee: narrow permissions, logging and approval for anything irreversible. I design those limits into the graph rather than hoping the prompt holds.
No owner for quality
Someone has to own the evaluation set, review failures each week and decide when a release is good enough. Without that role, quality drifts quietly. I help you define it and train the person who will hold it.
Relevant experience
- Production RAG pipelines built with LangChain and LangGraph. My open-source langchain-rag-production repository shows some of the patterns.
- Built a self-hostable AI platform where agents drive deployments over MCP on Kubernetes.
- Hands-on experiments with local LLM inference on an NVIDIA DGX Spark: open-weight models, LoRA fine-tuning and quantisation, compared against hosted APIs for quality, latency and cost.
- Co-author of peer-reviewed research in Science Immunology, which shapes how I think about evaluation: baselines, honest metrics, reproducibility.
- Data platforms behind the AI: event-driven systems at more than a million transactions a day.
For regulated settings, see healthcare data architecture. If you need someone to own the AI agenda at leadership level, that is the fractional CTO role. You can also browse case studies, including private on-prem LLMs.