A recurring conversation with clients goes like this. Leadership wants generative AI. Legal and security say the data cannot leave the building. The project stops at "we can't". Sometimes that answer is correct. Often it just means nobody has looked seriously at running models privately.
So I have been doing exactly that on my own bench: running open-weight models on an NVIDIA DGX Spark, a compact GB10-based machine with 128 GB of unified memory. These are working notes rather than a benchmark report. I am deliberately not publishing numbers here, because they depend heavily on the model, the quantisation and the workload, and I would rather you measure your own.
Why private LLMs at all?
- Data never leaves your infrastructure. For patient records, financial data or legal documents, this can be the difference between a project and no project.
- Residency and sovereignty. Regulators in the US and the Gulf increasingly ask where data is processed, not just where it is stored.
- Predictable cost. Steady, high-volume workloads can be cheaper on owned hardware than per-token pricing.
- Control. You choose when models change. No silent upgrades that shift behaviour overnight.
The costs are real too: hardware, operations, security patching, and the engineering time to evaluate and maintain models. Private AI is a platform decision, not a download.
What the bench is teaching me
A few qualitative observations so far:
- Memory is the constraint that matters most. Unified memory changes what fits on a single box. Model size, quantisation and context length all compete for the same pool, and long contexts eat more than people expect.
- Focused tasks are where local models shine. Extraction, classification, summarisation of known document types and grounded question answering work well with good retrieval. Open-ended reasoning across messy inputs is where hosted frontier models still pull ahead.
- Retrieval quality matters more than model size. A mid-sized model with excellent context often beats a larger model with poor context. Everything in my RAG production checklist applies doubly here.
- Throughput and latency are different problems. One user chatting and fifty concurrent agents calling the model are very different loads. Test the one you will actually have.
Sizing the model to the job
The instinct is to run the biggest model that fits. I now start the other way round: pick the smallest model that passes your evaluation set, then scale up only where it fails.
| Task | Starting point |
|---|---|
| Classification, routing, PII detection | Small models; often fast enough to run on every request |
| Extraction into structured fields | Small to mid-sized models with strict output schemas |
| Grounded Q&A over your documents | Mid-sized models plus strong retrieval |
| Complex multi-step reasoning, agents | Larger local models, or route to hosted models when data allows |
A mix of models, each sized to its task, is usually cheaper and faster than one large model doing everything.
Quantisation: the main trade-off
Quantisation stores weights at lower precision, which cuts memory and speeds up inference. It is what makes larger models practical on a single machine. The cost is some loss of quality, and that loss is uneven: it can be invisible on simple tasks and noticeable on precise ones, such as arithmetic, code or strict formats.
- Never trust a general leaderboard for this. Run your own evaluation set at each precision you are considering.
- Watch structured output closely. A model that occasionally breaks a JSON schema is a production incident waiting to happen.
- Remember the context window also needs memory. A more aggressive quantisation might be the price of the context length you need.
Fine-tuning with LoRA, carefully
Fine-tuning is often reached for too early. My order of operations is: better retrieval, better prompts and examples, then fine-tuning. LoRA adapters are a practical way to fine-tune on modest hardware, because you train a small set of additional weights rather than the whole model.
Where LoRA has been worth it in my experiments:
- Teaching a consistent output format or house style.
- Domain vocabulary, for example terms and abbreviations from a specific ontology.
- Making a smaller model reliable on one narrow task, so you can avoid a larger one.
Where it has not: injecting facts. Facts change; keep them in retrieval and the knowledge graph, not in model weights. I cover why in Ontology vs knowledge graph vs semantic layer.
Routing: local where it must be, hosted where it can be
Very few organisations need everything local. The practical architecture is a router that decides per request:
- Classify the request and its data: does it contain regulated or confidential information?
- Sensitive requests go to local models, always.
- Non-sensitive requests that need maximum capability can go to hosted models, under contract and with logging.
- Everything is evaluated with the same suite, so you know what quality you are getting on each path.
This gives you the safety of local inference without giving up the strengths of frontier models for the work that is allowed to use them.
The AI gateway
Whether models are local or hosted, I put a gateway in front of them. It is the single place to enforce policy:
- Authentication and quotas per user, team and application.
- PII detection and redaction before anything reaches a model that does not need it.
- Routing rules by data sensitivity, cost and task.
- Logging and audit of every prompt and response, with retention rules that match your regulations.
- Cost tracking per use case, so the business can see what each workflow costs.
The gateway also makes models swappable. When a better open-weight model appears, you change a route, not every application. In the self-hosted AI platform I built on Kubernetes, the same principle applies to agents calling tools over MCP: one governed entry point.
Evaluation is not optional
Every decision above, model size, quantisation, fine-tuning, routing, depends on measurement. Keep an evaluation set of real tasks with known good outputs, run it against every model and configuration, and track quality, latency and cost together. Without it, private AI becomes a debate about opinions; with it, the trade-offs are visible and the decisions are easy to defend to a risk committee.
Operating it like production
A model running on a desk is an experiment. A model serving a hospital or a bank is infrastructure, and needs the same care as any other critical service.
- Packaging. Containerise the serving stack and pin versions of the model weights, runtime and drivers. "It worked last week" is not a release process.
- Scheduling. For more than one box, run inference on Kubernetes with GPU-aware scheduling, so capacity is shared between teams and workloads.
- Health and capacity. Monitor queue depth, time to first token, tokens per second and memory headroom. Alert before users notice.
- Model lifecycle. Treat a model change like a schema migration: evaluate, stage, roll out gradually, keep the previous version ready to roll back.
- Security. Model weights are artefacts; scan, sign and store them like any other dependency. Isolate the inference network and log access.
A sensible first project
If you are considering private AI, I would not start with a general chat assistant for the whole organisation. I would start with one narrow, high-value workflow on sensitive data that is currently done by hand. Document triage, extracting fields from referral letters, or summarising case files for a reviewer are good examples. They have clear inputs and outputs, an obvious way to measure quality, and a human already in the loop.
Run it on a single machine, build the evaluation set from real examples, put the gateway in front from day one, and measure quality, latency and cost honestly against the manual process. If it works, you have a business case and a reusable platform. If it does not, you have learned that cheaply, with no data having left the building.
Questions to settle before buying hardware
Hardware is the easy part to buy and the hard part to justify later. Before ordering anything, I would want clear answers to these:
- Which specific workflows will run locally, and what data do they touch?
- What quality bar must they meet, measured on which evaluation set?
- What are peak and average loads: one analyst, a department, or automated agents running all night?
- Who will operate the platform, patch it and respond when it fails?
- What is the exit plan if a hosted option later becomes acceptable to your regulators?
A desk-side machine like the DGX Spark is a good way to answer most of these cheaply. It lets a small team prove quality and workflow on real data before committing to a larger cluster. That is how I am using mine: as a place to learn quickly, not as the end state. The lessons transfer directly to larger deployments, because the questions about quality, routing and governance stay the same at any scale.
How I can help
I help organisations design private and hybrid AI architectures: model selection, gateways, routing and evaluation, particularly for regulated data in healthcare and finance, across the US and the UAE.
If your AI plans are stuck at "the data can't leave", run the challenge analyser or read the private AI case study in my work.