A common story in private AI projects: the team downloads an open-weight model, gets it answering questions on a GPU, and declares the hard part done. Then fifty users arrive at once, latency climbs, the GPU bill looks alarming and someone suggests buying more hardware. Usually the problem isn't the hardware. It's how requests are being served.
Why serving is hard
Generating text has two phases. Prefill processes the whole prompt at once and is compute-heavy. Decode produces one token at a time and is limited by how fast weights and cached state can be read from memory. Each in-flight request also holds a key-value (KV) cache that grows with its length. Serving well means keeping the GPU busy with as many requests as possible without running out of memory for those caches.
The techniques that matter
| Technique | What it does | Why you care |
|---|---|---|
| PagedAttention | Stores the KV cache in fixed-size blocks, like virtual memory pages, instead of large contiguous chunks1 | Far less wasted memory, so many more concurrent requests per GPU |
| Continuous batching | Adds and removes requests from the running batch at each step instead of waiting for a whole batch to finish | Short requests don't wait behind long ones; utilisation stays high |
| Prefix caching | Reuses the KV cache for shared prompt prefixes across requests | Long system prompts and shared documents aren't recomputed every time |
| Quantisation | Stores weights at lower precision | Bigger models fit on fewer GPUs, at some quality risk you must test |
The vLLM paper, presented at SOSP 2023, reported 2 to 4 times higher throughput than earlier state-of-the-art systems at similar latency, with bigger gains for longer sequences and larger models12. SGLang took prefix reuse further with RadixAttention, which keeps cached prefixes in a radix tree so they can be shared across many requests34.
Most enterprise workloads share long prefixes: the same system prompt, the same policies, the same retrieved documents. Designing prompts so the shared part comes first is one of the cheapest performance wins available.
Choosing an engine
vLLM and SGLang are the two open-source engines I evaluate most often; hardware vendors and cloud providers offer their own. Rather than trusting published benchmarks, I test on the client's workload:
- Capture a realistic traffic sample: prompt lengths, output lengths, concurrency and how much prefix is shared.
- Measure what users feel: time to first token and tokens per second per user at your target concurrency, not just total throughput.
- Measure what finance feels: cost per thousand requests at that concurrency, on the hardware you will actually buy or rent.
- Check operational fit: model support, OpenAI-compatible APIs, metrics, upgrade cadence and how it runs on your platform.
- Re-test quality after quantisation with your evaluation set, not a generic benchmark.
Running it on Kubernetes
Ordinary load balancing works poorly for LLM servers, because requests vary so much in cost and because sending a request to the replica that already holds its prefix in cache is far cheaper. The Gateway API Inference Extension was introduced to make routing on Kubernetes aware of model-serving concerns like these5. Combined with GPU scheduling from Kubernetes DRA and the gateway changes in moving to Gateway API, it gives a coherent platform for private models.
Common mistakes
- Sizing from a demo. One user on one GPU tells you nothing about fifty. Load test at target concurrency.
- Ignoring output length. Long outputs dominate decode time. Cap them where the product allows.
- Prompt layouts that defeat caching. Putting a timestamp or user ID at the start of every prompt makes each prefix unique.
- Prototyping on different hardware. A desktop bench like the one in DGX Spark as an AI architect's bench is great for quality and integration, but it can't tell you how production hardware will perform.
How I can help
I design and tune private LLM platforms, from engine selection and load testing to Kubernetes routing and cost per request. See private LLMs on-prem and my AI architecture work, or analyse your challenge and bring the result to a call.
- 1Efficient Memory Management for Large Language Model Serving with PagedAttentionKwon et al. (arXiv; SOSP 2023) · 2023
- 2vLLM: a high-throughput and memory-efficient inference and serving engine for LLMsUC Berkeley Sky Computing Lab
- 3SGLangThoughtworks Technology Radar
- 4What is SGLang?DigitalOcean
- 5Introducing Gateway API Inference ExtensionKubernetes blog · 5 Jun 2025