When teams first put a self-hosted model behind Kubernetes, they usually reach for the familiar pattern: a Deployment of model servers, a Service in front, and a load balancer spreading requests evenly. It works in a demo. Under real load, some replicas sit with long queues and nearly full GPU memory while others are idle, and tail latency climbs even though the cluster as a whole has spare capacity.
Why LLM traffic breaks ordinary load balancing
The Kubernetes project's introduction to the Inference Extension puts it plainly: LLM inference sessions are often long-running, resource-intensive and partially stateful, and a single GPU-backed server may hold several sessions and in-memory token caches. Load balancers that route by HTTP path or round-robin don't account for model identity, request criticality or the current state of each server1.
Two requests to the same model can differ in cost by orders of magnitude, depending on prompt and output length. And as covered in LLM inference serving, sending a request to a replica that already holds its prompt prefix in cache is far cheaper than sending it anywhere else.
How the extension works
The Gateway API Inference Extension builds on Gateway API, keeping the familiar Gateway and HTTPRoute model and adding inference-specific steps15. Its central resource is the InferencePool: a pool of model-server pods on shared compute, similar to a Service but aware of model serving1. The project documentation lists InferencePool as generally available since v1.0.02.
The original design separated responsibilities by persona: platform operators manage where and how models are served through pools, while model owners manage what is served, including traffic splitting between versions1. The model-facing APIs have continued to evolve, so check the current documentation for your version.
The request flow
A request reaches the gateway, which matches an HTTPRoute to an InferencePool. Instead of forwarding to any pod, the gateway consults an endpoint selection extension, often called the endpoint picker, which examines live pod metrics such as queue length, memory use and loaded adapters, and chooses the pod that can serve the request most efficiently1. Gateway implementations describe the picker tracking signals such as KV-cache utilisation and queue length4.
To the client it's still one ordinary request. The intelligence lives in the gateway layer, where the platform team controls it.
What the benchmark showed
The project compared the extension with a standard Kubernetes Service for a vLLM deployment on H100 GPUs, ramping traffic from 100 to 1,000 queries per second. Throughput was roughly on par, but the extension showed significantly lower p90 per-output-token latency at higher load, and lower end-to-end tail latency as traffic rose beyond about 400 to 500 queries per second1. The authors attribute this to avoiding hotspots that ordinary load balancing creates for long-running inference requests1.
Tail latency is what users feel, and GPU hotspots are what finance pays for. Model-aware routing improves both without buying another GPU.
How I'd adopt it
- Start from Gateway API. If your clusters still use an Ingress controller, the move to Gateway API is the prerequisite and has its own value.
- Pick a gateway implementation that supports the Inference Extension, and confirm which version of the APIs it implements.
- Expose metrics from your model servers that the endpoint picker needs, and check them under load before trusting the routing.
- Load test against your own traffic, comparing tail latency and cost per request with your current setup, as in LLM inference serving.
- Use pools to separate concerns: interactive and batch traffic, regulated and general workloads, or different hardware classes.
- Combine with scheduling and cost controls: GPU allocation through DRA, training jobs through gang scheduling, and spend visibility through FOCUS-based cost allocation.
Where it fits in a private AI platform
An inference gateway is the routing layer in the platform described in private LLMs on-prem and where to run models in the UAE. Above it sits the policy gateway that handles authentication, model approval and logging; below it, the serving engines and GPUs. Keeping those layers distinct makes each easier to change.
How I can help
I design Kubernetes platforms for private and sovereign AI, from GPU scheduling to model-aware routing and cost control. See my AI architecture work, or analyse your challenge and bring the result to a call.
- 1Introducing Gateway API Inference ExtensionKubernetes blog · 5 Jun 2025
- 2InferencePoolGateway API Inference Extension documentation
- 3Gateway API Inference ExtensionKubernetes SIG Network
- 4Inference routingagentgateway documentation
- 5Gateway APIKubernetes documentation