One GPU, many models: consolidating small AI models for an Indian multilingual marketplace

The situation

An Indian online marketplace connected buyers with small and mid-sized sellers across the country, in English and many Indian languages. Over several years its teams had added small AI models wherever they helped: embedding models and rerankers for search in each major language, classifiers for listing quality and support routing, language identification, and fine-tuned language models that helped sellers write listings. Each team had deployed its models independently, and the simplest way to get a model into production had been to give it its own GPU.

The result was a large GPU bill for hardware that was mostly idle. Many models needed only a fraction of a GPU's memory and compute, and traffic followed Indian daily patterns, with quiet nights and sharp peaks around sales. Adding support for another language meant another set of models and, by habit, more GPUs. Finance wanted the bill down; product wanted more languages.

Why now. GPU partitioning was moving into mainstream managed services. AWS added GPU partitioning to SageMaker AI real-time inference, using NVIDIA Multi-Instance GPU so one GPU can be split into isolated partitions, each with dedicated compute and memory¹². At the same time, reports that Microsoft was winding down rented capacity on older GPUs underlined how quickly hardware choices age⁵. Using every GPU fully had become a design goal, not an afterthought.

What had to be true

• Far fewer GPUs for the same models and traffic.
• No noisy neighbours for latency-critical models such as search.
• Quality held per language and task after any consolidation.
• New languages cheap to add, without new hardware by default.
• Cost visible per model and language, so decisions follow value.
• Teams still independent: shared infrastructure, not a central bottleneck.

Architecture

We sorted every model into one of three groups. Small, latency-critical models, such as search embeddings and rerankers, went onto Multi-Instance GPU partitions, which give each model its own slice of compute and memory with isolation enforced in hardware³. Language and task variants of the same base language model were rebuilt as LoRA adapters on one shared base model, where evaluations showed quality held. Offline work, such as re-embedding the catalogue or scoring listing quality overnight, moved to batch queues that filled spare capacity.

The data flow, end to end

The hard part: isolation, adapters and fixed partitions

Choosing the right kind of sharing. Not all GPU sharing is equal. Partitions give each model guaranteed memory and compute; looser sharing lets models contend for the same resources, so one model's spike or failure can affect others. We kept partitions for anything on the search path and accepted looser sharing only for internal and offline models.

Routing to the right adapter. Adapters only save money if requests reach a server that already has the right adapter loaded; otherwise servers spend time loading and evicting them. The gateway therefore routed by model and adapter as well as load, the pattern the Kubernetes Gateway API Inference Extension describes, where an endpoint picker considers queue length, memory and loaded adapters before choosing a server⁴.

Living with fixed partition layouts. Partition profiles are set per GPU and changing them means draining the workloads on it. Rather than reshaping partitions on demand, we defined a small number of node pools, each with a fixed layout suited to a class of model, and let the scheduler place models on the right pool. Reshaping happened in planned maintenance, not in response to traffic.

How it ran

Key decisions and trade-offs

What I'd do again, and differently

Again: measure real GPU use per model before changing anything. It showed most of the waste sat in a few teams' habits, not in the models themselves. Differently: build per-language evaluation sets before moving to adapters; the first adapter rollout exposed a regression in a smaller language that an overall score had hidden. Related: Kubernetes inference gateways, GPU scheduling with DRA, AI capacity contracts and the India page.

If your GPU bill is growing faster than your AI usage, see my AI architecture work, or analyse your challenge and bring the result to a call.