- AI features served from more than one region and provider
- Traffic moved between regions and providers by configuration, not code changes
- Overflow to managed model APIs when self-hosted capacity was full
- Cost per feature and per customer visible, not just total spend
- Capacity decisions made from usage data and provider plans, not guesswork
Anonymised composite. Client names, identifying details and confidential figures are withheld; public facts are cited in the references.
The situation
A US software company sold a workflow platform to mid-sized businesses. Over two years it added AI features: summarising records, drafting replies, extracting data from documents and an assistant inside the product. Customers adopted them quickly. The company served open-weight models on GPUs in one cloud region, with a managed model API for a few tasks. As usage grew, adding GPU capacity in that region became slow and uncertain, and latency spiked whenever demand peaked.
Why now. Demand for AI compute was running ahead of supply: Fortune reported Microsoft's CFO saying demand for its cloud still exceeded supply3. New capacity was arriving on long timelines and through complex financing, such as Google's credit backstop for Anthropic's Kentucky data centre lease4. And in the 3 November 2026 midterms, voters sided against data centres in 40 of 41 local ballot measures across 10 states, according to CNBC's analysis1, a sign of growing opposition to new AI infrastructure2. The board's question was simple: what happens to our AI features if we cannot get more capacity where we are?
What had to be true
- No single point of capacity: AI features served from more than one region and provider.
- Moving traffic is configuration, not a code change or a rebuild.
- Customer data rules respected: some customers required US-only processing; some required specific providers.
- Costs visible per feature and per customer, so capacity decisions follow value.
- Quality held when traffic moves between models or providers.
Architecture
The data flow, end to end
- RequestA product feature asks for a task, such as "summarise this record"
- PolicyThe gateway checks the customer's allowed regions and providers
- RouteIt picks the primary region if it has capacity, else the secondary or an approved API
- ServeThe model runs; the response streams back to the product
- RecordModel, provider, region, tokens, latency and customer are recorded
- ReviewWeekly reports show cost and quality per feature, feeding capacity decisions
The hard part: moving traffic without changing behaviour
Routing is easy to draw and hard to trust. A summary from one model in one region should not read differently from a summary produced somewhere else. The team built a small evaluation set for each AI feature from real, consented examples, and ran it against every model and provider combination the gateway was allowed to use. A route was only enabled for a feature when it passed that feature's evaluation; see evaluation sets that predict production.
The second difficulty was operations. Running identical clusters in two regions doubles the surface area unless they are managed as one. All clusters were built from the same infrastructure code, ran the same serving stack, and were operated from a single multi-cluster view, so engineers could see and act on every region from one place5.
Customer data rules shaped routing as much as capacity did. Some contracts required US-only processing or excluded certain providers. Those rules lived in a per-customer policy the gateway checked on every request, so overflow never sent a restricted customer's data somewhere it should not go. See Kubernetes inference gateway and model routing.
Capacity you cannot move is a dependency you cannot manage. Make moving cheap: same code, same clusters, same evaluations, different place.
How it ran
- Second region and clusters
- Gateway and routing
- Failover drills
- Task-level interfaces
- Per-feature evaluations
- Quality monitoring
- Customer data rules
- Provider contracts
- Cost per feature
- Telemetry per request
- Cost per feature
- Capacity forecast
- Task-level interfaces
- Inference gateway
- Customer policies
- Secondary region
- Managed API overflow
- Per-route evaluations
- Failover drills
- Quarterly capacity review
- Contract renegotiation
Key decisions and trade-offs
| Decision | Alternative | Why |
|---|---|---|
| Primary plus secondary region | Active-active everywhere | Resilience without doubling cost |
| Managed APIs as overflow | Self-host all capacity | Absorbs peaks without buying idle GPUs |
| Routes enabled per feature after evaluation | Route anywhere with capacity | Quality stays consistent when traffic moves |
| Per-customer data policy in the gateway | Rules in each feature | One place to enforce contracts |
| Cost per feature and customer | Total AI spend only | Capacity follows value |
What I'd do again, and differently
Again: start with per-request telemetry. It showed that a few features drove most of the cost, which changed the whole capacity plan. Differently: write customer data rules into the gateway before adding the second region; retrofitting them slowed the rollout. Related: planning around the AI capacity crunch, AI cost allocation, the 5 November tech news and the US page.
If your AI features depend on capacity you cannot easily expand, see my AI architecture work, or analyse your challenge and bring the result to a call.
- 1
- 22026 election results show widespread opposition to AI data centersScientific American · Nov 2026
- 3Microsoft's cloud growth accelerates, and so too does the spendingFortune · 28 Oct 2026
- 4
- 5From Kubernetes Dashboard to Headlamp: Understanding the TransitionKubernetes blog · 1 Jun 2026