Our models, our metal: a sovereign LLM platform for a European telecom operator

The situation

A telecom operator in central Europe ran fixed and mobile networks for several million customers. Three teams wanted generative AI: network operations, to read alarms and configuration changes; customer care, to summarise interactions and draft replies; and security, to triage alerts and investigate incidents. Each had started its own pilot with a different hosted model, and the security and legal teams had stopped all three because network topology, customer data and incident details were leaving the company.

Why now. Telecom operators are essential entities under the EU's NIS2 Directive, with duties covering risk management, supply-chain security and incident reporting⁴. Transposition has been uneven, and the Commission took several member states to the Court of Justice in July 2026 for not implementing it³, but national laws are now arriving; Austria's, for example, applies from 1 October 2026². At the same time, capable open-weight models became realistic to run in-house; on 29 October 2026 Mistral released the weights of its trillion-parameter Large 4, pitched at security teams who want to run it on their own infrastructure¹. The operator decided to build one shared, sovereign platform rather than three pilots.

What had to be true

• Sensitive data stays in-house: network, customer and incident data only touch the operator's own infrastructure.
• One platform, three tenants, with quotas, policies and costs kept separate.
• Models chosen by evidence: evaluations on each team's real tasks.
• Reliable on imperfect hardware: GPU, driver and node failures must not take services down.
• A sensible escape hatch: hosted models for non-sensitive work, through the same controls.

Architecture

The data flow, end to end

The hard part: running GPUs like a utility

Buying GPUs was the easy part. Keeping them useful was not. Multi-GPU model servers are sensitive to hardware faults, driver updates and slow terminations, and Kubernetes' traditional view of devices as either present or absent does not cope well with partial failures. The Kubernetes community has documented these failure modes, from device plugin restarts to pods holding devices while they shut down⁵.

The team designed for those failures: health checks on GPUs and model servers, spare capacity so a failed node does not halve a model pool, staged driver and plugin updates, and a gateway that retries on a healthy replica. Scheduling moved to Dynamic Resource Allocation so model servers could ask for the exact accelerators they needed; see DRA for GPU scheduling. Sharing used quotas per tenant, with security given priority during incidents.

Model choice was the other discipline. The largest open models are impressive and expensive to hold in memory. Each tenant built an evaluation set from real tasks, and the gateway routed each task to the smallest model that met its quality bar, keeping the largest model for hard security investigations. More in running a trillion-parameter model yourself and the AI memory squeeze.

How it ran

Key decisions and trade-offs

What I'd do again, and differently

Again: put the gateway in first. It stopped the data leaks on day one and let the platform change underneath without tenant rework. Differently: run failure drills before the first production tenant, not after the first outage. Related: Kubernetes inference gateway, serving with vLLM, telecom observability and SLOs and the Europe page.

If you are planning private AI infrastructure in Europe or elsewhere, see my AI architecture work, or analyse your challenge and bring the result to a call.