Inference is a server too: securing self-hosted LLM serving for a South Korean fintech

The situation

A South Korean fintech offered payments, savings and investment features in a mobile app. Its AI features, a support assistant, transaction categorisation and document reading, had started on external model APIs. To keep customer data in-house and control cost, the company moved them to open-weight models served on its own GPUs, using vLLM for most models and SGLang for a few latency-sensitive ones. The data science team had done it quickly: models pulled from public repositories, servers running in the same Kubernetes cluster as other services, and serving APIs reachable from anything inside that cluster.

Korea's AI Basic Act, in force from January 2026, adds transparency and risk management expectations for AI systems, particularly high-impact ones⁵. The more immediate concern for the board was simpler: customer financial data now flowed through software the security team had never reviewed.

Why now. Inference engines had become a target. In November 2026, NEAR AI disclosed that an attacker used a remote code execution flaw in SGLang to run code on one of its inference servers and observe in-flight requests, including ones protected by confidential computing¹². vLLM had its own advisories during 2026, including a flaw that let a malicious model repository run code even when remote code was disabled, and a remote code execution path through video inputs³⁴.

What had to be true

• Patched: inference engines tracked and updated like any critical dependency.
• Unreachable: model servers accessible only through an authenticated gateway.
• Contained: a compromised server cannot reach the internet or unrelated systems.
• Trusted models: only vetted models, loaded without running their own code.
• Clean logs: prompts, customer data and keys kept out of telemetry.
• Watched: unusual behaviour on serving nodes raises an alert.

Architecture

All application calls went through an internal inference gateway that authenticated callers, enforced quotas and rejected input types a route did not need, such as video or arbitrary URLs. Model servers moved to a dedicated namespace and node pool, ran as non-root in a sandboxed container runtime, and accepted traffic only from the gateway.

Network policies blocked all traffic to and from the serving namespace except from the gateway and to an egress proxy, which allowed only the internal model registry and the telemetry endpoint⁶. A compromised server would find no route to the internet, to databases or to other services.

The data flow, end to end

The hard part: keeping up with engines and models

Patching fast-moving engines. Inference engines release often, and upgrades can change performance or model support. The team pinned versions per model, subscribed to the projects' security advisories, and kept an evaluation and load test for each model so an upgrade could be validated within days rather than deferred for months.

Trusting models, not just code. A model is not just weights; some repositories include code that runs when loaded. The vLLM advisory showed that disabling remote code was not always enough³. Models were therefore imported once into an internal registry after review, scanned, signed and loaded only from there, with remote code disabled and serving processes unable to reach public repositories.

Not over-trusting confidential computing. The team evaluated confidential computing for the most sensitive workloads, and the NEAR AI incident shaped how: it protects data from the host and operator, not from flaws in the engine inside the enclave¹. Confidential computing became one layer, alongside patching and isolation, not a replacement for them.

How it ran

Key decisions and trade-offs

What I'd do again, and differently

Again: put the gateway and network policies in place first; they cut exposure in days. Differently: build the upgrade evaluation for each model at the start, so security patches never waited on performance testing. Related: the 12 November tech news, AI model supply chain security, Kubernetes inference gateways, the Korean hiring platform case study and where I work.

If you serve your own models and have not reviewed them as you would any other service, see my AI architecture work, or analyse your challenge and bring the result to a call.