Sovereign vLLM deployment — and the vendor checklist

Hosting open weights is not sovereignty. This article describes a sovereign vLLM deployment: Kubernetes on GPUs you control, an internal gateway, no required egress, and state that stays on your volumes. It includes a vendor checklist for procurement and the cost drivers to model before buying hardware, and connects the serving design to the systems research behind continuous batching, cache management and quantisation.

Keywords vLLM · sovereign deployment · Kubernetes · no-egress · vendor checklist · GPU cost

The serving layer rests on two systems results. Iteration-level scheduling lets a server schedule one model iteration at a time, so finished requests return and new ones join without waiting for a whole batch^[1]. vLLM adds PagedAttention, which manages the per-request key-value cache like virtual memory and reports 2–4× higher throughput than earlier systems at the same latency^[2]. Weight quantization such as AWQ reduces the memory a model needs by protecting a small fraction of salient weights^[3].

Around the server, Kubernetes carries the lessons of more than ten years of container management at scale^[4], and a default-deny network starts from the zero-trust architecture defined by NIST^[5]. What these sources do not cover is the vendor relationship, which is what the checklist below addresses.

A sovereign vLLM deployment is an OpenAI-compatible inference service you operate on GPUs you control, in a network that does not need to call a US API to answer a prompt. vLLM gives you continuous batching, prefix caching and tensor parallelism.^[2] Sovereignty is the rest of the design: where the nodes live, who can SSH, where logs go, and whether a support engineer in another jurisdiction can pull a core dump.

LMXAI ships this pattern for enterprise workspaces and internal agents. The serving layer is the same family as the sovereign AI platform and the mdGPT gateway that sits in front of it.

• Cluster: Kubernetes in your DC or an EU bare-metal/private cloud account you own.
• Serving: vLLM (optionally AWQ/INT4) behind an internal gateway — FastAPI, SSE, per-namespace keys.
• Network: default-deny egress. Pull images from a mirror you control. No callback to a model vendor at request time.
• State: weights on your volume; KV-cache on the GPU nodes; application logs in your collector (OpenTelemetry).
• Ops: autoscaling on queue depth, not on a vendor’s quota page.

EU hosts that can pass: your own racks; EU bare metal (for example OVHcloud, Hetzner, Scaleway) if you own the account and disable unnecessary extras; a regulated private cloud with a written no-egress design. A hyperscaler “EU region” can be part of a design, but it is not automatically sovereign — read the subprocessors and the privileged-access story.

Tokens are not the bill. The bill is idle GPU time, context length, concurrent sessions, and how often you re-embed a corpus. A 32B AWQ model on a pair of high-memory GPUs often beats a 70B that sits at 15% utilisation. LMXAI sizes this from a traffic sketch and a latency SLO, then adds speculative decoding or expert offload only when the numbers justify the complexity.

A running vLLM service, a gateway with auth and traces, a runbook for weight updates, and the completed vendor checklist for whoever hosts the boxes. You should be able to change GPU vendor later without rewriting the application.

• vLLM GPU sizing for sovereign inference
• vLLM quantization: AWQ, GPTQ, FP8 and INT4
• Production vLLM on Kubernetes
• Sovereign AI FAQ
• Sovereign AI workspace vs Copilot
• Production RAG and agents
• NL: Soevereine LLM-deployment
• Production-grade agentic systems