Sovereign vLLM deployment — and the vendor checklist

What “sovereign vLLM” means

A sovereign vLLM deployment is an OpenAI-compatible inference service you operate on GPUs you control, in a network that does not need to call a US API to answer a prompt. vLLM gives you continuous batching, prefix caching and tensor parallelism. Sovereignty is the rest of the design: where the nodes live, who can SSH, where logs go, and whether a support engineer in another jurisdiction can pull a core dump.

LMXAI ships this pattern for enterprise workspaces and internal agents. The serving layer is the same family as the sovereign AI platform and the mdGPT gateway that sits in front of it.

Reference shape

• Cluster: Kubernetes in your DC or an EU bare-metal/private cloud account you own.
• Serving: vLLM (optionally AWQ/INT4) behind an internal gateway — FastAPI, SSE, per-namespace keys.
• Network: default-deny egress. Pull images from a mirror you control. No callback to a model vendor at request time.
• State: weights on your volume; KV-cache on the GPU nodes; application logs in your collector (OpenTelemetry).
• Ops: autoscaling on queue depth, not on a vendor’s quota page.

Vendor checklist (use this in procurement)

Ask	Pass	Fail
Where do the GPUs physically sit?	Named EU metro / your rack	“EU region” with unspoken replication
Who is the data processor for prompts?	You, or a named EU entity with a DPA	Unclear subprocessors, US parent by default
Can inference run with egress blocked?	Yes, documented	Licence check or telemetry must phone home
Who holds the weights?	Your object store / PVC	Vendor decrypts weights per request
Support access	Break-glass, logged, EU staff optional	Standing admin from outside the Union
Exit	You keep weights, traces and IaC	Model or index trapped in their control plane
Jurisdiction of the contract	EU member state	California ToS only

Cost drivers you should model before buying GPUs

Tokens are not the bill. The bill is idle GPU time, context length, concurrent sessions, and how often you re-embed a corpus. A 32B AWQ model on a pair of high-memory GPUs often beats a 70B that sits at 15% utilisation. LMXAI sizes this from a traffic sketch and a latency SLO, then adds speculative decoding or expert offload only when the numbers justify the complexity.

What LMXAI hands over

A running vLLM service, a gateway with auth and traces, a runbook for weight updates, and the completed vendor checklist for whoever hosts the boxes. You should be able to change GPU vendor later without rewriting the application.