Production vLLM on Kubernetes: a sovereign deployment pattern

Starting a model server is easy; operating it as an enterprise service is the engineering work. This article gives a reference architecture for vLLM on Kubernetes without arbitrary egress: GPU workers, an internal gateway, default-deny networking, monitoring, model rollouts treated like software releases, and spare capacity without public-cloud burst. It connects those practices to research on container orchestration, LLM serving and production readiness.

Keywords vLLM · Kubernetes · model serving · egress control · monitoring · rollouts

Running vllm serve is easy. Operating it as an enterprise service is the real engineering work.

A production vLLM deployment needs model lifecycle controls, authentication, observability, safe rollout, capacity management and failure handling. In a sovereign environment, it also needs explicit control over egress, model storage and privileged access.

Kubernetes carries design lessons from more than ten years of managing containers at scale at Google across three successive systems^[1]; running model servers as ordinary, declaratively managed workloads is the reason to use it here. The serving layer itself comes from systems research: vLLM's PagedAttention manages the key-value cache like virtual memory and reports 2–4× higher throughput than earlier systems at the same latency^[2]. For the network, a default-deny posture starts from the zero-trust architecture defined by NIST^[3].

Research on ML operations adds two cautions. ML systems accumulate hidden technical debt beyond the model code^[4], and production readiness is better judged against an explicit rubric of tests and monitoring than by a successful demo^[5]. The rollout and monitoring practices below apply those ideas to model serving.

A practical pattern contains five layers:

1. GPU worker layer — vLLM pods or workloads on dedicated accelerator nodes.
2. Internal gateway — authentication, rate limits, request policy, streaming and model routing.
3. Model storage — object storage or persistent volumes controlled by the organisation.
4. Observability — vLLM metrics, gateway traces, application logs and alerting.
5. Control plane — Kubernetes scheduling, secrets, rollout strategy and infrastructure policy.

The inference pods should not need arbitrary internet access to answer a request.

vLLM maintains Kubernetes deployment guidance for CPU and GPU environments and can also be integrated through several ecosystem stacks.

For a straightforward sovereign design, keep the first version boring:

• dedicated GPU node pool;
• explicit resource requests and limits;
• pinned model and container versions;
• readiness/startup checks that understand model-load time;
• internal-only service exposure;
• pod disruption and rollout settings that protect capacity.

The goal is predictable operation, not maximum platform complexity.

Do not expose the model server directly to every internal application.

An internal gateway can provide:

• API keys or workload identity;
• per-team quotas;
• audit logging;
• request IDs and trace propagation;
• model aliases and controlled version changes;
• streaming policy;
• prompt-size and context limits;
• optional DLP or content controls.

This separates inference lifecycle from application lifecycle. Teams can change models without rewriting every client.

Sovereignty is easier to reason about when inference can run without outbound internet access.

A typical pattern is:

• mirror container images internally;
• pre-stage model weights in controlled storage;
• disable unnecessary telemetry;
• permit only explicitly approved internal destinations;
• use a separate controlled workflow for model updates.

This makes the data path auditable and reduces dependence on an external model vendor at request time.

vLLM exposes production metrics through /metrics. Combine them with gateway and Kubernetes telemetry.

Watch:

• TTFT and generation latency;
• request queueing;
• token throughput;
• GPU memory pressure and utilisation;
• active sequence count;
• model-load failures;
• pod restarts and OOM events;
• error rate by model version.

Autoscaling should follow a signal related to real saturation — for example queue depth or concurrency — rather than only raw CPU utilisation.

A model update can change output style, tool reliability, JSON validity or safety behaviour even when the API contract remains identical.

Use a controlled rollout:

1. register the new checkpoint;
2. run offline evaluations;
3. deploy a canary replica;
4. compare latency and task metrics;
5. shift a small percentage of traffic;
6. promote or roll back;
7. retain the previous artefact until acceptance is complete.

For quantised models, record both the base model and the quantisation pipeline.

If a deployment cannot burst into a public cloud, maintenance reserve matters.

Design for at least the failures that are plausible in your environment: one pod, one GPU, one node, a storage interruption or a bad model rollout. The right redundancy level depends on the business criticality of the service.

• vLLM — Kubernetes deployment
• vLLM — Production Metrics
• vLLM — Parallelism and Scaling
• LMXAI — Sovereign vLLM deployment

• Sovereign vLLM deployment + vendor checklist
• vLLM GPU sizing for sovereign inference
• vLLM quantization: AWQ, GPTQ, FP8 and INT4
• Sovereign AI FAQ
• NL: Soevereine LLM-deployment