Production vLLM on Kubernetes: a sovereign deployment pattern

Running vllm serve is easy. Operating it as an enterprise service is the real engineering work.

A production vLLM deployment needs model lifecycle controls, authentication, observability, safe rollout, capacity management and failure handling. In a sovereign environment, it also needs explicit control over egress, model storage and privileged access.

Reference architecture

A practical pattern contains five layers:

1. GPU worker layer — vLLM pods or workloads on dedicated accelerator nodes.
2. Internal gateway — authentication, rate limits, request policy, streaming and model routing.
3. Model storage — object storage or persistent volumes controlled by the organisation.
4. Observability — vLLM metrics, gateway traces, application logs and alerting.
5. Control plane — Kubernetes scheduling, secrets, rollout strategy and infrastructure policy.

The inference pods should not need arbitrary internet access to answer a request.

Kubernetes deployment basics

vLLM maintains Kubernetes deployment guidance for CPU and GPU environments and can also be integrated through several ecosystem stacks.

For a straightforward sovereign design, keep the first version boring:

• dedicated GPU node pool;
• explicit resource requests and limits;
• pinned model and container versions;
• readiness/startup checks that understand model-load time;
• internal-only service exposure;
• pod disruption and rollout settings that protect capacity.

The goal is predictable operation, not maximum platform complexity.

Put a gateway in front of vLLM

Do not expose the model server directly to every internal application.

An internal gateway can provide:

• API keys or workload identity;
• per-team quotas;
• audit logging;
• request IDs and trace propagation;
• model aliases and controlled version changes;
• streaming policy;
• prompt-size and context limits;
• optional DLP or content controls.

This separates inference lifecycle from application lifecycle. Teams can change models without rewriting every client.

Use default-deny egress

Sovereignty is easier to reason about when inference can run without outbound internet access.

A typical pattern is:

• mirror container images internally;
• pre-stage model weights in controlled storage;
• disable unnecessary telemetry;
• permit only explicitly approved internal destinations;
• use a separate controlled workflow for model updates.

This makes the data path auditable and reduces dependence on an external model vendor at request time.

Monitor the serving layer

vLLM exposes production metrics through /metrics. Combine them with gateway and Kubernetes telemetry.

Watch:

• TTFT and generation latency;
• request queueing;
• token throughput;
• GPU memory pressure and utilisation;
• active sequence count;
• model-load failures;
• pod restarts and OOM events;
• error rate by model version.

Autoscaling should follow a signal related to real saturation — for example queue depth or concurrency — rather than only raw CPU utilisation.

Roll models like software releases

A model update can change output style, tool reliability, JSON validity or safety behaviour even when the API contract remains identical.

Use a controlled rollout:

1. register the new checkpoint;
2. run offline evaluations;
3. deploy a canary replica;
4. compare latency and task metrics;
5. shift a small percentage of traffic;
6. promote or roll back;
7. retain the previous artefact until acceptance is complete.

For quantised models, record both the base model and the quantisation pipeline.

High availability and spare capacity

If a deployment cannot burst into a public cloud, maintenance reserve matters.

Design for at least the failures that are plausible in your environment: one pod, one GPU, one node, a storage interruption or a bad model rollout. The right redundancy level depends on the business criticality of the service.

Primary sources

• vLLM — Kubernetes deployment
• vLLM — Production Metrics
• vLLM — Parallelism and Scaling
• LMXAI — Sovereign vLLM deployment

Related reading

• Sovereign vLLM deployment + vendor checklist
• vLLM GPU sizing for sovereign inference
• vLLM quantization: AWQ, GPTQ, FP8 and INT4
• Sovereign AI FAQ
• NL: Soevereine LLM-deployment