vLLM GPU sizing: size the workload, not just the model

Parameter count is only a lower bound for GPU memory. This article sizes vLLM deployments from traffic, latency objectives and the distribution of context lengths, covering model footprint, key-value cache, concurrency, parallelism strategy, quantisation and spare capacity under sovereignty constraints. It rests on the systems research behind vLLM, which shows that cache management and scheduling determine throughput.

Keywords vLLM · GPU sizing · KV cache · concurrency · quantisation · sovereign inference

The fastest way to overspend on sovereign LLM infrastructure is to size a cluster from parameter count alone.

A model that technically fits in GPU memory can still miss production latency targets once long contexts, concurrent users, KV cache, retrieval payloads and tool calls enter the workload. Conversely, a heavily quantised model on fewer GPUs can deliver better economics when traffic is moderate and the quality trade-off is acceptable.

A useful sizing exercise starts with the workload and works backwards to the hardware.

Sizing follows from how serving systems use memory. For each request the server keeps a key-value cache that is large and grows and shrinks dynamically; managed poorly, it wastes memory and limits batch size. vLLM's PagedAttention addresses this and reports 2–4× higher throughput at the same latency than earlier systems, with larger gains for longer sequences and larger models^[1]. Scheduling matters as much as memory: with request-level batching, finished requests cannot return and new ones must wait, the problem iteration-level scheduling was proposed to solve^[2]. This is why concurrency and the distribution of context lengths, not parameter count, drive the hardware bill.

Quantization changes the footprint side of the equation. A survey notes that moving to integers of four bits or less can in theory reduce memory footprint and latency by up to 16×, with 4× to 8× often realised in practice^[3]; AWQ is a common 4-bit weight-only method for language models^[4]. Those figures come from the papers' own settings and need re-measuring on the target model and hardware.

A rough lower bound for weight memory is:

model parameters × bits per weight ÷ 8

That is only the starting point. Real deployments also need memory for KV cache, runtime buffers, CUDA graphs or compilation artefacts, and operational headroom.

Quantisation can reduce the weight footprint substantially, but the effect on throughput and quality depends on the model, hardware and quantisation format.

Long context is expensive because KV cache grows with active sequences and token length. If an application sends large documents or long conversation histories, the cluster may become KV-cache-bound before it becomes compute-bound.

That is why a 200k-token model limit should not automatically become a 200k-token production default.

Measure the actual distribution of input length, output length and concurrent requests. Then define guardrails: maximum context, truncation or summarisation policy, and retrieval limits.

For enterprise systems, the important question is usually not "What is the maximum tokens per second?" but "How many real users can the service support while meeting our latency SLO?"

Track at least:

• time to first token (TTFT);
• inter-token latency / generation speed;
• request queue time;
• concurrent sequences;
• prompt and completion token distributions;
• GPU utilisation and memory pressure;
• error and preemption rates.

vLLM exposes production metrics through its /metrics endpoint, which makes these signals available to Prometheus-compatible monitoring.

Current vLLM guidance is straightforward:

• if the model fits comfortably on one GPU, avoid distributed complexity;
• if it needs multiple GPUs in one node, tensor parallelism is the normal next step;
• if it exceeds one node, combine tensor and pipeline parallelism;
• for multiple independent replicas or traffic scaling, data parallel deployment becomes relevant.

For MoE models, expert parallelism can also matter. The right topology depends on whether the bottleneck sits in attention, expert layers, interconnect or memory.

INT4, INT8 and FP8 can reduce memory requirements and sometimes improve throughput, but they should be treated as deployment variants that must pass the same evaluation set as the full-precision baseline.

For example, a 32B model in a suitable low-bit format may fit a much smaller deployment envelope than its BF16 version. That can change the entire business case when the workload does not need a larger model.

Before procurement, model three traffic levels:

Baseline — normal office-hour usage.

Burst — a meeting, campaign or internal workflow causes a short concurrency spike.

Failure mode — one replica is unavailable and remaining capacity must absorb critical requests.

For each level, estimate input tokens, output tokens, concurrency and target TTFT. Benchmark the actual model with those shapes rather than relying on a generic leaderboard.

A sovereign deployment may intentionally trade some elasticity for control. If the environment has no public-cloud burst capacity, the cluster needs enough reserve to survive maintenance and node failure.

The architecture should also account for:

• model-storage capacity and update windows;
• local container/image registry;
• observability retention;
• default-deny egress requirements;
• failover strategy;
• spare capacity for evaluation or canary models.

We size sovereign inference from traffic + latency SLO + context distribution + model quality target, then choose the smallest architecture that passes the acceptance tests.

That avoids both common mistakes: buying a cluster that is too small for real concurrency, or buying expensive GPUs that remain mostly idle.

• vLLM — Parallelism and Scaling
• vLLM — Production Metrics
• LMXAI — Sovereign vLLM deployment

• Sovereign vLLM deployment + vendor checklist
• vLLM quantization: AWQ, GPTQ, FP8 and INT4
• Production vLLM on Kubernetes
• Sovereign AI FAQ
• NL: Soevereine LLM-deployment