Parameter count is a lower bound. Size from workload, KV cache and latency SLO, then buy the smallest cluster that passes.
The fastest way to overspend on sovereign LLM infrastructure is to size a cluster from parameter count alone.
A model that technically fits in GPU memory can still miss production latency targets once long contexts, concurrent users, KV cache, retrieval payloads and tool calls enter the workload. Conversely, a heavily quantised model on fewer GPUs can deliver better economics when traffic is moderate and the quality trade-off is acceptable.
A useful sizing exercise starts with the workload and works backwards to the hardware.
A rough lower bound for weight memory is:
model parameters × bits per weight ÷ 8
That is only the starting point. Real deployments also need memory for KV cache, runtime buffers, CUDA graphs or compilation artefacts, and operational headroom.
Quantisation can reduce the weight footprint substantially, but the effect on throughput and quality depends on the model, hardware and quantisation format.
Long context is expensive because KV cache grows with active sequences and token length. If an application sends large documents or long conversation histories, the cluster may become KV-cache-bound before it becomes compute-bound.
That is why a 200k-token model limit should not automatically become a 200k-token production default.
Measure the actual distribution of input length, output length and concurrent requests. Then define guardrails: maximum context, truncation or summarisation policy, and retrieval limits.
For enterprise systems, the important question is usually not "What is the maximum tokens per second?" but "How many real users can the service support while meeting our latency SLO?"
Track at least:
vLLM exposes production metrics through its /metrics endpoint, which makes these signals available to Prometheus-compatible monitoring.
Current vLLM guidance is straightforward:
For MoE models, expert parallelism can also matter. The right topology depends on whether the bottleneck sits in attention, expert layers, interconnect or memory.
INT4, INT8 and FP8 can reduce memory requirements and sometimes improve throughput, but they should be treated as deployment variants that must pass the same evaluation set as the full-precision baseline.
For example, a 32B model in a suitable low-bit format may fit a much smaller deployment envelope than its BF16 version. That can change the entire business case when the workload does not need a larger model.
Before procurement, model three traffic levels:
Baseline — normal office-hour usage.
Burst — a meeting, campaign or internal workflow causes a short concurrency spike.
Failure mode — one replica is unavailable and remaining capacity must absorb critical requests.
For each level, estimate input tokens, output tokens, concurrency and target TTFT. Benchmark the actual model with those shapes rather than relying on a generic leaderboard.
A sovereign deployment may intentionally trade some elasticity for control. If the environment has no public-cloud burst capacity, the cluster needs enough reserve to survive maintenance and node failure.
The architecture should also account for:
We size sovereign inference from traffic + latency SLO + context distribution + model quality target, then choose the smallest architecture that passes the acceptance tests.
That avoids both common mistakes: buying a cluster that is too small for real concurrency, or buying expensive GPUs that remain mostly idle.
Next step. If you want this to ship, LMXAI scopes the integration as a system — not a workshop series.
