vLLM quantization: choosing between AWQ, GPTQ, FP8 and INT4

There is no universally best quantisation format for serving large language models. This guide compares AWQ, GPTQ-style formats, FP8 and INT4 in vLLM, explains what each changes and when to use it, and gives a six-step decision process based on hardware support and evaluation on the production task set. It summarises the research behind each method and treats a quantised checkpoint as a governed artefact.

Keywords quantisation · AWQ · GPTQ · FP8 · INT4 · vLLM

Quantisation is one of the most useful tools for reducing sovereign LLM infrastructure cost, but there is no universally best format.

The right choice depends on hardware, model architecture, available checkpoints, latency targets and how much quality degradation the use case can tolerate.

Current vLLM supports a broad set of quantisation approaches, including AWQ, GPTQ-style formats, FP8, INT8, INT4, compressed-tensors and newer online quantisation schemes.

Quantization maps numerical values to fewer bits. A survey of the field notes that moving from floating point to integers of four bits or less can in theory reduce memory footprint and latency by up to 16×, and that 4× to 8× is often realised in practice^[1].

For large language models, three post-training families matter. GPTQ quantizes weights in one shot using approximate second-order information; its authors report 3 to 4 bits per weight with negligible accuracy degradation on models of up to 175 billion parameters^[2]. AWQ observes that not all weights matter equally and protects about 1% of salient weights, chosen from activation statistics, without backpropagation^[3]. SmoothQuant targets 8-bit weights and activations by moving the quantization difficulty from activations to weights, reporting up to 1.56× speedup and 2× memory reduction^[4]. FP8 is a floating-point alternative: an 8-bit interchange format with two encodings whose authors report results matching 16-bit training across image and language tasks^[5]. Fine-tuning can also run on quantized weights: QLoRA trains low-rank adapters through a frozen 4-bit model^[6].

These results were measured on the authors' models, tasks and hardware. They justify trying a format. They do not replace evaluation on the production task set.

Quantisation stores weights — and in some schemes activations — at lower precision.^[1] The main benefits can include:

• lower GPU memory footprint;
• larger models on the same hardware;
• higher batch capacity;
• lower cost per request;
• in memory-bound workloads, higher throughput.

The trade-off is that reduced precision can alter model quality or kernel performance. Every quantised variant should therefore be evaluated against the same production task set.

AWQ is a popular weight-only approach for 4-bit deployment.^[3] It is widely available for open models and can be attractive when the goal is to fit a larger model into a constrained GPU envelope.

Use it when:

• a high-quality AWQ checkpoint already exists;
• the target hardware has an efficient supported kernel path;
• memory reduction matters more than maintaining full precision.

Do not assume an AWQ checkpoint is automatically faster than every higher-precision alternative. Benchmark the actual model and hardware.

GPTQ-family formats are also common in the open-model ecosystem and can provide strong memory savings.^[2]

They are useful when an existing checkpoint and supported backend align with your hardware. In a new deployment, compare them with compressed-tensors or other current vLLM-native paths rather than choosing based only on checkpoint availability.

FP8 is especially relevant on newer accelerator generations with strong native support.^[5] It can preserve more numerical range than very low-bit weight-only formats while reducing memory and bandwidth requirements relative to BF16/FP16.

Current vLLM also supports online FP8-style quantisation for supported configurations, allowing some models to be converted at load time instead of requiring a separately prepared checkpoint.

Use FP8 when:

• hardware support is strong;
• you want a moderate precision reduction with a production-friendly kernel path;
• the model and workload benefit from higher throughput or reduced memory pressure.

INT4 can produce very large memory reductions. For enterprise serving, the key question is whether that reduction creates a better system-level outcome.

A smaller footprint may let you:

• serve on fewer GPUs;
• increase concurrent sequences;
• allocate more memory to KV cache;
• keep a larger model entirely inside one node.

But the quality and performance impact is model-specific. Validate tool calling, structured output, retrieval grounding and long-context behaviour — not only generic benchmark scores.

1. Establish a BF16/FP16 baseline. Record quality, TTFT, throughput and memory.
2. Choose formats supported well by the target hardware. Avoid formats that force a slow fallback kernel.
3. Test at least two quantised variants. For example FP8 versus a 4-bit checkpoint.
4. Run domain evaluations. Include tool calls, JSON validity, multilingual prompts and RAG if those are production requirements.
5. Measure concurrency, not only single-request speed. Memory savings often create value through batching.
6. Keep the model artefact reproducible. Record quantisation method, source checkpoint, calibration data if used, and runtime version.

For regulated or high-stakes systems, changing the numerical representation of a model is a material deployment change. Treat a new quantised checkpoint as a new evaluated artefact.

The model register should record:

• base model;
• quantisation format and toolchain;
• checkpoint hash/version;
• evaluation result;
• hardware target;
• acceptance decision.

• vLLM — Quantization
• vLLM — LLM Compressor
• vLLM — Online Quantization

• Sovereign vLLM deployment + vendor checklist
• vLLM GPU sizing for sovereign inference
• Production vLLM on Kubernetes
• Sovereign AI FAQ
• GPAI vs downstream integrator