GGUF, AWQ & LLM Quantization

Comparing post-training 4-bit and 8-bit weight quantization formats for CPU, Apple Silicon, and GPU inference.

🔴 advanced5 min readefficiencyinference

LLM Weight Quantization converts 16-bit floating-point weights (FP16/BF16) into lower-precision integer or float representations (INT8, INT4, FP8) to shrink VRAM memory footprint and accelerate inference. GGUF (llama.cpp) is optimized for CPU and Apple Silicon unified memory using k-quant block quantization. GPTQ (One-shot Post-Training Quantization) uses second-order Hessian information for 4-bit GPU inference. AWQ (Activation-aware Weight Quantization) protects the 1% most critical weight channels based on activation magnitudes, achieving superior 4-bit accuracy on GPUs.

The Quantization Ecosystem

                               QUANTIZATION ecosystem
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│  1. GGUF (llama.cpp)     │     2. AWQ (Activation)  │     3. GPTQ (Hessian)    │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ - Best For: CPU, Mac,    │ - Best For: Production   │ - Best For: Standard 4-bit│
│   Ollama local inference │   GPU vLLM serving       │   vLLM / TGI GPU serving │
│ - Single binary file     │ - Protects 1% salient    │ - One-shot second-order  │
│ - Mixed k-quants Q4_K_M  │   activation channels    │   Hessian matrix solver  │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Comparative Format Matrix

Format / Method	Hardware Target	Best Serving Framework	Accuracy / Perplexity	Key Mechanism
GGUF	CPU, Apple Silicon (Metal), Local	llama.cpp, Ollama, LM Studio	Excellent (Q4_K_M, Q5_K_M)	Single binary format with embedded metadata & k-quants
AWQ	NVIDIA / AMD GPUs	vLLM, SGLang, TGI	Highest 4-bit Accuracy	Scales 1% salient weight channels based on activation magnitudes
GPTQ	NVIDIA / AMD GPUs	vLLM, AutoGPTQ, TGI	High 4-bit Accuracy	Minimizes mean squared quantization error using inverse Hessian matrix
FP8 (E4M3)	NVIDIA Hopper / Ada (H100, L40S)	vLLM, TensorRT-LLM	Near-Lossless (99.9%)	Native 8-bit floating point hardware execution on Tensor Cores

AWQ: Activation-Aware Weight Quantization

Standard Post-Training Quantization (PTQ) quantizes all weights $W$ uniformly based solely on weight values.

AWQ observes that weight significance depends on activation magnitudes $X$:

$$S_j = \text{Mean}(|X_j|) \quad \text{for feature channel } j$$

  Activations X ──► Measure Channel Magnitudes S_j ──► Identify Top 1% Salient Channels
                                                                │
                                                                ▼
                                                   Protect Top 1% Channels in FP16!
                                                   Quantize Remaining 99% to INT4

By protecting the 1% most active channels from quantization distortion, AWQ maintains near-FP16 perplexity while cutting VRAM size by 4x.

Say this out loud

"LLM Quantization reduces 16-bit FP16 weights to 4-bit or 8-bit representations. GGUF is optimized for CPU and Apple Silicon local inference via llama.cpp. For GPU serving in vLLM, AWQ achieves superior 4-bit accuracy by protecting the 1% most salient weight channels based on activation magnitudes, while FP8 delivers near-lossless 8-bit performance on NVIDIA Hopper GPUs."

Follow-ups to expect

What is k-quant in GGUF (e.g. Q4_K_M)? Mixed-precision quantization where critical layers (attention projections and output heads) are kept at higher precision (5-bit or 6-bit) while less sensitive FFN layers are quantized to 4-bit, balancing size and perplexity.
What is the difference between PTQ and QAT? Post-Training Quantization (PTQ - AWQ/GPTQ) quantizes a pre-trained model offline in minutes without retraining. Quantization-Aware Training (QAT) simulates quantization errors during fine-tuning backpropagation, achieving higher accuracy at the cost of full training runs.

Check yourself

Question 1 of 3

Why does AWQ (Activation-aware Weight Quantization) achieve better accuracy than standard GPTQ at 4-bit precision?

More in LLMs & GenAI

See all →

Pretraining → SFT → RLHF5 min Fine-Tune vs RAG vs Prompt: Choosing5 min Retrieval-Augmented Generation5 min