Pruning & Quantization

Deploying Models to Edge Hardware

Deep learning models are often trained on high-power GPU clusters, but must be deployed to edge devices (mobile phones, embedded cameras, real-time API servers) with strict VRAM, latency, and power constraints.

Two complementary techniques dominate model compression:

┌──────────────────────────┬──────────────────────────┐
│ 1. PRUNING               │ 2. QUANTIZATION          │
├──────────────────────────┼──────────────────────────┤
│ Zeroes out or removes    │ Reduces numerical bit    │
│ uninformative weight     │ precision of weights and │
│ parameters.              │ activations (FP32 -> INT8)│
│ Reduces parameter count. │ Reduces RAM & latency!   │
└──────────────────────────┴──────────────────────────┘

1. Network Pruning

Pruning removes unnecessary weights based on magnitude ($|w_i| \approx 0$) or saliency gradient criteria.

┌──────────────────────────┬──────────────────────────┐
│ UNSTRUCTURED PRUNING     │ STRUCTURED PRUNING       │
├──────────────────────────┼──────────────────────────┤
│ Zeroes out individual    │ Removes ENTIRE channels, │
│ weight elements.         │ rows, or attention heads.│
│ High sparsity (90%), but │ Directly reduces tensor  │
│ requires specialized sparse│ dimensions. Accelerates │
│ GPU hardware kernels!    │ standard GPUs instantly! │
└──────────────────────────┴──────────────────────────┘

The Lottery Ticket Hypothesis (Frankle & Carbin, 2018)

States that dense, randomly initialized neural networks contain small sparse sub-networks (winning tickets) that, when trained in isolation from initial weights, can match the accuracy of the full dense model!

2. Model Quantization

Quantization maps continuous high-precision floating point numbers (FP32 / FP16) to discrete low-bit integer values (INT8 / INT4):

$$r = S \cdot (q - Z)$$

• $r$: Real continuous float value.
• $q$: Quantized integer value (e.g. $0 \dots 255$ for INT8).
• $S$: Quantization Scale factor.
• $Z$: Zero-point integer shift offset.

┌──────────────────────────┬──────────────────────────┐
│ POST-TRAINING QUANT (PTQ)│ QUANT-AWARE TRAIN (QAT)  │
├──────────────────────────┼──────────────────────────┤
│ Quantizes pretrained     │ Inserts Fake Quantization│
│ weights after training   │ nodes during training.   │
│ completes. Fast, zero    │ Fine-tunes weights to    │
│ training compute cost.   │ adapt to low-bit noise.  │
│ Good for INT8.           │ Mandatory for INT4/INT2! │
└──────────────────────────┴──────────────────────────┘

Memory & Latency Savings

Converting a 70B LLM from 16-bit BFloat16 to 4-bit INT4:

• VRAM Memory Footprint: Drops from $140\text{GB}$ down to $35\text{GB}$ ($75%$ reduction!).
• Inference Speed: Enables 4-bit quantization engines (AWQ, GGUF, vLLM) to load full 70B models onto single 48GB consumer GPUs.

Say this out loud

Pruning zeroes out redundant weight parameters based on magnitude, using structured pruning to remove entire channels or unstructured pruning for sparse kernels. Quantization converts high precision floating point weights into low bit integers like INT8 or INT4, using Post Training Quantization for fast conversion or Quantization Aware Training to maintain accuracy at low bit limits.

Followups to expect

1. What is SmoothQuant (Xiao et al., 2023)? An 8-bit quantization technique for LLMs that smooths out activation outliers across channels, enabling INT8 quantization for both weights and activations ($W8A8$).
2. What is AWQ (Activation-aware Weight Quantization)? Protects the top 1% most salient weight channels (identified via activation magnitudes) in 16-bit precision while quantizing remaining weights to 4-bit, retaining model accuracy.