QLoRA & 4-bit Fine-Tuning

Fine-Tuning 70B Models on a Single Consumer GPU

Standard LoRA fine-tuning on a 70B parameter model requires keeping the 70B base model in 16-bit BFloat16 precision ($140\text{GB VRAM}$ for base weights alone).

This required expensive enterprise multi-GPU nodes.

QLoRA (Tim Dettmers et al., 2023 / UW) made fine-tuning 70B models possible on a single 48GB GPU (like an NVIDIA A6000 or RTX 6000) by combining three innovations:

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. NF4 DATA TYPE         │ 2. DOUBLE QUANTIZATION   │ 3. PAGED OPTIMIZERS      │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Quantizes base weights to│ Quantizes quantization   │ Uses CUDA Unified Memory │
│ 4-bit NormalFloat4 (NF4) │ scale constants from     │ to page memory spikes to │
│ optimal quantile data.   │ FP32 down to FP8.        │ CPU RAM without OOMs.    │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

                     QLORA FINE-TUNING PIPELINE
  FROZEN BASE WEIGHTS (Quantized to 4-bit NF4!)  ──► De-quantize on-the-fly to BF16
                                                              │
                                                              ▼
  16-BIT BFLOAT16 LORA ADAPTERS (Trainable Parameters!) ──► Forward / Backward Pass!

Innovation 1: NormalFloat 4 (NF4) Data Type

Pretrained neural network weight parameters are normally distributed ($\mathcal{N}(0, \sigma^2)$).

Standard 4-bit integer quantization (INT4) uses equally spaced bin steps, which assigns too many bins to rare tail values and too few bins to dense central values.

NormalFloat 4 (NF4) builds a 4-bit data type where each of the 16 bin levels represents an equal probability quantile under a standard normal distribution:

$$q_i = \frac{1}{2} \left( Q_X\left( \frac{i}{2^k} \right) + Q_X\left( \frac{i+1}{2^k} \right) \right)$$

NF4 is information-theoretically optimal for normally distributed weights, preserving higher model accuracy than standard 4-bit integer quantization.

Innovation 2: Double Quantization (DQ)

Block quantization converts FP32 weights into 4-bit parameters using a scaling factor $c_1$:

$$W^{\text{FP32}} \approx c_1 \cdot W^{\text{NF4}}$$

Storing scaling factor $c_1$ for every 64 weights adds an extra $32 / 64 = 0.5$ bits per parameter.

Double Quantization quantizes the scaling factors $c_1$ themselves from 32-bit float to 8-bit float with a second scaling factor $c_2$:

This reduces quantization scale memory from $0.5$ bits/param down to $0.127$ bits/param (saving $\sim 3\text{GB VRAM}$ on a 70B model!).

Innovation 3: Paged Optimizers

During long sequence training, sudden VRAM memory bursts can trigger Out of Memory (CUDA out of memory) crashes.

QLoRA uses Paged Optimizers backed by CUDA Unified Memory:

• When GPU VRAM fills up during a backward step, optimizer states for LoRA parameters are automatically paged out to CPU System RAM.
• As GPU VRAM frees up, states are paged back to GPU VRAM seamlessly without crashing training!

Say this out loud

QLoRA enables fine tuning 70B parameter models on a single 48GB GPU by quantizing frozen base model weights into 4 bit NormalFloat (NF4). It uses Double Quantization to compress scaling constants and Paged Optimizers to page VRAM spikes out to CPU RAM, achieving full 16 bit fine tuning accuracy at a fraction of the VRAM footprint.

Followups to expect

1. Does QLoRA slow down training speed compared to standard LoRA? Yes, QLoRA is roughly 20 to 30 percent slower than 16-bit LoRA because base weights must be de-quantized from 4-bit NF4 to 16-bit BFloat16 on the fly during every forward and backward pass.
2. Can QLoRA be merged back into 16-bit base models? Yes! De-quantize the 4-bit base weights back to 16-bit BFloat16, add the 16-bit LoRA adapter weights, and save a full 16-bit merged model checkpoint.