LoRA & Parameter-Efficient Fine-Tuning

What is Parameter-Efficient Fine-Tuning (PEFT)?

Fine-tuning a 70B parameter model using standard Full Fine-Tuning requires updating all 70 Billion weights.

Storing 70B model weights, 70B gradients, and 140B Adam optimizer states requires over 800 Gigabytes of GPU VRAM (a massive $10\times$ GPU cluster!).

Parameter-Efficient Fine-Tuning (PEFT) techniques freeze the base model and train only a tiny fraction ($<1%$) of parameters.

Low-Rank Adaptation (LoRA - Edward Hu et al., 2021 / Microsoft) is the industry standard PEFT method.

                            LORA ARCHITECTURE
                     Input Vector x [1 x d]
                               │
            ┌──────────────────┴──────────────────┐
            │                                     │
            ▼                                     ▼
     [ FROZEN BASE WEIGHT W ]              [ TRAINABLE MATRIX A ]  (Rank r = 8)
        [d x k] (No Grad!)                        [d x r]
            │                                     │
            │                                     ▼
            │                              [ TRAINABLE MATRIX B ]
            │                                     [r x k]
            │                                     │
            ▼                                     ▼
           (+) ◄───────────── Scaling (α / r) ────┘
            │
            ▼
     Output Vector y [1 x k]

The Low-Rank Matrix Factorization Trick

Aghajanyan et al. (2020) showed that weight updates $\Delta W$ during fine-tuning have a low intrinsic rank.

Instead of updating a full matrix $\Delta W \in \mathbb{R}^{d \times k}$:

LoRA decomposes $\Delta W$ into two low-rank matrices $A$ and $B$:

$$\Delta W = B \cdot A, \quad \text{where } B \in \mathbb{R}^{d \times r}, ; A \in \mathbb{R}^{r \times k}, ; \text{with } r \ll \min(d, k)$$

Rank $r$ is typically set to $r = 8$ or $r = 16$.

Parameter Reduction Example

Suppose $d = 4096$ and $k = 4096$:

• Full Fine-Tuning: $4096 \times 4096 = \mathbf{16,777,216 \text{ Parameters}}$.
• LoRA ($r = 8$): $(4096 \times 8) + (8 \times 4096) = \mathbf{65,536 \text{ Parameters}}$ ($99.6%$ reduction!).

Forward Pass Math

$$y = W x + \Delta W x = W x + \frac{\alpha}{r} (B A) x$$

• $\alpha$ (alpha): Constant scaling hyperparameter (typically $\alpha = 2r$ or $\alpha = 16$).
• Initialization: $A \sim \mathcal{N}(0, \sigma^2)$ and $B = 0$. Because $B = 0$, $\Delta W = 0$ at step 0!

Zero Inference Latency Overhead

When deploying to production, you do NOT need to compute two matrix multiplications during inference!

Simply fold the LoRA weights back into the base model:

$$W_{\text{merged}} = W + \frac{\alpha}{r} (B A)$$

The merged model has the exact same architecture as the base model, adding zero inference latency!

Say this out loud

LoRA freezes base model weights and injects trainable low rank decomposition matrices A and B alongside attention layers (W_new = W + B * A). This reduces trainable parameter counts by 99 percent and VRAM usage by 3 times. Before deployment, LoRA weights can be folded directly back into base weights for zero inference latency overhead.

Followups to expect

1. Which layers should LoRA be applied to? Original LoRA applied adapters only to Query and Value matrices ($W_q, W_v$). Modern practice applies LoRA to all linear layers ($W_q, W_k, W_v, W_o, W_{\text{gate}}, W_{\text{up}}, W_{\text{down}}$) for maximum fine-tuning performance.
2. Can you swap LoRA adapters dynamically at runtime? Yes! Because base weights are frozen, a single serving GPU can load 1 base model into VRAM and dynamically swap small 50MB LoRA adapters (Coding LoRA, Legal LoRA, Medical LoRA) per user request.