Mixture of Experts

What is Mixture of Experts (MoE)?

In standard dense Transformers (like LLaMA 3 70B), 100 percent of parameter weights are activated for every single token processed by the model.

As models grow to 500B+ parameters, dense execution becomes prohibitively expensive.

Mixture of Experts (MoE) replaces standard dense Feed-Forward Network (FFN) blocks with multiple sparse sub-networks called Experts:

                       DENSE LAYER vs MOE LAYER

  DENSE FFN LAYER:                           SPARSE MOE LAYER:
  Token x ──► [ Dense FFN (70B Params) ]     Token x ──► [ ROUTER GATING NETWORK ]
                                                              │
                                                              ├─► Expert 1 (Math)
                                                              ├─► Expert 2 (Code) ──► Top 2
                                                              ├─► Expert 3 (French)──► Active!
                                                              └─► Expert 8 (Logic)

How MoE Works: The Router Network

Given a token vector $x$, a Router / Gating Network computes Softmax routing probabilities across $E$ candidate expert networks:

$$G(x) = \text{Softmax}\left( \text{TopK}(x \cdot W_g, k) \right)$$

• $W_g \in \mathbb{R}^{d_{\text{model}} \times E}$: Router weight matrix.
• $\text{TopK}(\cdot, k)$: Keeps the $k$ highest scoring experts (typically $k = 2$) and sets all other expert scores to $-\infty$.

The output of the MoE layer is the weighted sum of the selected $k$ Experts:

$$y = \sum_{i \in \text{TopK}} G(x)_i \cdot E_i(x)$$

Real-World Example: Mixtral 8x7B (Mistral AI)

• Total Parameters: $47$ Billion parameters across 8 Experts.
• Active Parameters per Token: Only $13$ Billion active parameters!
• Performance: Matches LLaMA 2 70B accuracy while executing at 6x faster inference speed!

MoE Architectural Challenges

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. ROUTER LACK OF BALANCE│ 2. VRAM RAM CAPACITY     │ 3. HIGHWAY DISPATCH      │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Router prefers 1 or 2    │ Full 47B model weights   │ Routing different tokens │
│ favorite experts. Requires│ MUST stay loaded in GPU  │ to different GPUs causes │
│ Auxiliary Load Loss!     │ VRAM memory at all times.│ inter-GPU network latency│
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Auxiliary Load Balancing Loss

If left unconstrained, the router network quickly develops a preference for 1 or 2 favorite experts, routing 99 percent of tokens to them while leaving remaining experts untrained (Expert Collapse).

MoE models add an Auxiliary Load Balancing Loss to force tokens to distribute evenly across all $E$ experts during training.

2. VRAM Memory vs Active FLOPs

MoE cuts compute FLOPs ($13\text{B active}$), but does NOT cut VRAM memory requirements!

Because any token might be routed to any of the 8 experts at any step, all 47B parameters must remain loaded in GPU VRAM.

Say this out loud

Mixture of Experts replaces dense feed forward layers with multiple sparse expert sub networks. A Router network evaluates input tokens dynamically, routing each token to the Top K most relevant experts like Top 2 out of 8. MoE scales total parameter capacity to hundreds of billions while activating a fraction per token, achieving high accuracy at low compute FLOP costs.

Followups to expect

1. What is DeepSeek-V2 / DeepSeek-V3 Fine-Grained MoE? Uses a larger number of smaller experts (e.g. 64 or 256 fine-grained experts with Top-8 routing) alongside Shared Experts that remain active for all tokens, improving expert specialization.
2. What is Expert Parallelism (EP)? Distributing individual Expert networks across different GPUs in a cluster, requiring all-to-all communication collectives to route token vectors between GPUs during forward passes.