Two-Tower Retrieval Models

Two-Tower Architecture (YouTube DNN Pattern)

  USER CONTEXT FEATURES                                ITEM METADATA FEATURES
  (User ID, Age, Watch History, Device, Location)       (Video ID, Title Embeddings, Channel, Duration)
           │                                                     │
           ▼                                                     ▼
  [ USER TOWER (MLP + Embeddings) ]                     [ ITEM TOWER (MLP + Embeddings) ]
           │                                                     │
           ▼                                                     ▼
     User Vector u [1 × 128]                               Item Vector v [1 × 128]
           │                                                     │
           └──────────────────────────┬──────────────────────────┘
                                      ▼
                            Dot Product: Score(u, v) = u · v

Real-Time Production Serving Loop

  OFFLINE BATCH PHASE:
  Run Item Tower over 10,000,000 Catalog Items ──► Pre-compute Items Matrix V [10M × 128]
  Index V in Vector DB (HNSW / Qdrant)

  ONLINE REAL-TIME INFERENCE (< 10ms):
  User Requests Feed ──► Pass User Features through User Tower ──► Output Vector u [1 × 128]
  Execute Vector Search: Top 1000 = HNSW_Search(u, V) ──► Pass 1,000 Candidates to Ranking Stage!

Loss Formulation: In-Batch Sampled Softmax

Given a mini-batch of $B$ user-item interaction pairs ${(u_1, v_1), (u_2, v_2), \dots, (u_B, v_B)}$:

$$\mathcal{L} = -\frac{1}{B} \sum_{i=1}^B \log \frac{\exp(\mathbf{u}_i \cdot \mathbf{v}i / \tau)}{\sum{j=1}^B \exp(\mathbf{u}_i \cdot \mathbf{v}_j / \tau)}$$

• Positive Pair: $(u_i, v_i)$ (User $i$ clicked Item $i$).
• Negative Pairs: Items $v_j$ ($j \neq i$) clicked by OTHER users in the mini-batch (In-Batch Negatives).
• $\tau$: Temperature scaling hyperparameter (typically $\tau = 0.05\text{--}0.1$).

Say this out loud

"Two-Tower models decouple user and item feature processing into independent neural networks, projecting both into a shared embedding space. Pre-computing item embeddings offline enables sub-10ms candidate retrieval over millions of items via ANN vector search (u · v). We train two-tower models using in-batch sampled softmax loss."

Follow-ups to expect

• What is Sampling Bias in In-Batch Negatives? Popular items appear more frequently in mini-batches, causing them to be sampled as negative items more often. Correct using Log Q Correction: $\text{Score}(u_i, v_j) = \mathbf{u}_i \cdot \mathbf{v}_j - \ln P(v_j)$.
• What is Mixed Negative Sampling? Combining In-Batch Negatives with explicit Random Uniform Negatives to ensure the model learns both popular item discrimination and hard tail item retrieval.