Two-Stage: Retrieval then Ranking
The industry-standard funnel architecture for recommending top items from millions of candidates in sub-50ms.
Production Recommender Systems (RecSys) use a multi-stage funnel architecture to balance latency and accuracy. Stage 1: Candidate Generation (Retrieval) filters millions of items down to top-100s in < 10ms using fast approximate vector search (ANN / Two-Tower models) or heuristics. Stage 2: Heavy Ranking scores and orders these 100s of candidates using complex multi-task deep learning models (DeepFM, DLRM). Stage 3: Re-ranking & Diversity applies business constraints, deduplication, position debiasing, and exploration.
The Production RecSys Funnel
ALL CATALOG ITEMS (10,000,000+)
│
▼
[ STAGE 1: CANDIDATE GENERATION / RETRIEVAL ] <-- Sub-10ms, High Recall
(Two-Tower Vector Search, Collaborative Filtering, Trending)
│
▼ (~500 Candidate Items)
[ STAGE 2: HEAVY RANKING ] <-- ~20ms, High Precision
(Deep & Cross Networks, DLRM, Multi-Task CTR/CVR)
│
▼ (Top 50 Ranked Items)
[ STAGE 3: RE-RANKING & DIVERSITY ] <-- < 5ms, Business Logic
(Deduplication, Freshness, Category Diversity, Ban Rules)
│
▼
RECOMMENDED FEED TO USER (Top 10-20 Items)
Stage 1 vs Stage 2 Comparison
| Dimension | Stage 1: Retrieval (Candidate Generation) | Stage 2: Heavy Ranking |
|---|---|---|
| Input Candidates | 1,000,000+ items | 100 – 1,000 candidates |
| Latency Budget | < 10 ms | 15 – 30 ms |
| Primary Metric | Recall@k (Did we catch good items?) | NDCG@k, AUC, CTR |
| Features Used | Static User & Item IDs, Embeddings | Cross-features, Real-time user logs, Context |
| Model Architecture | Two-Tower Vector Search, Collaborative Filtering | Multi-Task Deep Neural Nets (DLRM, DeepFM) |
| Evaluation Speed | $O(\log N)$ via ANN HNSW index | $O(K)$ forward inference passes |
Stage 1: Two-Tower Network Architecture
User Features (Age, Device, History) ──► [ User Tower ] ──► u [128-d] ──┐
├──► Cosine Similarity (u · v)
Item Features (Category, Text, Tags) ──► [ Item Tower ] ──► v [128-d] ──┘
- Item Tower Embeddings $v_i$: Computed offline once and stored in HNSW Vector DB.
- User Tower Embedding $u$: Computed online once per user query in 2ms.
- Retrieval: Query HNSW Vector DB with $u$ to fetch top-500 nearest item vectors $v_i$ using dot product $u \cdot v_i$.
Say this out loud
"Production RecSys uses a multi-stage funnel architecture to serve millions of users under strict 50ms SLAs. Stage 1 Retrieval uses Two-Tower models and ANN vector search to filter millions of items down to ~500 candidates in sub-10ms with high recall. Stage 2 Heavy Ranking scores these candidates using multi-task deep neural nets (DLRM/DeepFM) predicting CTR and conversion. Stage 3 Re-ranking applies business logic, position debiasing, and diversity filtering."
Follow-ups to expect
- What is the Candidate Retrieval Cold-Start problem? New items lack interaction history for collaborative filtering. Resolve by generating item embeddings from content metadata (text/image features) using the Item Tower.
- How do you handle position bias in Stage 2 Ranking? Users click top-ranked items simply because they are displayed first. Train the ranking model with
positionas an explicit input feature, but setposition = 0during inference evaluation.
Check yourself
Question 1 of 3
Why can production recommender systems NOT run a heavy deep learning ranking model directly across all 10,000,000 items in a catalog for every user request?