Batch vs Real-Time Inference
Choosing between scoring predictions one at a time in real time versus processing millions of predictions in bulk overnight.
The Two Approaches
When your model needs to make predictions, you have two fundamental options:
┌──────────────────────────┬──────────────────────────┐
│ BATCH INFERENCE │ REAL-TIME INFERENCE │
├──────────────────────────┼──────────────────────────┤
│ Run predictions for ALL │ Run prediction for ONE │
│ users/items at once, │ user/item when the │
│ store results in a table.│ request arrives. │
│ Scheduled (nightly/hourly│ On-demand, per request. │
│ Spark/Airflow job). │ API endpoint. │
└──────────────────────────┴──────────────────────────┘
Batch Inference
Compute predictions for every user (or every item) in a single large job, then store the results in a database or cache for fast lookup.
Example: Every night at 2 AM, run a Spark job that computes the top 100 recommended items for each of your 10 million users. Store the results in Redis. When a user opens the app, look up their precomputed recommendations instantly.
Advantages:
- Simple infrastructure. No model server needed at request time, just a database lookup.
- Can use large, expensive models because you are not constrained by per-request latency.
- Predictable compute costs (runs once per schedule).
Disadvantages:
- Predictions go stale. If a user buys a laptop at 9 AM and the batch ran at 2 AM, they will keep seeing laptop recommendations all day.
- Cannot handle new users or items that appeared after the batch ran.
- Wastes compute predicting for users who may never log in that day.
Real-Time Inference
Run the model when a request arrives, using the freshest available features.
Example: When a user opens the app, fetch their latest features from the Feature Store (including what they clicked 30 seconds ago), run the model, and return personalized results.
Advantages:
- Predictions are always fresh. Incorporates the latest user behavior.
- Handles new users and items immediately.
- Only computes predictions for active users (no wasted compute).
Disadvantages:
- Requires a model server with strict latency SLAs (typically under 100ms).
- Limited to fast models that can score within the latency budget.
- More complex infrastructure (model server, feature store, autoscaling).
The Hybrid Approach
Most production systems combine both:
- Batch: Nightly, precompute a broad set of candidate items for each user and store them.
- Real-Time: When the user opens the app, retrieve the precomputed candidates and re-score them with a lightweight real-time model that incorporates the latest session features.
This gives you the computational efficiency of batch (heavy candidate generation runs offline) with the freshness of real-time (final ranking uses live signals).
Decision Framework
| Factor | Choose Batch | Choose Real-Time |
|---|---|---|
| Feature freshness | Features change slowly (daily) | Features change every second |
| Latency requirement | No strict latency (email, reports) | Strict latency (<100ms) |
| User base | Predictable, known users | Unknown users, new visitors |
| Model complexity | Can use heavy models | Must use fast models |
| Cost priority | Predictable scheduled cost | Pay-per-request cost |
Say this out loud
Batch inference precomputes predictions for all users in advance and stores them for fast lookup. Real-time inference computes predictions on demand using the freshest features. Most production systems use a hybrid approach where batch handles heavy candidate generation and real-time handles lightweight re-scoring with live session signals.
Followups to expect
- What is near-real-time inference? Processing predictions in micro-batches every few minutes (not nightly but not per-request), using stream processing like Flink. Good for features that change hourly.
- How do you handle cold-start users in batch inference? Fall back to a default set of popular items or demographic-based recommendations since no precomputed personalization exists for unknown users.
Check yourself
When is batch inference the better choice over real-time inference?