Latency vs Throughput vs Cost
Understanding the three way tradeoff between response speed, request throughput, and compute cost in ML serving.
The Three Way Tradeoff
Every ML serving system operates within a triangle of competing constraints:
LATENCY
(Response Speed per Request)
╱╲
╱ ╲
╱ ╲
╱ ╲
THROUGHPUT ◄──────────────► COST
(Requests per Second) (Hardware $)
You can optimize any two, but the third usually suffers:
| Want Low Latency + High Throughput? | You need expensive hardware (more GPUs, faster networking). Cost goes up. |
|---|---|
| Want Low Latency + Low Cost? | You get limited throughput because cheap hardware cannot handle many concurrent requests. |
| Want High Throughput + Low Cost? | Use large batches on fewer GPUs, but individual request latency increases. |
Latency: How Fast Is One Request?
Latency is the time from when a request arrives to when the response is sent back. Measured in milliseconds.
Key contributors to inference latency:
- Network Round Trip: Time for the request to travel to the server and back (~1 to 10ms).
- Feature Lookup: Fetching user/item features from the Feature Store (~2 to 10ms).
- Model Inference: The actual neural network forward pass (~5 to 200ms depending on model size).
- Post-Processing: Applying business rules, formatting the response (~1 to 5ms).
For real-time applications (search, ads, chat), total latency budgets are strict: P99 under 100ms is common.
Throughput: How Many Requests Per Second?
Throughput is the total number of requests the system processes per second (QPS or RPS).
Techniques to increase throughput:
- Batching: Collect multiple incoming requests and process them together in a single GPU forward pass. GPUs are most efficient when processing batches.
- Horizontal Scaling: Add more server replicas behind a load balancer.
- Asynchronous Processing: Use request queues so the server is never idle waiting.
Cost: What Does It Take To Run?
GPU inference is expensive. A single A100 GPU costs roughly $2 to $3 per hour in the cloud.
Cost reduction techniques:
- Quantization: Run INT8 instead of FP32, cutting memory and compute by 4x.
- Smaller Models: Use distilled or pruned models that maintain 95% of accuracy at 10x lower cost.
- Spot/Preemptible Instances: Use cheaper interruptible cloud instances for batch workloads.
- Autoscaling: Scale replicas up during peak traffic and down during quiet periods to avoid paying for idle GPUs.
Making The Right Choice
| Use Case | Priority | Typical Setup |
|---|---|---|
| Real-time search/ads | Low Latency | Small optimized models, no batching, TensorRT |
| Overnight batch recommendations | High Throughput, Low Cost | Large batches, spot instances, bigger models |
| Chat assistant | Moderate Latency, Moderate Cost | Streaming tokens, medium-sized model, autoscaling |
Say this out loud
ML serving involves a three way tradeoff between latency, throughput, and cost. Low latency requires fast models and dedicated hardware. High throughput uses batching and horizontal scaling. Low cost uses quantization, smaller models, and autoscaling. The right balance depends on whether the application is real-time, interactive, or batch.
Followups to expect
- What is P99 latency and why does it matter more than average latency? P99 is the latency experienced by the slowest 1% of requests. A good average can hide terrible tail latency that affects many users.
- How does dynamic batching work? The server accumulates incoming requests for a short window (e.g. 5ms), then processes them as a batch. This balances latency (short wait window) with throughput (batch efficiency).
Check yourself
What happens to per-request latency when you increase batch size to improve GPU throughput?