Autoscaling Inference Workloads

The Need for Dynamic Scaling

User traffic follows daily cycles:

• Peak Hours (12 PM to 8 PM): High traffic requiring 50 GPU server instances.
• Off-Peak Hours (2 AM to 6 AM): Low traffic requiring only 2 GPU server instances.

If you provision 50 instances permanently, you waste thousands of dollars nightly on idle GPUs. If you provision 5 instances permanently, your service crashes during peak hours.

Autoscaling dynamically adjusts server replica counts to match live traffic demand.

Low Traffic (Night) ──► 2 Container Replicas Active (Low Cost)
                            │ (Traffic Spike Detected)
                            ▼
High Traffic (Day)   ──► 50 Container Replicas Active (Maintains Latency SLA)

Scaling Metrics: Choosing the Right Trigger

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. REQUEST QUEUE DEPTH   │ 2. GPU UTILIZATION       │ 3. P99 LATENCY           │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Number of unhandled      │ GPU compute usage or VRAM│ Average or tail response │
│ requests waiting in the  │ memory consumption       │ time crossing SLA        │
│ queue. Best overall!     │ percentage.              │ thresholds.              │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Avoid relying solely on CPU usage, as neural network models saturate GPU memory while leaving host CPUs idle.

Overcoming Cold Start Delays

A major challenge in GPU autoscaling is Cold Start Latency:

Autoscale Triggered ──► Provision Cloud Instance ──► Pull 10GB Docker Image ──► Load Model to VRAM ──► Ready (3 to 5 Minutes!)

During those 3 to 5 minutes, user traffic can back up, causing API timeouts.

Strategies to Reduce Cold Start Delays

1. Pre-warmed Buffer Pools: Maintain a small safety margin of idle container replicas ready to accept instant traffic spikes.
2. Predictive Autoscaling: Use historical traffic patterns to scale up instances 15 minutes before expected daily traffic spikes begin.
3. Model Weight Caching: Store model weights on local high speed NVMe drives on cluster nodes so containers load weights instantly into VRAM.

Say this out loud

Autoscaling inference workloads matches server capacity to incoming traffic volume. Scaling policies monitor metrics like request queue depth, GPU utilization, or response latency to add container replicas during traffic spikes and remove them during quiet periods. Techniques like predictive scaling and model caching minimize cold start delays.

Followups to expect

1. What is Scale-to-Zero in Knative or KServe? A serverless container pattern where workloads scale down to zero active instances during zero traffic, accepting a cold start delay on the first incoming user request.
2. What is Cooldown Period in autoscaling rules? A mandatory waiting period after a scaling action that prevents the system from rapidly scaling up and down repeatedly due to temporary noise spikes.