Docker & Kubernetes for ML

Why Containerization is Essential for ML

Machine learning applications have complex dependency chains:

• Specific Python version ($3.10$).
• Exact deep learning framework versions (PyTorch $2.2$, CUDA $12.1$).
• Low level C++ GPU drivers and system libraries.

Deploying raw Python scripts directly to cloud servers leads to dependency conflicts and environment failures.

Docker Containerization packages application code, model weights, framework libraries, and system dependencies into a single immutable Container Image.

[ Application Code ] + [ Model Weights ] + [ PyTorch / CUDA ] + [ OS Dependencies ]
                                       │
                                       ▼
                       [ DOCKER CONTAINER IMAGE ]
                                       │ (Deploy Anywhere!)
                                       ▼
                  Dev Laptop  ──►  Staging Server  ──►  Kubernetes Cluster

Docker Packaging Best Practices for ML

1. Use Official GPU Base Images: Start from official NVIDIA CUDA base images (nvidia/cuda:12.1.0-runtime-ubuntu22.04) to ensure pre-configured GPU driver support.
2. Separate Model Weights from Code: For large models ($> 1\text{ GB}$), download weights from S3 or Mount persistent volumes at container runtime rather than baking huge model files directly into Docker images.
3. Multi-Stage Builds: Use multi-stage Docker builds to keep final production container images lightweight by excluding build compilers and temporary files.

Kubernetes Orchestration for Model Serving

While Docker packages single containers, Kubernetes (K8s) manages clusters of hundreds of container instances:

                          ┌──► [ Pod 1: Model Server Container ] (GPU Node 1)
Incoming User Traffic ────┼──► [ Pod 2: Model Server Container ] (GPU Node 2)
  (K8s Load Balancer)     └──► [ Pod 3: Model Server Container ] (GPU Node 3)

Key Kubernetes Features for ML

• Horizontal Pod Autoscaling (HPA): Automatically scales container replicas up or down based on CPU, GPU memory, or request queue depth.
• GPU Resource Allocation: Assigns specific GPU fractions or dedicated NVIDIA GPUs to individual container pods (resources.limits.nvidia.com/gpu: 1).
• Self Healing: Automatically restarts container pods if a model server crashes or experiences out of memory errors.

Say this out loud

Docker and Kubernetes provide the containerization and orchestration standard for machine learning serving. Docker packages Python code, deep learning frameworks, CUDA drivers, and dependencies into reproducible container images. Kubernetes manages container deployment across clusters, handling horizontal autoscaling, load balancing traffic, and allocating GPU resources.

Followups to expect

1. What is KServe (formerly KFServing)? A Kubernetes native model serving platform that provides serverless inference features, scale to zero capabilities, and standardized APIs across PyTorch, TensorFlow, and XGBoost.
2. How do you optimize Docker image build speed for ML? Leverage Docker layer caching by placing static dependency installations (pip install) before copying frequently changing application code.