CI/CD for ML

What is CI/CD for Machine Learning?

Traditional Continuous Integration and Continuous Deployment (CI/CD) automates software testing and deployment:

Traditional CI/CD:   Code Commit ──► Unit Tests ──► Build Container ──► Deploy Application

Machine learning applications depend on Code, Data, and Models. CI/CD for ML (CICO/CT) automates testing across all three components:

ML CI/CD:   Code & Data Commit ──► Data Validation ──► Retrain Model ──► Evaluation Gate ──► Deploy Endpoint

The 4 Stages of an ML CI/CD Pipeline

┌──────────────────────────┬──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. CONTINUOUS INTEGRATION│ 2. DATA VALIDATION       │ 3. CONTINUOUS TRAINING   │ 4. CONTINUOUS DEPLOYMENT │
├──────────────────────────┼──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Lint code, run unit      │ Validate data schemas,   │ Automate model training  │ Deploy verified model to │
│ tests, test data preprocessing| check for nulls and out│ jobs on fresh data      │ staging or production via│
│ functions.               │ of range values.         │ commits.                 │ canary rollout.          │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Stage 1: Continuous Integration (Code)

Whenever an engineer pushes code to Git:

• Run code linters (flake8, black).
• Run unit tests on feature engineering functions (pytest).
• Verify data transformation pipelines using small synthetic datasets.

Stage 2: Data Validation

Run automated data checks using tools like Great Expectations:

• Confirm column data types match required schemas.
• Ensure null values do not exceed allowed thresholds.
• Detect distribution anomalies before training starts.

Stage 3: Continuous Training and Evaluation Gates

The pipeline automatically trains a candidate model. Before deployment, the candidate model must pass an Automated Evaluation Gate:

$$\text{Candidate Metric (NDCG)} > \text{Production Metric (NDCG)} + \text{Threshold}$$

If the new model performs worse than the live production model, the pipeline cancels deployment and alerts the engineering team.

Stage 4: Continuous Deployment (CD)

Once verified, the pipeline builds a new Docker container image, registers model artifacts in the Model Registry, and deploys the container using canary deployment.

Say this out loud

CI/CD for machine learning extends traditional software testing by automating code tests, data validation, model retraining, and deployment gates. When new code or data is committed, the pipeline validates data schemas, trains a candidate model, and evaluates performance against live baselines. Only models that pass quality evaluation gates are automatically deployed.

Followups to expect

1. What is Causal Test Driven Development in ML? Writing test cases that verify specific invariant model behaviors, such as confirming that increasing credit score never increases predicted loan interest rates.
2. What is CML (Continuous Machine Learning)? An open source tool integrated with GitHub Actions that runs model training on cloud GPUs and posts evaluation metrics directly as comments on pull requests.