Experiment Tracking & Reproducibility

The Chaos of Untracked Experiments

Machine learning development requires testing dozens of ideas:

• Trying different learning rates ($0.01$ vs $0.001$).
• Testing distinct model architectures (ResNet vs Vision Transformer).
• Trying new feature transformations.

Without Experiment Tracking, teams forget which code state, hyperparameter combination, or dataset split generated a winning model checkpoint.

Manual Approach:   "model_v2_final_final_real.pt"  (Which code produced this? NO IDEA!)
Tracked Approach:  Run #842 ──► Commit: #a9f2 ──► lr: 0.001 ──► Val Accuracy: 94.2%

What to Log During Experiments

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. PARAMETERS            │ 2. METRICS               │ 3. ARTIFACTS             │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Learning rate, batch     │ Training loss, validation│ Model checkpoint files,  │
│ size, optimizer type,    │ accuracy, F1 score per   │ confusion matrix plots,  │
│ model depth.             │ epoch.                   │ feature importance graphs│
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Popular Experiment Tracking Tools

• MLflow: Open source experiment tracking and model packaging framework.
• Weights and Biases (W&B): Cloud and enterprise platform providing rich real time loss visualizations and hyperparameter sweep dashboards.
• Neptune.ai: Scalable metadata store for team collaboration on machine learning experiments.

Enforcing Reproducibility

Logging parameters is only half the battle. To guarantee $100%$ reproducible results across training runs:

1. Set Random Seeds: Set deterministic seeds for Python, NumPy, and PyTorch (torch.manual_seed(42)).
2. Log Environment Specifications: Export exact library versions (pip freeze or Docker container hashes).
3. Link Data Versions: Record the exact DVC dataset hash used during the training run.

Say this out loud

Experiment tracking records hyperparameters, loss metrics, code commits, and output artifacts for every model run. Specialized tools like MLflow or Weights and Biases automatically log training curves and parameter sweeps, allowing teams to compare models visually. Combining tracking tools with fixed random seeds and environment containers guarantees total experiment reproducibility.

Followups to expect

1. What are hyperparameter sweeps? Automated searches (using Grid Search, Random Search, or Bayesian Optimization) that launch dozens of parallel training runs across defined parameter ranges to find optimal configurations.
2. What is model checkpointing? Periodically saving model weight weights to disk during training whenever validation metrics reach a new peak score.