Reproducibility & Nondeterminism

Why Is Deep Learning Nondeterministic?

You run a training script twice on the exact same dataset, but get different final accuracy validation scores:

• Run 1: Validation Accuracy = $92.4%$
• Run 2: Validation Accuracy = $91.8%$

Why does this happen?

In deep learning, nondeterminism arises from multiple software and GPU hardware layers:

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. SOFTWARE RANDOMNESS   │ 2. DATA PIPELINE         │ 3. GPU CUDA HARDWARE     │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Weight initializations,  │ Multi-threaded worker    │ Asynchronous atomicAdd   │
│ Dropout masks, Data      │ data loader shuffling    │ operations in cuDNN      │
│ augmentation transforms. │ and batch ordering.      │ non-associative floats.  │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

Hardware Level Nondeterminism: The Floating Point Trap

Why do GPU operations produce different numbers even with random seeds set?

Floating point addition is NON-ASSOCIATIVE:

$$(a + b) + c \neq a + (b + c) \quad \text{in 32-bit floating point arithmetic!}$$

During parallel GPU backpropagation, thousands of CUDA threads execute atomicAdd operations to sum gradients into a shared memory address.

Because thread execution order depends on microsecond hardware timing and temperature, gradients are added in slightly different orders on every run, accumulating tiny rounding errors that compound over thousands of steps!

The Complete Reproducibility Checklist

To make a PyTorch training run 100 percent reproducible:

Step 1: Set Global Random Seeds

import random
import os
import numpy as np
import torch

def set_seed(seed: int = 42):
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed) # For multi-GPU setups!

Step 2: Enforce Deterministic CUDA Kernels

torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False # Disables auto-tuner for convolution algorithms
torch.use_deterministic_algorithms(True) # Enforces error if non-deterministic operation is called

Step 3: Seed PyTorch DataLoader Workers

def seed_worker(worker_id):
    worker_seed = torch.initial_seed() % 2**32
    np.random.seed(worker_seed)
    random.seed(worker_seed)

g = torch.Generator()
g.manual_seed(42)

dataloader = DataLoader(
    dataset,
    batch_size=32,
    shuffle=True,
    worker_init_fn=seed_worker,
    generator=g,
)

The Performance Trade-off

Enforcing 100 percent deterministic algorithms causes a $10%$ to $20%$ training speed slowdown because fast asynchronous atomic CUDA kernels are replaced by slower deterministic algorithms.

  DEVELOPMENT & DEBUGGING:  Enable Deterministic Mode to isolate bugs!
  PRODUCTION PRETRAINING:   Disable Deterministic Mode for maximum GPU FLOP speed!

Say this out loud

Deep learning nondeterminism stems from random seeds, data loader worker shuffling, and non associative floating point atomicAdd operations in GPU CUDA kernels. Achieving reproducibility requires setting global seeds across Python, NumPy, and PyTorch, seeding DataLoader workers, and enabling torch use deterministic algorithms at a small speed penalty.

Followups to expect

1. What happens if an operation lacks a deterministic CUDA implementation in PyTorch? Calling torch.use_deterministic_algorithms(True) throws a RuntimeError listing the exact non-deterministic operator (e.g. ctc_loss or unfold).
2. Is cross-hardware reproducibility possible? No. Running the exact same seeded code on an NVIDIA A100 vs an H100 GPU or CPU will produce slightly different numerical outputs due to different underlying low-level hardware ISA instructions.