Deep Learningbeginnermust-know4 min

Activation Functions

How non linear activation functions turn simple linear math into powerful deep learning models.

Activation functions introduce non linearity into neural networks, allowing them to learn complex non linear patterns. Without non linear activation steps, a multi layer neural network collapses into a simple linear model regardless of depth. Common activations include Sigmoid for probabilities between 0 and 1, Tanh for zero centered signals between minus 1 and plus 1, ReLU for fast computation, and GELU for modern transformer architectures.

Read the deep dive →
Deep Learningbeginnermust-know4 min

Dropout

Preventing neural network overfitting by randomly turning off neurons during training.

Dropout (Srivastava et al., 2014) is a popular regularization technique for neural networks. During training, dropout randomly deactivates a fraction p of hidden neurons at each forward step. This forces the network to learn redundant feature representations instead of relying on fragile co adaptations between specific neurons. During inference evaluation, all neurons remain active, and weights are scaled by 1 minus p so output magnitude stays constant.

Read the deep dive →
Deep Learningintermediatemust-know5 min

Backpropagation

The engine of deep learning. Most candidates know the intuition; few can trace matrix dimensions correctly.

Backpropagation computes gradients of scalar loss L with respect to all network weights W using the multivariable chain rule in reverse mode. Running backwards from output to input computes all ∂L/∂W in O(W) operations, whereas finite differences or forward-mode differentiation require O(W²) complexity. Key interview topics include computational graphs, local gradients, error signals (δ), and matrix shape matching.

Read the deep dive →
Deep Learningintermediatemust-know4 min

Vanishing & Exploding Gradients

Why deep neural networks fail when gradients shrink to zero or explode to infinity during backpropagation.

Vanishing and Exploding Gradients are major training instabilities in deep neural networks and recurrent models. The chain rule multiplies local derivatives across layers during backpropagation. If local derivatives are smaller than 1, multiplying them repeatedly causes gradients to shrink exponentially toward zero, preventing early layers from learning. If local derivatives are larger than 1, gradients grow exponentially, causing weight instability or NaN overflow.

Read the deep dive →
Deep Learningintermediatemust-know5 min

Batch Normalization

Accelerating deep network training by stabilizing layer input distributions across mini batches.

Batch Normalization (Ioffe and Szegedy, 2015) normalizes layer activations across mini batch samples during training. It computes mean and variance for each feature dimension over the mini batch, standardizes activations to zero mean and unit variance, and applies learned scale and shift parameters. Batch Normalization smooths the loss landscape, acts as a mild regularizer, and enables higher learning rates without exploding gradients.

Read the deep dive →
Deep Learningintermediatemust-know5 min

SGD, Momentum, Adam & AdamW

How optimizers update neural network weights from basic SGD to Momentum and AdamW.

Optimizers adjust neural network weights to minimize loss. Stochastic Gradient Descent (SGD) updates weights along the negative loss gradient. SGD with Momentum adds a velocity vector to smooth updates and push past small local bumps. Adam tracks both first moments (momentum) and second moments (uncentered variance) per weight. AdamW fixes weight decay regularization in Adam by applying weight decay directly to parameters rather than mixing it into momentum.

Read the deep dive →
Deep Learningintermediatemust-know4 min

Transfer Learning & Fine-Tuning

Leveraging pre-trained representations from massive datasets to achieve state-of-the-art results on small target tasks.

Transfer Learning adapts a model pretrained on a large source dataset (ImageNet, C4, Wikipedia) to a specific target task. Strategies range from Feature Extraction (freezing backbone weights and training a new linear classification head) to Full Fine-Tuning and Parameter-Efficient Fine-Tuning (PEFT / LoRA). Key decision factors include target dataset size and domain similarity to the source data.

Read the deep dive →
Deep Learningintermediatemust-know5 min

Debugging a Training Run

Systematic protocols for diagnosing exploding gradients, loss spikes, and silent bugs in neural network training.

Debugging a Training Run requires a structured diagnostic protocol to isolate bugs in data pipelines, loss functions, and model architectures. Key steps include overfitting a single batch of 10 samples to zero loss, verifying loss values at step zero, monitoring gradient norms, checking for NaN/Inf floats, and inspecting activation statistics. Systematic debugging prevents wasted GPU compute costs and resolves training stalls quickly.

Read the deep dive →
Deep Learningbeginner4 min

Perceptron & the MLP

How simple artificial neurons combine to build deep neural networks.

A Perceptron is the simplest artificial neuron. It takes multiple input numbers, multiplies each input by a weight number, adds a bias number, and passes the result through an activation step. Single perceptrons can only learn straight line decision boundaries. A Multi Layer Perceptron combines layers of connected neurons with non linear activation steps, allowing the network to learn complex curved patterns in data.

Read the deep dive →
Deep Learningbeginner4 min

Choosing a Loss Function

Selecting the right loss function to guide neural network updates for regression, classification, and ranking.

A Loss Function measures the numerical error between model predictions and true ground truth targets. For continuous regression, Mean Squared Error penalizes large outliers heavily while Mean Absolute Error provides robust median predictions. For classification, Binary Cross Entropy and Categorical Cross Entropy measure divergence between predicted probability distributions and target one hot vectors.

Read the deep dive →
Deep Learningbeginner5 min

Data Augmentation

Expanding training dataset size and invariance by applying transformations across vision, audio, and text domains.

Data Augmentation artificially expands training datasets by applying domain specific transformations to existing data samples. In computer vision, techniques range from geometric spatial transforms (crops, flips, rotations) to color jitter and AutoAugment. In natural language processing, techniques include Back Translation, Synonym Replacement, and Contextual Word Insertion.

Read the deep dive →
Deep Learningintermediate4 min

Universal Approximation Theorem

Why a simple neural network can theoretically learn any continuous mathematical relationship.

The Universal Approximation Theorem proves that a feedforward neural network with a single hidden layer containing enough neurons and non linear activation functions can approximate any continuous function to any desired degree of accuracy. While the theorem guarantees that a sufficiently wide network can represent complex functions theoretically, it does not guarantee that gradient descent optimization can easily find those weights or that the network will generalize well to unseen data.

Read the deep dive →
Deep Learningintermediate5 min

Computational Graphs & Autograd

Building dynamic Directed Acyclic Graphs to compute automatic differentiation gradients via reverse mode autograd.

Computational Graphs and Automatic Differentiation (Autograd) power deep learning frameworks like PyTorch and TensorFlow. A Computational Graph is a Directed Acyclic Graph (DAG) where nodes represent data tensors or mathematical operations. PyTorch builds Dynamic Computational Graphs on the fly during forward passes, using Reverse Mode Automatic Differentiation to execute backpropagation gradients automatically.

Read the deep dive →
Deep Learningintermediate4 min

The Dying ReLU Problem

How neurons become permanently inactive when trapped in negative activation regions.

The Dying ReLU Problem occurs when neurons using the Rectified Linear Unit activation function become permanently inactive. Because ReLU outputs zero with zero gradient for all negative inputs, a large gradient update can push a neuron weights into a state where it outputs negative numbers for all training samples. Once a neuron dies, its gradient remains zero forever, preventing backpropagation from ever updating its weights again. Fixes include using Leaky ReLU, ELU, lower learning rates, or proper weight initialization.

Read the deep dive →
Deep Learningintermediate4 min

Gradient Clipping

A simple safety net that prevents exploding gradients from crashing deep learning models.

Gradient Clipping is a practical optimization technique that prevents exploding gradients during neural network training. If the length or magnitude of a gradient vector exceeds a maximum threshold value, it is rescaled back down to the threshold while preserving its direction. Gradient clipping is essential when training Recurrent Neural Networks, Transformer Language Models, and Deep Reinforcement Learning agents.

Read the deep dive →
Deep Learningintermediate5 min

Weight Initialization (Xavier/He)

Why starting neural network weights with the right variance prevents training from dying on step one.

Weight Initialization sets the starting random values for neural network parameters before training begins. Initializing all weights to zero causes symmetry problems where all neurons learn identical features. Random initialization must scale weight variance relative to layer input and output dimensions. Xavier Glorot initialization scales variance for Sigmoid and Tanh activations, while He Kaiming initialization doubles variance to account for zeroed out ReLU inputs.

Read the deep dive →
Deep Learningintermediate4 min

Learning Rate Schedules & Warmup

How dynamically adjusting learning rates during training accelerates convergence and improves model accuracy.

Learning Rate Schedules change the optimizer step size over the course of training. Using a fixed learning rate is sub optimal: high rates cause divergence or bouncing around minima, while low rates slow training. Common schedules include Linear Warmup (gradually raising rate at step one), Cosine Decay (smoothly reducing rate to near zero), and Step Decay.

Read the deep dive →
Deep Learningintermediate4 min

How Batch Size Changes Training

Understanding how batch size impacts training speed, GPU utilization, gradient noise, and model generalization.

Batch size defines the number of training samples processed in a single forward and backward pass before updating model weights. Small batch sizes provide noisy gradient updates that act as implicit regularization, helping models escape sharp sub optimal minima. Large batch sizes maximize GPU parallel hardware throughput, but require scaling the learning rate linearly to prevent generalization degradation.

Read the deep dive →
Deep Learningintermediate4 min

Softmax, Logits & Numerical Stability

Converting raw network scores into normalized probabilities while avoiding numerical overflow.

Logits are raw unnormalized real number outputs from the final layer of a neural network. The Softmax function converts a vector of logits into a normalized probability distribution where all values are positive and sum to 1. To prevent floating point numerical overflow when exponentiating large numbers, implementations subtract the maximum logit value before applying Softmax.

Read the deep dive →
Deep Learningintermediate4 min

Label Smoothing

Softening hard one hot targets to prevent neural networks from becoming overconfident.

Label Smoothing is a regularization technique that replaces hard binary targets with softened probability distributions. Standard cross entropy loss uses hard one hot targets like 1.0 for the correct class and 0.0 for all other classes, forcing the network to output infinite logit values to reach zero loss. Label smoothing softens targets to values like 0.9 for the correct class and redistributes 0.1 uniformly across incorrect classes, improving model calibration and generalization.

Read the deep dive →
Deep Learningintermediate5 min

Convolutional Neural Networks

How small sliding convolutional filters extract spatial features from images.

Convolutional Neural Networks (CNNs) are specialized deep learning architectures designed for 2D and 3D grid data like images and video. Instead of using dense fully connected layers that flatten spatial layouts, CNNs slide small feature filters over input pixels to extract local spatial patterns. Key building blocks include Convolutional Layers for feature extraction, Activation Layers for non linearity, Pooling Layers for spatial downsampling, and Dense Layers for final predictions.

Read the deep dive →
Deep Learningintermediate5 min

Pooling, Strides & Receptive Field

Controlling feature map dimensions and expanding spatial context in convolutional networks.

Pooling, Strides, and Receptive Field control spatial resolution and information flow in CNNs. Pooling downsamples feature maps by taking maximum (Max Pooling) or average (Average Pooling) values across local regions. Stride defines the filter step size across spatial dimensions. Receptive Field measures the total input image region that influences a specific feature unit in a deeper layer.

Read the deep dive →
Deep Learningintermediate4 min

Residual Connections

How identity skip connections allowed neural networks to scale past 100 layers without vanishing gradients.

Residual Connections (Skip Connections) add the original un-transformed input tensor directly to the output of a neural network block: y = F(x) + x. Introduced in ResNet (He et al., 2015), residual connections solve the vanishing gradient problem in deep networks. During backpropagation, the derivative of the identity path is 1.0, guaranteeing an unattenuated gradient highway back to early layers.

Read the deep dive →
Deep Learningintermediate5 min

RNNs, LSTMs & GRUs

Processing sequential data using hidden memory states from basic RNNs to LSTMs and GRUs.

Recurrent Neural Networks (RNNs) process sequential data by passing a hidden state vector from one time step to the next. Standard RNNs suffer from severe vanishing gradients over long sequences, preventing them from remembering distant past information. Long Short Term Memory (LSTM) networks solve vanishing gradients using a cell state memory highway controlled by input, forget, and output gates. Gated Recurrent Units (GRUs) simplify LSTMs by combining gates into reset and update gates.

Read the deep dive →
Deep Learningintermediate5 min

Seq2Seq & Encoder–Decoder

Mapping variable length input sequences to variable length output sequences for translation and summarization.

Sequence to Sequence (Seq2Seq - Sutskever et al., 2014) is an encoder decoder architecture that converts variable length input sequences into variable length output sequences. The Encoder processes input tokens step by step, compressing the sequence into a single context vector. The Decoder takes the context vector and generates output tokens one by one until an end of sequence token is produced. Early Seq2Seq models suffered from a context vector bottleneck on long sentences, which motivated Bahdanau Attention.

Read the deep dive →
Deep Learningintermediate4 min

Autoencoders

Compressing high dimensional inputs into low dimensional bottleneck vectors and reconstructing them.

An Autoencoder is an unsupervised neural network designed to compress inputs into a low dimensional bottleneck vector and reconstruct the original input. It consists of an Encoder that maps input x to a bottleneck representation z, and a Decoder that reconstructs x_hat from z. Autoencoders are used for dimensionality reduction, anomaly detection, image denoising, and representation learning.

Read the deep dive →
Deep Learningintermediate4 min

Which Layers to Freeze

Deciding which backbone layers to freeze versus fine tune during transfer learning.

Freezing Layers sets parameter requires_grad to False during Transfer Learning. Pretrained early layers capture domain independent low level features (edges, textures, grammar roots) that generalize across tasks. Freezing early layers accelerates training, cuts GPU memory overhead, and prevents catastrophic forgetting when fine tuning on small target datasets.

Read the deep dive →
Deep Learningintermediate4 min

Reproducibility & Nondeterminism

Controlling random seed initializations and CUDA nondeterminism to achieve reproducible deep learning experiments.

Reproducibility in deep learning requires controlling random seeds and hardware nondeterminism. Nondeterminism stems from random weight initializations, data loader shuffling, dropout masks, and asynchronous atomic floating point operations in GPU CUDA algorithms (cuDNN convolutions and atomicAdd). Achieving reproducible training requires setting global seeds across Python, NumPy, PyTorch, and enabling deterministic CUDA flags at a small performance cost.

Read the deep dive →
Deep Learningadvanced5 min

LayerNorm vs BatchNorm vs RMSNorm

Comparing how normalization across features versus across samples powers computer vision and transformer models.

Normalization techniques stabilize neural network training by normalizing activation distributions. Batch Normalization computes mean and variance across mini batch samples for each feature channel, excelling in computer vision. Layer Normalization computes mean and variance across feature dimensions independently for each single sample, making it ideal for sequential text and transformers. RMSNorm simplifies Layer Normalization by removing mean calculation, saving GPU compute time in modern LLMs like LLaMA.

Read the deep dive →
Deep Learningadvanced5 min

Depthwise Separable Convolutions

Cutting convolutional computation and parameter counts by 90 percent for mobile AI devices.

Depthwise Separable Convolutions split standard convolution into two separate steps: Depthwise Convolution (filtering spatial features per channel independently) and Pointwise Convolution (1x1 convolution combining channels). This factorization reduces computational FLOPs and parameter counts by 8 to 9 times with almost zero drop in model accuracy. Depthwise separable convolutions form the architectural engine behind efficient mobile models like MobileNet and ConvNeXt.

Read the deep dive →
Deep Learningadvanced4 min

Backprop Through Time

Extending backpropagation to unrolled recurrent networks across sequential time steps.

Backpropagation Through Time (BPTT) is the training algorithm used for Recurrent Neural Networks. BPTT unrolls an unrolled recurrent sequence across time steps into a deep computational graph, calculating gradients for shared weight matrices by summing gradient contributions from each time step. Truncated BPTT splits long sequences into shorter fixed sub sequences to prevent memory overflow and vanishing gradient degradation.

Read the deep dive →
Deep Learningadvanced4 min

Teacher Forcing & Exposure Bias

How feeding true target tokens during training accelerates sequence model convergence while introducing exposure bias.

Teacher Forcing is a training technique for autoregressive sequence models (RNN decoders, Transformers). During training, instead of feeding the model's own predicted token from step t back as input for step t+1, Teacher Forcing feeds the ground truth target token. This prevents early prediction mistakes from compounding, stabilizing training. However, it creates Exposure Bias because the model never sees its own mistakes during training, causing performance degradation during real world generation.

Read the deep dive →
Deep Learningadvanced5 min

Variational Autoencoders

Mapping inputs into continuous latent probability distributions for generative data sampling.

A Variational Autoencoder (VAE - Kingma & Welling, 2013) is a probabilistic generative model. Unlike standard autoencoders that map inputs to fixed vector points, VAEs map inputs to mean vector mu and variance vector sigma of a Gaussian latent distribution. The model is trained using the Evidence Lower Bound (ELBO) loss, combining Reconstruction Loss with KL Divergence regularization to enforce a smooth continuous latent space. The Reparameterization Trick enables backpropagation through stochastic random sampling.

Read the deep dive →
Deep Learningadvanced5 min

Generative Adversarial Networks

Pitting a Generator against a Discriminator in a minimax game to synthesize realistic images.

Generative Adversarial Networks (GANs - Goodfellow et al., 2014) are a class of generative models framed as a two player zero sum game. The Generator G maps random noise z to synthetic data samples G(z) attempting to fool the Discriminator. The Discriminator D classifies samples as real or fake D(x). Adversarial training optimizes a minimax objective function min_G max_D V(D, G) until Nash Equilibrium is reached where fake samples are indistinguishable from real data.

Read the deep dive →
Deep Learningadvanced5 min

Mode Collapse & GAN Instability

Understanding mode collapse, gradient vanishing, and Wasserstein distance stabilization in GAN training.

GAN Training Instability describes the severe optimization challenges when training Generative Adversarial Networks. Key instabilities include Mode Collapse (where the Generator produces a small repetitive set of outputs), Vanishing Gradients when the Discriminator becomes too dominant, and Non-Convergence due to non-convex minimax dynamics. Wasserstein GAN (WGAN-GP) stabilizes training by replacing JS divergence with Earth Mover (Wasserstein-1) Distance and Gradient Penalties.

Read the deep dive →
Deep Learningadvanced5 min

Diffusion Models

Synthesizing high fidelity images and audio by learning to reverse a gradual Gaussian noise degradation process.

Diffusion Models (DDPM - Ho et al., 2020) are state-of-the-art generative models for image, audio, and video synthesis. They consist of a Forward Noising Process that incrementally adds Gaussian noise to data over T steps until it becomes pure white noise, and a Reverse Denoising Process where a U-Net or Transformer network learns to predict and remove noise step by step. Latent Diffusion Models (Stable Diffusion) run denoising inside a compressed VAE latent space to achieve fast high resolution image generation.

Read the deep dive →
Deep Learningadvanced5 min

Flow Matching & Rectified Flow

Connecting noise to data using straight vector fields for fast few step generative sampling.

Flow Matching (Lipman et al., 2022) and Rectified Flow (Liu et al., 2022) are modern generative modeling paradigms that unify and outperform traditional Diffusion Models. Instead of simulating complex curved stochastic SDE paths, Flow Matching learns a continuous Vector Field v_theta(x, t) that pushes a simple Gaussian noise distribution straight into target data distributions along linear probability paths. Rectified Flow uses straight line paths, allowing generation of high quality images in as few as 1 to 4 Ordinary Differential Equation (ODE) steps.

Read the deep dive →
Deep Learningadvanced5 min

Graph Neural Networks

Processing graph structured data using Message Passing operations across nodes and edges.

Graph Neural Networks (GNNs) are specialized deep learning architectures designed for non Euclidean graph structured data (social networks, molecular graphs, knowledge graphs). GNNs operate via Message Passing: each node collects feature vectors from its connected neighbors, aggregates them using permutation invariant functions (sum, mean, max), and updates its own node embedding. Popular variants include Graph Convolutional Networks (GCN), Graph Attention Networks (GAT), and GraphSAGE.

Read the deep dive →
Deep Learningadvanced4 min

Mixup & CutMix

Blending image pixels and target labels to regularize deep computer vision networks.

Mixup and CutMix are advanced data augmentation and regularization techniques for computer vision. Mixup (Zhang et al., 2017) blends two random images and their one hot target labels linearly using convex combinations x = lambda * x1 + (1 - lambda) * x2. CutMix (Yun et al., 2019) cuts a rectangular patch from image 2 and pastes it onto image 1, scaling target labels proportionally to patch pixel area.

Read the deep dive →
Deep Learningadvanced5 min

Mixed Precision Training

Combining FP16/BF16 tensor math with FP32 master weights and loss scaling for 2x faster GPU training.

Automatic Mixed Precision (AMP - Micikevicius et al., 2017) accelerates deep learning training while cutting GPU memory usage by half. It executes forward and backward tensor matrix multiplications in 16 bit lower precision (FP16 or BF16) to leverage GPU Tensor Cores, while maintaining master model weights in 32 bit float (FP32) to prevent numerical underflow. Loss Scaling multiplies loss values by a scaling factor to prevent small FP16 gradient underflow before backpropagation.

Read the deep dive →
Deep Learningadvanced5 min

Gradient Accumulation & Checkpointing

Bypassing GPU memory bottlenecks using gradient accumulation loops and activation checkpointing.

Gradient Accumulation and Activation Checkpointing are two foundational GPU memory optimization techniques for deep learning. Gradient Accumulation simulates large global batch sizes by accumulating gradients over N micro batches before calling optimizer steps. Activation Checkpointing (Gradient Checkpointing) trades compute for memory by discarding intermediate activations during the forward pass and recomputing them on demand during backpropagation, reducing activation memory by up to 80 percent.

Read the deep dive →
Deep Learningadvanced5 min

Data vs Model Parallelism

Scaling deep learning training across multi GPU clusters using Data Parallelism and Model Parallelism.

Distributed Training scales neural network training across clusters of multiple GPUs and nodes. Distributed Data Parallelism (DDP) replicates full model weights across all GPUs, splitting mini batches across GPUs and synchronizing gradients via AllReduce collectives. Model Parallelism splits large model parameters across GPUs when a model is too large to fit inside a single GPU VRAM, using Tensor Parallelism (intra-layer) or Pipeline Parallelism (inter-layer).

Read the deep dive →
Deep Learningadvanced5 min

ZeRO, FSDP & Sharded Training

Eliminating memory redundancy in distributed training by sharding optimizer states, gradients, and model parameters.

ZeRO (Zero Redundancy Optimizer - Rajbhandari et al., 2020) and FSDP (Fully Sharded Data Parallel - PyTorch) eliminate memory redundancy in distributed data parallel training. Standard DDP replicates full model weights, gradients, and optimizer states across every GPU, causing massive memory duplication. ZeRO Stage 1 shards optimizer states, ZeRO Stage 2 shards gradients, and ZeRO Stage 3 / FSDP shards model parameters across GPUs, enabling training of billion parameter LLMs across standard GPU clusters.

Read the deep dive →
Deep Learningadvanced5 min

Contrastive & Self-Supervised Learning

Learning rich vector representations without human labels using InfoNCE loss and data augmentations.

Contrastive Learning is a self-supervised representation learning paradigm that trains neural networks without human annotations. It pulls augmented positive pairs of the same sample together in embedding space while pushing negative pairs apart using InfoNCE loss. Foundational architectures include SimCLR, MoCo, and CLIP, building high quality representations for vision, text, and multimodal retrieval.

Read the deep dive →
Deep Learningadvanced5 min

Knowledge Distillation

Compressing massive Teacher models into fast Student networks using dark knowledge soft targets.

Knowledge Distillation (Hinton et al., 2015) compresses large Teacher neural networks into smaller, faster Student models. The Student is trained on Soft Targets generated by passing data through the Teacher at a high Temperature T. Soft probabilities reveal Dark Knowledge (inter class similarity relationships), allowing a compact Student model to match 95 percent of a giant Teacher's accuracy at 10x faster inference speed.

Read the deep dive →
Deep Learningadvanced5 min

Pruning & Quantization

Compressing deep neural networks via zero weight elimination and bit precision reduction.

Pruning and Quantization are the two core model compression techniques for deploying deep neural networks to edge hardware. Pruning removes redundant weight connections based on magnitude or gradient importance (Magnitude Pruning, Structured Pruning). Quantization converts high precision 32 bit float weights (FP32) into lower bit integer representations (INT8 / INT4), using Post Training Quantization (PTQ) or Quantization Aware Training (QAT).

Read the deep dive →
Deep Learningadvanced5 min

Catastrophic Forgetting

Preventing neural networks from overwriting previously learned tasks when fine tuning on new datasets.

Catastrophic Forgetting occurs when a neural network fine-tuned sequentially on a new task drastically degrades or completely forgets capabilities learned during initial pretraining. Because standard backpropagation updates shared parameter weights globally, gradient steps optimized for Task B overwrite weight configurations critical for Task A. Mitigations include Parameter Efficient Fine Tuning (LoRA), Elastic Weight Consolidation (EWC), Experience Replay, and Regularization bounds.

Read the deep dive →
Deep Learningadvanced5 min

Double Descent & Grokking

How modern over-parameterized neural networks break classic U-curve bias-variance trade-offs.

Double Descent (Belkin et al., 2019 / Nakkiran et al., 2019) describes a phenomenon where test error decreases, spikes near the interpolation threshold, and then decreases again as model size or training time increases. Classical statistics predicts a U-curve where over-parameterization causes overfitting. Deep learning breaks the U-curve: past the Interpolation Threshold where a model fits training data perfectly, additional capacity acts as an implicit regularizer, leading to better test performance and Grokking.

Read the deep dive →
SCROLL · SAVE · TAP TO GO DEEPER