Classical MLbeginnermust-know4 min

Bias–Variance Tradeoff

How to fix a model that makes bad predictions using simple error tuning.

When a machine learning model makes mistakes, the errors come from three main sources. First, bias happens when a model is too simple and misses the real pattern completely. Second, variance happens when a model is too complex and memorizes random noise instead of learning general patterns. Third, irreducible noise comes from unavoidable random mistakes in the data itself. Finding the right balance means tweaking model size until predictions are accurate on new data.

Classical MLbeginnermust-know3 min

Overfitting vs Underfitting

Everyone can define it. Fewer can diagnose it from a learning curve.

Overfitting is learning the training set's noise as if it were signal — training error keeps dropping while validation error rises. Underfitting is failing to learn the signal at all — both errors stay high and close together. Detect it by tracking both curves as a function of training set size and epochs. Fix overfitting with more data, regularization, augmentation and early stopping; fix underfitting with capacity, features and longer training.

Classical MLbeginnermust-know4 min

Linear Regression

Simple on paper, brutal in interviews. Most candidates fail when asked to derive OLS or explain heteroscedasticity.

Linear Regression models continuous targets as y = Xβ + ε. Ordinary Least Squares (OLS) minimizes squared residuals, yielding the closed-form solution β = (XᵀX)⁻¹Xᵀy. Under the Gauss-Markov assumptions, OLS is the Best Linear Unbiased Estimator (BLUE). Key interview topics include closed-form derivation, assumption validation, multicollinearity, and OLS vs Gradient Descent trade-offs.

Classical MLbeginnermust-know4 min

Logistic Regression

The baseline classifier every ML candidate is expected to derive on a whiteboard.

Logistic Regression models binary probabilities using the Sigmoid activation: P(y=1|x) = σ(wᵀx + b). It maps linear log-odds to [0, 1]. Because OLS yields non-convex loss for probabilities, we use Binary Cross-Entropy (Log Loss), optimized iteratively via Gradient Descent or L-BFGS. Key interview topics include log-odds interpretation, loss derivation, linear decision boundaries, and multi-class extension via Softmax.

Classical MLbeginnermust-know5 min

Cross-Validation Done Right

Evaluating model generalization accuracy across K-Fold, Stratified, Group, and Time Series validation splits.

Cross Validation evaluates how a machine learning model generalizes to unseen test data by partitioning training datasets into multiple validation splits. Standard K-Fold splits data into K equal folds. Stratified K-Fold preserves class label proportions across folds, essential for imbalanced classification. Group K-Fold prevents data leakage by ensuring all samples from a specific subject or user stay in the same fold. Time Series Split respects temporal causality, using past data to predict future time steps.

Classical MLintermediatemust-know5 min

L1 vs L2 Regularization

Preventing model overfitting by penalizing large parameter weights using L1 Lasso and L2 Ridge penalties.

L1 (Lasso) and L2 (Ridge) Regularization prevent machine learning models from overfitting by adding weight penalty terms to the loss function. L1 Regularization adds the absolute sum of weights |w|, driving irrelevant feature weights to exact zero and performing automatic feature selection. L2 Regularization adds the squared sum of weights w^2, shrinking weight magnitudes smoothly without driving them to exact zero.

Classical MLintermediatemust-know4 min

Gradient Descent & Its Variants

Comparing Batch, Stochastic, and Mini Batch Gradient Descent optimization strategies.

Gradient Descent is an optimization algorithm that iteratively minimizes a loss function by taking steps in the direction of steepest descent. Batch Gradient Descent calculates loss over the entire dataset before making one weight update. Stochastic Gradient Descent (SGD) updates weights after every single data sample. Mini Batch Gradient Descent updates weights after small batches of samples (such as size 32 to 256), combining hardware efficiency with gradient noise regularization.

Classical MLintermediatemust-know5 min

Bagging vs Boosting

Contrasting parallel variance reduction in Bagging against sequential bias reduction in Boosting.

Bagging and Boosting are the two dominant ensemble learning paradigms in machine learning. Bagging (Bootstrap Aggregating) trains independent complex base models in parallel on random data subsets to reduce variance (Random Forest). Boosting trains simple weak base models sequentially, focusing each new model on errors made by previous models to reduce bias (AdaBoost, XGBoost).

Classical MLintermediatemust-know5 min

Principal Component Analysis

Compressing high dimensional data into orthogonal principal components while maximizing preserved variance.

Principal Component Analysis (PCA - Pearson, 1901) is an unsupervised linear dimensionality reduction algorithm. PCA finds orthogonal linear combinations of features (Principal Components) that maximize data variance. It computes the covariance matrix of mean centered data, calculating eigenvectors (component directions) and eigenvalues (component variance magnitude) to project high dimensional data into lower dimensions.

Classical MLintermediatemust-know5 min

Handling Imbalanced Datasets

Techniques for handling severe class imbalances in fraud, medical, and anomaly detection models.

Class Imbalance occurs when majority classes dominate minority target classes (e.g. 99.9% non-fraud vs 0.1% fraud). Standard models achieve 99.9% dummy accuracy by predicting majority class everywhere while completely missing minority events. Solutions include Resampling (Random Undersampling, SMOTE oversampling), Cost Sensitive Learning (class weighting, Focal Loss), and evaluating via PR-AUC and F1-Score rather than raw Accuracy.

Classical MLadvancedmust-know5 min

Gradient Boosting (XGBoost/LightGBM)

Sequentially fitting decision trees to negative loss gradients for state of the art tabular predictions.

Gradient Boosting (Friedman, 2001) is a powerful ensemble method that builds decision trees sequentially to minimize a loss function. Unlike AdaBoost which reweights samples, Gradient Boosting fits each new decision tree directly to the negative gradient (pseudo residuals) of the loss function calculated from previous tree predictions. Shrinkage (learning rate) scales tree contributions, preventing overfitting while achieving top performance on tabular datasets.

Classical MLbeginner4 min

Supervised, Unsupervised & Self-Supervised

The fundamental taxonomy of learning paradigms: from explicit labels to hidden patterns and self-supervised pretraining.

Supervised Learning trains models on labeled input-output pairs (X, Y) for classification and regression. Unsupervised Learning finds inherent structural patterns, clusters, or lower-dimensional representations in unlabeled data (X). Self-Supervised Learning bridges the gap by automatically generating pseudo-labels from raw data structure (e.g. Next-Token Prediction, Masked Autoencoders), serving as the foundational engine for modern Foundation Models and LLMs.

Classical MLbeginner5 min

Decision Trees

Building interpretable hierarchical decision boundaries by recursively splitting feature spaces.

Decision Trees are non parametric supervised learning algorithms that partition data into hierarchical axis aligned decision regions. Starting from a root node, the tree greedily splits features at thresholds that maximize impurity reduction (Gini Impurity or Information Gain Entropy). Decision Trees handle non linear patterns and mixed feature types without feature scaling, but overfit easily if unpruned.

Classical MLbeginner4 min

k-Nearest Neighbours

Making non parametric predictions based on distance metrics to the k closest training samples.

k-Nearest Neighbors (k-NN) is a non parametric instance based supervised learning algorithm. Instead of learning explicit model parameter weights during training, k-NN stores all training data points and makes real time predictions by locating the k closest neighbors using distance metrics like Euclidean or Manhattan distance. Predictions are computed via majority voting for classification or mean averaging for regression, but inference speed scales poorly on large high dimensional datasets.

Classical MLbeginner4 min

Naive Bayes

The ultra-fast probabilistic classifier that assumes feature independence and handles text baselines effortlessly.

Naive Bayes applies Bayes' Theorem to classification under the 'naive' assumption that all features x_i are conditionally independent given class label y. This simplifies class posterior calculation to P(y | x) ∝ P(y) ∏ P(x_i | y). Variants include Gaussian Naive Bayes (continuous features), Multinomial Naive Bayes (word counts in text), and Bernoulli Naive Bayes (binary features). Laplace smoothing (additive α-smoothing) prevents zero-frequency zero probability traps.

Classical MLbeginner5 min

k-Means Clustering

Partitioning unlabelled data points into k compact clusters via iterative centroid updates.

k-Means Clustering (MacQueen, 1967) is an unsupervised learning algorithm that partitions N data points into k distinct clusters. It alternates between two steps: Assigning points to the nearest centroid using Euclidean distance, and Updating centroids to be the mean center of all assigned points. k-Means minimizes Within-Cluster Sum of Squares (Inertia), but is sensitive to initial random centroid placement (fixed by k-means++) and assumes spherical clusters.

Classical MLbeginner4 min

Missing Data & Imputation

Understanding missing data mechanisms (MCAR, MAR, MNAR) and selecting robust imputation strategies.

Missing data degrades model quality and causes pipeline failures if unhandled. Missingness falls into three mechanisms: Missing Completely at Random (MCAR), Missing at Random (MAR), and Missing Not at Random (MNAR). Imputation techniques range from simple Mean/Median/Mode insertion and indicator flags to Iterative MICE (Multivariate Imputation by Chained Equations) and KNN imputation. Modern GBDTs (XGBoost, LightGBM) handle missing values natively during split finding.

Classical MLintermediate5 min

Assumptions of Linear Regression

Evaluating the core mathematical assumptions required for unbiased Ordinary Least Squares regression.

Linear Regression using Ordinary Least Squares (OLS) relies on key statistical assumptions for valid parameter inference. Core assumptions include Linearity between features and target, Independence of error residuals, Homoscedasticity (constant residual variance), and Normality of residuals. Violating these assumptions introduces coefficient bias, invalidates p-values and confidence intervals, and requires diagnostic fixes like log transforms or Robust Standard Errors.

Classical MLintermediate4 min

Multicollinearity & VIF

Detecting and resolving linear dependence among predictor features using Variance Inflation Factor.

Multicollinearity occurs when two or more predictor features in a linear model are highly correlated with each other. While multicollinearity does not reduce overall model predictive power, it inflates coefficient standard errors, flips parameter signs, and makes individual feature importance uninterpretable. Multicollinearity is diagnosed using the Variance Inflation Factor (VIF) and resolved via feature removal, PCA, or L2 Ridge Regularization.

Classical MLintermediate4 min

Elastic Net

Combining L1 Lasso sparse selection with L2 Ridge group stability for high dimensional regression.

Elastic Net (Zou & Hastie, 2005) combines L1 Lasso and L2 Ridge regularization penalties into a single objective function. While L1 Lasso zeroing out weights struggles on correlated feature groups by picking one at random, Elastic Net uses L2 regularization to group correlated features together while using L1 regularization for sparse selection. Elastic Net is ideal when feature count p is much larger than sample count n, or when input features show high multi-collinearity.

Classical MLintermediate4 min

Normal Equation vs Gradient Descent

Comparing closed form analytical matrix inversion against iterative gradient descent for linear regression.

The Normal Equation provides an exact closed form analytical solution for Linear Regression: w = (X^T X)^-1 X^T y. While the Normal Equation computes exact optimal weights in a single matrix operation without tuning a learning rate, it requires inverting an n x n feature matrix with cubic complexity O(n^3). Gradient Descent scales far better to high dimensional datasets with millions of features or samples.

Classical MLintermediate4 min

Gini Impurity vs Entropy

Comparing Gini Impurity and Information Gain Entropy for selecting decision tree splits.

Gini Impurity and Entropy are mathematical metrics used to measure node impurity when building Decision Trees. Gini Impurity measures the probability of misclassifying a randomly chosen sample if labeled according to class distributions in the node. Entropy measures Information Disorder in bits using logarithmic scaling. In practice, Gini Impurity and Entropy yield nearly identical decision tree structures 98 percent of the time, but Gini is faster because it avoids logarithm calculations.

Classical MLintermediate4 min

Pruning & Tree Regularization

Trimming unconstrained decision tree branches to control model complexity and eliminate overfitting.

Tree Pruning removes non critical branches from Decision Trees to prevent overfitting and improve generalization on unseen test data. Pre Pruning (Early Stopping) halts tree growth during training using stopping rules like max depth or min samples per leaf. Post Pruning (Cost Complexity Pruning) grows a fully deep tree first, then collapses subtrees that add excessive complexity relative to validation accuracy gain.

Classical MLintermediate5 min

Random Forests

Combining hundreds of decorrelated decision trees using bagging and random feature sampling.

Random Forest (Breiman, 2001) is an ensemble learning method that combines hundreds of deep Decision Trees using Bootstrap Aggregation (Bagging). To decorrelate individual trees, Random Forest samples a random subset of features sqrt(d) at every split node in addition to bootstrap sample drawing. Predictions are aggregated by majority voting for classification or mean averaging for regression, achieving high accuracy without overfitting.

Classical MLintermediate4 min

AdaBoost

Sequentially reweighting misclassified samples to convert weak decision stumps into strong classifiers.

Adaptive Boosting (AdaBoost - Freund & Schapire, 1997) is the foundational sequential boosting algorithm. AdaBoost fits a sequence of weak base classifiers (1-split decision stumps), increasing sample weights for misclassified instances at each step so subsequent stumps focus on hard samples. Final predictions are computed using a weighted majority vote where more accurate stumps receive higher voting weights.

Classical MLintermediate5 min

Support Vector Machines

Finding the maximum margin decision boundary between classification classes.

Support Vector Machines (SVM - Cortes & Vapnik, 1995) are supervised learning algorithms that find a maximum margin hyperplane separating classes. SVM maximizes the geometric margin distance between the decision boundary hyperplane and the closest data points from each class (Support Vectors). Soft Margin SVM introduces slack variables xi and regularization hyperparameter C to balance margin width against classification errors.

Classical MLintermediate5 min

Choosing k: Elbow & Silhouette

Determining the optimal number of clusters k using Elbow inertia plots and Silhouette coefficient scores.

Choosing the optimal number of clusters k is a central challenge in unsupervised learning algorithms like k-Means. The Elbow Method plots Inertia against k, searching for an elbow bend point where marginal inertia reduction levels off. Silhouette Analysis measures how well separated clusters are by comparing mean intra-cluster distance a(i) to mean nearest-neighbor cluster distance b(i), outputting scores between minus 1 and plus 1.

Classical MLintermediate4 min

Hierarchical Clustering

Building nested clusters into a tree structure without specifying cluster count k upfront.

Hierarchical Clustering constructs a nested tree of clusters called a Dendrogram. Agglomerative (bottom-up) starts with N individual clusters and iteratively merges the closest pair of clusters until 1 root remains. Divisive (top-down) starts with 1 root cluster and recursively splits. Distance between clusters depends on Linkage criteria: Single (minimum distance), Complete (maximum distance), Average, or Ward's (minimum variance increase).

Classical MLintermediate4 min

DBSCAN & Density Clustering

Finding arbitrary-shaped clusters and identifying noise outliers without specifying cluster count k.

DBSCAN (Density-Based Spatial Clustering of Applications with Noise) groups points based on spatial density rather than centroids. Given neighborhood radius ε (eps) and minimum points minPts, points are categorized as Core Points, Border Points, or Noise Outliers. DBSCAN discovers clusters of arbitrary shapes (concentric rings, spirals), handles noise natively, and does not require specifying k upfront.

Classical MLintermediate4 min

Generative vs Discriminative Models

Contrasting joint probability distribution learning against direct decision boundary estimation.

Generative and Discriminative models represent the two foundational approaches to probabilistic machine learning. Discriminative Models estimate conditional probability P(Y | X) directly, learning decision boundaries to classify inputs into classes (Logistic Regression, SVM, BERT). Generative Models estimate joint probability P(X, Y) = P(X | Y) P(Y), modeling how data features X are generated for each class Y (Naive Bayes, VAEs, GANs, GPT).

Classical MLintermediate5 min

Hyperparameter Tuning Strategies

Systematically searching the hyperparameter space to optimize model generalization without wasting GPU compute.

Hyperparameter Tuning optimizes non-trainable configuration settings (learning rate, tree depth, batch size, regularization λ). Grid Search exhaustively evaluates a Cartesian product of predefined values. Random Search samples configurations randomly, outperforming Grid Search when only a few hyperparameters dominate performance (Bergstra & Bengio, 2012). Bayesian Optimization constructs a Gaussian Process surrogate model to balance exploration and exploitation via acquisition functions (Expected Improvement, UCB).

Classical MLintermediate5 min

Outlier & Anomaly Detection

Detecting anomalous data points that distort statistical metrics or represent fraud, intrusions, and system failures.

Outlier and Anomaly Detection identifies observations that deviate significantly from expected normal data distributions. Statistical methods use Z-score thresholds or Interquartile Range (IQR bounds: Q1 - 1.5·IQR, Q3 + 1.5·IQR). Unsupervised ML algorithms include Isolation Forest (isolating anomalies via random splits), One-Class SVM (fitting tight decision boundaries around normal points), Local Outlier Factor (LOF), and Autoencoder reconstruction error thresholds.

Classical MLintermediate5 min

Time Series Forecasting Basics

Predicting future values of time-indexed data by modeling trend, seasonality, autocorrelation, and exogenous features.

Time Series Forecasting predicts future observations y_{t+h} given historical observations y_1, ..., y_t. Time series data decomposes into Trend T_t, Seasonality S_t, Cyclical patterns C_t, and Residual Noise I_t. Classical statistical approaches (ARIMA, Exponential Smoothing) rely on stationarity and autocorrelation. Modern Machine Learning approaches (XGBoost, Prophet, DeepAR, Temporal Fusion Transformers) frame forecasting as supervised regression using lagged features and rolling window statistics.

Classical MLintermediate4 min

Stationarity & Differencing

Why classical time series models fail completely on non-stationary data, and how differencing transforms raw data into stationary signals.

Stationarity is a fundamental requirement for classical time series models (ARIMA, Vector Autoregression). A time series is Strictly Stationary if its joint distribution is invariant to time shifts. It is Weakly (Strictly Weak/Second-Order) Stationary if its Mean E[Y_t] = μ is constant over time, Variance Var(Y_t) = σ² is constant, and Autocovariance Cov(Y_t, Y_{t+k}) depends only on lag k. Non-stationary series containing trends or seasonal shifts are transformed into stationary series via First Differencing (ΔY_t = Y_t - Y_{t-1}) or Log Transforms.

Classical MLintermediate4 min

The No Free Lunch Theorem

Why no single machine learning algorithm can outperform all others across all possible problem domains.

The No Free Lunch Theorem (Wolpert & Macready, 1997) states that no single machine learning algorithm universally outperforms every other algorithm when averaged over all possible data distributions. An algorithm that performs exceptionally well on image recognition must make inductive assumptions that render it sub optimal on tabular data or time series. The theorem highlights that machine learning performance depends entirely on matching an algorithm's Inductive Biases to the true data generating distribution.

Classical MLadvanced5 min

XGBoost vs LightGBM vs CatBoost

Comparing the three modern gradient boosting frameworks that power competitive data science.

XGBoost, LightGBM, and CatBoost are the three dominant gradient boosting libraries for tabular data. XGBoost introduced second order Hessian optimization, exact pre sorted greedy splits, and hardware regularization. LightGBM introduced Leaf Wise tree growth, Histogram Binning, and GOSS (Gradient-based One-Side Sampling), running 10 times faster with lower memory usage. CatBoost introduced Ordered Boosting and Target Encoding to handle high cardinality categorical features without target leakage.

Classical MLadvanced5 min

Stacking & Blending

Combining diverse machine learning model predictions using voting, weighted averaging, and meta learning.

Ensembling combines predictions from multiple machine learning models to improve overall generalization accuracy. Simple ensembling uses Voting (majority rule for classification) or Weighted Averaging (scaling outputs by validation accuracy). Stacking (Stacked Generalization) trains a second stage Meta Learner model on out of fold predictions generated by diverse base Level 0 models, leveraging complementary strengths across distinct algorithms.

Classical MLadvanced5 min

The Kernel Trick

Projecting non linearly separable data into higher dimensional spaces without computing explicit feature vectors.

The Kernel Trick allows linear algorithms like Support Vector Machines to operate in high dimensional non linear feature spaces. Instead of explicitly transforming inputs phi(x) into high dimensional space, Kernel Functions compute inner dot products K(x, z) = <phi(x), phi(z)> directly in raw low dimensional input space. Popular kernels include Polynomial Kernel and Radial Basis Function (RBF / Gaussian) Kernel, which implicitly projects data into infinite dimensional Hilbert space.

Classical MLadvanced5 min

Gaussian Mixtures & EM

Probabilistic soft-clustering using mixtures of Gaussians optimized via Expectation-Maximization.

Gaussian Mixture Models (GMMs) model complex data distributions as a weighted sum of K multivariate Gaussian components. Unlike k-Means which assigns hard cluster memberships (0/1), GMM provides soft probabilistic assignments P(Component k | x_i). Parameters (means μ_k, covariance matrices Σ_k, mixing weights π_k) are estimated via the Expectation-Maximization (EM) algorithm, alternating between computing responsibility probabilities (E-step) and updating Gaussian parameters (M-step).

Classical MLadvanced5 min

LDA for Dimensionality Reduction

Projecting supervised data into lower dimensions while maximizing between class separation and minimizing within class variance.

Linear Discriminant Analysis (LDA - Fisher, 1936) is a supervised dimensionality reduction and classification algorithm. Unlike unsupervised PCA which maximizes total data variance without class labels, LDA uses class labels to find projection axes that maximize between class variance relative to within class variance. For C classes, LDA can reduce dimensions to at most C minus 1 principal components, assuming normally distributed classes with equal covariance matrices.

Classical MLadvanced5 min

t-SNE vs UMAP

Visualizing high dimensional embeddings in 2D using non linear neighborhood manifold projections.

t-SNE and UMAP are non linear dimensionality reduction techniques designed for 2D and 3D data visualization. t-SNE maps pairwise high dimensional Gaussian similarities to low dimensional Student t distributions, resolving crowding issues but sacrificing global distance structure. UMAP uses Riemannian geometry and fuzzy simplicial sets to preserve both local cluster neighborhoods and global inter cluster relationships, running much faster than t-SNE.

Classical MLadvanced5 min

Bayesian Optimization for HPO

Efficiently tuning expensive black box hyperparameters using Gaussian Processes and Acquisition Functions.

Bayesian Optimization is a sequential strategy for optimizing expensive black box functions (such as hyperparameter tuning for deep neural networks). It builds a probabilistic Surrogate Model (Gaussian Process) to estimate the objective function mean and uncertainty across hyperparameter space. An Acquisition Function (Expected Improvement or Upper Confidence Bound) balances Exploration (sampling high uncertainty regions) against Exploitation (sampling near current best parameters).

Classical MLadvanced5 min

ARIMA vs Prophet vs Gradient Boosting

Comparing linear statistical autoregression against additive generalized additive models and tree-based gradient boosting for time series.

ARIMA (AutoRegressive Integrated Moving Average) is a linear statistical model combining autoregressive lags (p), differencing (d), and moving average error lags (q). Prophet (Meta) is a Generalized Additive Model (GAM) fitting non-linear trend, multi-period seasonality (weekly, yearly), and holiday effects via curve fitting. Gradient Boosting (XGBoost / LightGBM) treats forecasting as tabular regression over lag and rolling window features.

Classical MLadvanced5 min

Survival Analysis

Modeling time-to-event outcomes while handling censored observations in medical and customer churn analysis.

Survival Analysis models time to event data (such as customer churn, hardware failure, or patient relapse). Standard regression algorithms fail on survival data due to Right Censoring, where study periods end before an event occurs for some subjects. Kaplan Meier Estimator provides non parametric survival probability curves S(t), while Cox Proportional Hazards Model evaluates how covariate features scale hazard rates h(t) linearly.

Classical MLadvanced5 min

Semi-Supervised Learning

Leveraging vast unlabelled data pools alongside small labelled training sets.

Semi Supervised Learning combines a small dataset of labelled samples with a large pool of unlabelled samples. Because manual data annotation is expensive, semi-supervised techniques extract structural data geometry from unlabelled samples to improve decision boundaries. Key paradigms include Pseudo Labeling (Self Training), Consistency Regularization (FixMatch), and Generative Pretraining followed by Fine Tuning.

Classical MLadvanced5 min

Active Learning

Interactively selecting the most informative unlabelled samples for human annotation.

Active Learning is a subfield of machine learning where the algorithm interactively queries a human annotator (Oracle) to label specific unlabelled samples. Instead of labeling data at random, Active Learning uses Query Strategies (Uncertainty Sampling, Query By Committee, Expected Model Change) to select the single most informative data points. This achieves high model accuracy using up to 90 percent fewer human annotation labels.

SCROLL · SAVE · TAP TO GO DEEPER