Data & Feature Engineering

Sampling from Huge Datasets

Extracting representative training samples from multi terabyte datasets using Reservoir, Stratified, and Importance Sampling.

🟡 intermediate5 min readdata
Sampling from Huge Datasets enables training machine learning models on multi-terabyte data lakes. Loading massive raw datasets into memory is computationally impossible. Engineers use Stratified Sampling to preserve class balances, Reservoir Sampling to sample streaming data in a single pass, and Importance Sampling to focus compute on informative or rare samples.

The Scale Bottleneck

Modern data lakes store terabytes or petabytes of raw interaction logs.

Attempting to load petabyte scale datasets into GPU memory for model training is computationally impossible.

To train models efficiently, we use Data Sampling Strategies to extract smaller, statistically representative training subsets.

Petabyte Data Lake (100 Billion Rows) ──► [ STATISTICAL SAMPLING ] ──► Representative Sub-sample (1 Million Rows)

Key Sampling Algorithms

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. STRATIFIED SAMPLING   │ 2. RESERVOIR SAMPLING    │ 3. IMPORTANCE SAMPLING   │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Divides data into class  │ Uniformly samples fixed  │ Samples instances        │
│ strata first, sampling   │ size K elements from a   │ proportional to their    │
│ proportionally to preserve│ streaming data pipeline  │ informational value or   │
│ exact class ratios!      │ in a SINGLE pass.        │ difficulty.              │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Stratified Sampling (Preserving Class Balances)

When working with imbalanced classes (for example 99.9% legitimate transactions and 0.1% fraud), simple random sampling might extract a subset with zero fraud examples!

Stratified Sampling splits data into class subgroups (strata) first, then draws samples proportionally from each class to preserve exact population ratios.

2. Reservoir Sampling (Streaming Data Streams)

Suppose data arrives as a continuous stream of unknown total length $N$. How do you extract a uniform random sample of size $K$ without storing all $N$ items?

Algorithm:

  1. Keep the first $K$ items in an array (the Reservoir).
  2. For each incoming item $i$ (where $i > K$), generate a random integer $j$ between $1$ and $i$.
  3. If $j \le K$, replace item $j$ in the Reservoir with the new incoming item.

Every item in the stream receives an exact equal probability $\frac{K}{N}$ of being included in the final sample!

3. Importance Sampling (Prioritizing High Loss Cases)

Instead of uniform sampling, draw samples proportional to their loss or difficulty:

Say this out loud

Sampling from huge datasets extracts representative subsets for efficient model training. Stratified sampling preserves exact class proportions on imbalanced datasets. Reservoir sampling extracts uniform random samples from streaming data in a single pass without storing the full stream. Importance sampling prioritizes informative or high loss samples to accelerate training.

Followups to expect

  1. What is Downsampling vs Upsampling? Downsampling reduces majority class samples to match minority class counts. Upsampling duplicates or synthetically generates (SMOTE) minority class samples to balance classes.
  2. How does distributed Spark sampling work? Spark partitions data across cluster worker nodes, executing parallel local sampling on each partition before aggregating sample results.

Check yourself

Question 1 of 3

Why is simple random sampling dangerous when dealing with heavily imbalanced datasets like credit card fraud?

More in Data & Feature Engineering

See all →
Feature Engineering Fundamentals4 minSQL Questions in ML Interviews5 minEncoding Categorical Variables4 min