Making Pandas Fast

Why Pandas Pipelines Become Slow

Pandas is convenient, but unoptimized code can run for hours and crash due to out of memory errors (OOM).

Common causes of slow Pandas code:

• Iterating through DataFrame rows with df.iterrows().
• Storing text strings as generic object datatypes.
• Keeping 64-bit float numbers when 32-bit floats are sufficient.

Unoptimized Pandas: Slow iterrows() loops + Object dtypes ──► Hours of processing & RAM crashes!
Optimized Pandas:   Vectorization + Category dtypes + Polars ──► Seconds of processing & 80% RAM reduction!

4 Rules for Making Pandas Fast

┌──────────────────────────┬──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. NEVER USE ITERROWS    │ 2. VECTORIZE OR USE NUMPY│ 3. USE CATEGORY DTYPES   │ 4. DOWNCAST NUMERIC TYPES│
├──────────────────────────┼──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Replace iterrows() loops │ Use built-in vectorized  │ Convert repetitive text  │ Convert int64/float64 to │
│ with vectorized operations│ methods or pass arrays to│ strings to 'category' to │ int32/float32 to cut     │
│ or np.select().          │ C-compiled NumPy!        │ cut RAM by up to 90%!    │ RAM memory in half!      │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Eliminate iterrows() and apply()

# TERRIBLE: Slow iterrows() loop (~30 seconds for 1,000,000 rows)
for idx, row in df.iterrows():
  df.at[idx, 'total'] = row['price'] * row['quantity']

# GOOD: Vectorized operation (~0.003 seconds for 1,000,000 rows!)
df['total'] = df['price'] * df['quantity']

# GOOD: Vectorized conditional logic using np.select()
conditions = [df['age'] < 18, df['age'] < 65]
choices = ['minor', 'adult']
df['age_group'] = np.select(conditions, choices, default='senior')

2. Downcasting Numeric Types and Categoricals

Default Pandas reads numbers as int64 ($8$ bytes) and float64 ($8$ bytes), and strings as object ($28+$ bytes per string pointer).

# Convert repetitive string columns to 'category'
df['user_state'] = df['user_state'].astype('category')

# Downcast float64 to float32
df['score'] = df['score'].astype('float32')

Converting a 1,000,000 row text column to category reduces RAM usage from 80 Megabytes to 1 Megabyte!

3. Alternative High Performance Engines (Polars / DuckDB)

For massive datasets ($> 10\text{ GB}$), replace Pandas with Polars:

import polars as pl

# Polars runs multi-threaded in Rust with lazy query optimization!
df = pl.read_csv('massive_data.csv')
result = (
    df.lazy()
    .filter(pl.col('age') > 18)
    .group_by('country')
    .agg(pl.col('amount').mean())
    .collect()
)

Polars executes query plans using all CPU cores in parallel, outperforming Pandas by 10 to 30 times!

Say this out loud

Making Pandas fast requires eliminating row iteration loops in favor of vectorized operations. Converting text strings to categorical datatypes and downcasting float64 to float32 cuts RAM memory footprint by up to 90 percent. For massive datasets, switching to multi-threaded Rust engines like Polars multiplies query performance.

Followups to expect

1. What is eval() and query() in Pandas? Methods that compile string expressions using numexpr to execute element-wise array operations without allocating intermediate memory arrays.
2. What is Modin? A drop-in replacement library for Pandas that automatically distributes DataFrame computations across all available CPU cores using Ray or Dask backends.