Parquet & Columnar Storage
Understanding why columnar binary storage formats outperform row oriented CSV text files for feature engineering and analytical querying.
Row-Oriented vs Column-Oriented Storage
How a file format lays data out on disk determines query performance:
Row-Oriented (CSV, JSON, PostgreSQL):
[Row 1: ID, Age, Salary, Country] [Row 2: ID, Age, Salary, Country] ...
Column-Oriented (Apache Parquet, ORC):
[All IDs...] [All Ages...] [All Salaries...] [All Countries...]
Why Columnar Layout Wins for Machine Learning
Imagine querying a 100-column dataset to extract 3 feature columns (Age, Income, Credit_Score) for model training.
┌──────────────────────────┬──────────────────────────┐
│ ROW-ORIENTED (CSV) │ COLUMNAR (PARQUET) │
├──────────────────────────┼──────────────────────────┤
│ Must read ALL 100 columns│ Reads ONLY the 3 requested│
│ from disk byte by byte! │ column byte blocks! │
│ 97% of disk I/O wasted! │ Zero wasted disk I/O! │
└──────────────────────────┴──────────────────────────┘
Columnar storage reduces disk I/O by $90%+$ for feature selection queries!
3 Core Features of Apache Parquet
┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. DICTIONARY ENCODING │ 2. MIN/MAX SKIPPING │ 3. BIT PACKING & SNAPPY │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Replaces repetitive text │ Page headers store Min │ Compresses similar data │
│ strings (e.g. "California")│ and Max values for each │ types efficiently using │
│ with small 1-byte keys. │ column to skip blocks! │ Snappy compression. │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘
1. Dictionary Encoding
In a column containing state names ("California" repeated 1,000,000 times), Parquet creates a small dictionary lookup table (0: "California") and stores tiny 1-byte integer keys instead of full text strings.
2. Min/Max Statistic Predicate Pushdown
Parquet files split data into Row Groups ($128\text{ MB}$ chunks). Each Row Group stores metadata headers listing Min and Max values for every column:
$$\text{Query: } \text{WHERE Age} > 80 \quad (\text{Row Group Header: Age Min}=18, \text{Max}=65) \implies \text{SKIP FULL BLOCK!}$$
Query engines skip reading entire file blocks from disk without opening them!
3. High Compression Ratios
Because an entire column block contains identical datatypes (all Float32 numbers or all Booleans), compression algorithms like Snappy or ZSTD compress columnar data far more effectively than mixed-type CSV text rows.
Say this out loud
Apache Parquet is a columnar binary file format optimized for analytical feature engineering. By storing data organized by column rather than by row, query engines read only the specific feature columns requested while skipping unneeded data. Dictionary encoding, Snappy compression, and Min/Max page statistic pushdowns achieve 10x faster query speeds and 75% smaller storage sizes compared to CSV files.
Followups to expect
- What is Apache ORC? Optimized Row Columnar format, a high performance columnar file format developed for Apache Hive workloads, similar in design and benefits to Apache Parquet.
- Why avoid using Parquet for OLTP web applications? Inserting single individual rows into Parquet files requires rewriting entire row groups, making columnar formats poor choices for transactional database writes.
Check yourself
Why do analytical feature engineering queries run significantly faster on Parquet columnar files than on CSV row files?