Data & Feature Engineering

Spark & Distributed Data Processing

Processing multi terabyte feature engineering datasets in parallel across distributed cluster worker nodes using Apache Spark.

🔴 advanced5 min readdatainfra
Apache Spark provides in memory distributed data processing for large scale feature engineering. Single machine Pandas processing fails when datasets exceed local RAM memory limits. Spark partitions massive datasets across a cluster of worker nodes managed by a Driver node, performing parallel transformations, lazy evaluation, and distributed feature extraction across billions of rows.

Why Single Machine Processing Fails at Scale

Python Pandas processes datasets in local RAM memory on a single machine.

When feature datasets grow to hundreds of gigabytes or terabytes, Pandas runs out of memory and crashes (MemoryError).

Apache Spark is an open source engine for Distributed Data Processing:

                              [ DRIVER NODE ]
                         (Maintains Query DAG Plan)
                                     │
             ┌───────────────────────┼───────────────────────┐
             ▼                       ▼                       ▼
   [ EXECUTOR WORKER 1 ]   [ EXECUTOR WORKER 2 ]   [ EXECUTOR WORKER 3 ]
   (Processes Part 1)      (Processes Part 2)      (Processes Part 3)

Spark splits large datasets into Data Partitions and distributes processing across a cluster of worker machines in parallel.

Core Spark Architecture Concepts

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. SPARK DATAFRAME       │ 2. LAZY EVALUATION       │ 3. IN-MEMORY CACHING     │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Distributed table        │ Postpones execution until│ Stores intermediate      │
│ partitioned across       │ an Action is called,     │ feature Dataframes in    │
│ cluster worker nodes.    │ optimizing the query DAG.│ RAM across iterations.   │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Spark DataFrame

A distributed collection of data organized into named columns, similar to a relational database table or Pandas Dataframe, but partitioned across cluster machines.

2. Transformations vs Actions

3. In-Memory Computing

Unlike legacy Hadoop MapReduce (which wrote intermediate outputs to disk after every step), Spark keeps intermediate datasets in cluster RAM memory, enabling 100x faster feature processing speeds.

Say this out loud

Apache Spark processes large scale datasets across distributed worker clusters. A Driver node plans execution DAGs while Executor worker nodes process data partitions in parallel. By using in memory computing and lazy evaluation, Spark optimizes query plans and processes terabyte scale feature pipelines efficiently.

Followups to expect

  1. What is PySpark? The Python API for Apache Spark, allowing data scientists to write familiar Python code while executing distributed Spark jobs on backend JVM worker nodes.
  2. What is Catalyst Optimizer in Spark SQL? An automated query optimization engine that analyzes logical execution plans, reordering filters and joins to minimize computational cost.

Check yourself

Question 1 of 3

What is the primary architecture model of an Apache Spark cluster?

More in Data & Feature Engineering

See all →
Feature Engineering Fundamentals4 minSQL Questions in ML Interviews5 minEncoding Categorical Variables4 min