Streaming vs Batch Data

Two Data Processing Paradigms

Data pipelines feed machine learning models in one of two ways:

┌──────────────────────────┬──────────────────────────┐
│ BATCH DATA PROCESSING    │ STREAMING DATA PROCESSING│
├──────────────────────────┼──────────────────────────┤
│ Bounded historical data. │ Unbounded live events.   │
│ Scheduled intervals      │ Continuous processing    │
│ (Nightly Spark jobs).    │ (Kafka + Flink).         │
│ High throughput, 24h lag.│ Sub-second freshness!    │
└──────────────────────────┴──────────────────────────┘

1. Batch Data Processing

Accumulate incoming event logs into data lakes (S3) and process them in large scheduled bulk jobs:

• Tools: Apache Spark, Snowflake, BigQuery, AWS EMR.
• Latency: Hours to days (24 hour point in time lag).
• Pros: High computational throughput, simple error recovery, low infrastructure costs.
• Best for: Model training datasets, monthly reporting, non urgent batch recommendations.

2. Streaming Data Processing

Process individual event logs or micro-batches continuously as events occur in the real world:

• Tools: Apache Kafka, Apache Flink, Spark Structured Streaming.
• Latency: Sub-second (Milliseconds to Seconds).
• Pros: Real time feature freshness, immediate adaptation to user session shifts.
• Cons: High infrastructure complexity, state management challenges, complex out of order event handling.
• Best for: Real time fraud detection, ad click scoring, dynamic surge pricing.

Watermarks and Late Arriving Data

A major engineering challenge in stream processing is Out of Order & Late Arriving Data:

Real-world Event Occurs (Timestamp: 14:00:00) ──► Network Delay / Mobile Offline ──► Arrives at Flink (Timestamp: 14:15:00!)

Stream processors use Watermarks to define how long to wait for late arriving event data before closing time window aggregations.

The Lambda and Kappa Architectures

• Lambda Architecture: Runs both a Batch Layer (accurate, historical) and a Speed Layer (streaming, real time) in parallel, merging results at query time.
• Kappa Architecture: Replaces the batch layer entirely by using a single unified Stream Processing engine (Flink) for both real time and historical processing.

Say this out loud

Streaming vs Batch processing balances data freshness against system complexity and cost. Batch processing computes features over bounded historical data on scheduled cycles, providing high throughput at low cost. Streaming processing ingests unbounded event logs continuously using Kafka and Flink, delivering sub second feature freshness for real time prediction systems.

Followups to expect

1. What is Sliding Window vs Tumbling Window in Flink? Tumbling windows divide time into non-overlapping fixed blocks (for example 5 minute blocks). Sliding windows overlap continuously (for example last 5 minutes updated every 10 seconds).
2. How do you handle exactly once processing in streaming? Use distributed snapshot checkpointing (Chandy-Lamport algorithm in Flink) combined with idempotent writes to guarantee no event is counted twice.