Batch vs Stream Processing: When Data Cannot Wait

Batch vs Stream Processing is a choice about how long an answer can wait. A batch job collects data and works on it later, on a schedule. A stream job works on each event as it arrives. Both are good, and each has a price.

Two Ways to Handle Events

An event is a small record of something that happened, such as an order, a click or a sensor reading. A busy system creates thousands of them every minute. Sooner or later, someone wants totals, trends or alerts from those events.

You have two choices. Wait until you’ve collected a pile, then process the whole pile. Or process each event as it shows up. The first is batch, and the second is stream. The words describe when the work happens, not how much data there is.

Batch Processing: Wait, Then Work

A batch job reads a bucket of data and processes it in one go. Say a grocery store collects orders all day, and a job runs at the top of every hour. It reads the last hour of orders, adds them up and writes a report.

Batch is simple. The input is a fixed set of files or rows. You can test it, rerun it after a failure and compare two runs. It’s also efficient, because a big job reads data in large chunks, and the cluster stops when the job ends. Spark batch jobs on Parquet or Delta Lake tables work this way.

The price is delay. An order placed one minute after the job starts waits almost an hour for the next run. Then it waits for the job to finish.

Stream Processing: Work as It Arrives

A stream job never finishes. Events flow into a log such as Apache Kafka. A stream processor, such as Apache Flink or Spark Structured Streaming, reads them as they come. The totals update continuously, so a dashboard stays current within seconds or minutes.

Because a stream has no end, you can’t add up everything. You group events into windows instead. A tumbling window is a fixed slice of time with no overlap, for example five minutes. The processor counts the orders in each slice and publishes the result when the slice closes.

One Clock, Two Answers

The diagram shows both styles on the same clock. Events arrive all day along the top. In the batch lane, events wait in a bucket until the job runs, and the result arrives late. In the stream lane, each event is handled as it arrives and grouped into five-minute windows. The result updates continuously, and the gap between the lanes is the delay.

Here are six orders from the grocery store. Assume the batch job runs on the hour and takes 10 minutes. Assume the stream windows wait one extra minute for late events.

The batch average delay is about 50 minutes. The stream average is about 4 minutes. The 9:02 order waited longest in the batch, because it came in right after its hour began.

The Cost of Real Time

A faster answer isn’t free, and batch vs stream processing is partly a question of cost. A batch job runs and stops. A stream job runs all day, so it needs servers, monitoring and someone to look at it when it breaks.

Streams also ask for harder design. You must handle events that arrive twice, out of order or late. You must keep state, such as the running count in each window, and recover it after a crash. Testing is harder, because there’s no fixed input to replay, unless the log keeps the events.

Batch also wins on history. When a rule changes, you rerun last year’s data with the same job. A stream can replay its log too, but only as far back as the log keeps events.

Windows bring a new problem. Events can arrive late, after their window has closed. A window can follow the time an event happened, called event time, or the time it arrived. Late events matter only for event time. The rule for how long to wait for them is called a watermark. I explain how events get into a platform in Data Ingestion: Batch Loads, Change Data Capture and Streams.

Then there’s the question of value. A result that arrives in four minutes instead of fifty only matters if someone acts on it in that time.

How to Choose

Batch vs stream processing comes down to one question: what decision changes if the answer is an hour old? A monthly revenue report doesn’t change. A fraud check, a delivery map or a server alert does. Those need a stream. Most reports and dashboards can wait for a batch.

Many platforms run both. A stream gives fast numbers, and a nightly batch recalculates them with late events included. If a batch every minute is fast enough, you can also run small, frequent batches and skip the stream’s complexity. Spark Structured Streaming works this way by default, running small batches.

Why not build the stream every time? You can roll a stream up into batch totals later, and that’s true in principle. In practice you pay for the always-on system every day, even for numbers nobody looks at until Monday.

What to Remember

Batch vs stream processing is a choice about delay. Batch waits and then works, and it’s simple and cheap. Stream works as events arrive, and it’s fast and demanding. The delay you can accept decides between them.

When I review a design, I ask each team how old a number can be before it stops being useful. When the answer is a day, a nightly batch is enough.

Real time is not a feature you switch on, it is a price you decide to pay.

Published by Pinal Dave on SQLAuthority. More of my work at pinaldave.com.