Two Ways Data Can Move
Concepts covered: Batch vs Streaming
The last lesson gave every pipeline 4 roles: a source, a transform, storage, and a consumer. Rhythm is a different question: when does the transform run, and how much data does it handle each time? Every pipeline answers in one of 2 ways, and the answer sets the cost, the freshness of the output, and the kind of failure you will be paged for.
Batch: collect a pile, then process it
A batch pipeline lets records accumulate and processes them together on a schedule. Signups land in an application database all day, and nothing counts them. At 02:00 the next morning a job wakes up, reads every signup from the previous day, counts them by country, writes the counts to a table, and exits. The job has a start, an end, and a window of data it owns: the 24 hours before it ran. Between runs the transform is a definition waiting for its next trigger, with no process running anywhere.
Streaming: process each record as it arrives
A streaming pipeline keeps the transform running at all times and hands it records one at a time, within seconds of each record being created. The same signups flow into a queue. A consumer process reads the queue continuously, adds 1 to the count for that signup's country, and writes the new count to a serving table. The unit of work is a single record, so the output changes 6 times on a day with 6 signups, and the consumer is still running at midnight waiting for the seventh.
- signups (queue, Kafka)
- stream_counter (transform, streaming)
- live_counts (storage, serving table)
- nightly_batch (transform, Airflow)
- daily_counts (storage, warehouse)
- signups -> stream_counter
- stream_counter -> live_counts
- signups -> nightly_batch
- nightly_batch -> daily_counts
- Result: live_counts and daily_counts both say US 3, DE 2 and IN 1. live_counts had each record 2 s after it landed; daily_counts had all 6 only at Apr 25 02:08
Watch the clock in the animation. At 07:40 the second signup, from Germany, lands in the queue. Within 2 seconds live_counts shows DE at 1, while daily_counts holds nothing, because its job will not run for another 18 hours. At 02:00 on April 25 the nightly job fires, reads all 6 signups, and 8 minutes later writes US 3, DE 2, IN 1 in a single overwrite. Both tables now hold identical numbers. They arrived on 2 different rhythms, and for most of the day only one of them was right.
What changes, in one table
| Dimension | Batch | Streaming |
|---|---|---|
| Trigger | A time on a schedule | A record arriving |
| Unit of work | A window of records, such as one day | 1 record |
| Compute lifetime | Starts at the trigger, exits when finished | Started once, runs for months |
| Output write | The whole result, replaced each run | 1 row updated per record |
| Freshness | As old as the time since the last run | Seconds |
| Cost | Minutes of compute per run | Every hour of every day |
Freshness and lifetime
Two words from that table carry the rest of this lesson. Freshness is the delay between a record being created and its effect showing in the output. For s4, the signup from India at 15:30, that delay is 2 seconds in live_counts and 10 hours and 38 minutes in daily_counts. Compute lifetime is how long the transform exists as a running process. The batch job lives for 8 minutes a day. The stream consumer lives for all 1,440 minutes of it, including the roughly 1,434 in which no signup arrives. Freshness is what the consumer feels. Lifetime is what the bill feels.
Which of these 4 pipelines is batch?
- A fraud check that scores each card swipe within 200 milliseconds of the swipe reaching the bank.
- A counter that raises a city's active-driver total each time a driver taps the go-online button.
- A job that reads every order placed yesterday at 03:00 each morning and writes a totals table.
- A process that reads a queue of clicks continuously and appends each one to a log table.
About this lesson
This section is part of the Batch vs Streaming: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.