Batch: Picture, Rhythm, Example
Concepts covered: Batch Processing
A batch pipeline has 3 moving parts beyond the data itself: a scheduler that knows when to fire, a job that does the work, and the storage the job reads from and writes to. The scheduler is the only part awake all the time, and its whole task is to watch a clock. The job is where compute starts, runs, and shuts down, and that lifecycle is what this section traces, minute by minute.
- signups db (source, application database)
- nightly count job (transform, scheduler)
- daily_counts (storage, warehouse table)
- morning dashboard (consumer, dashboard)
- signups db -> nightly count job
- nightly count job -> daily_counts
- daily_counts -> morning dashboard
One run, minute by minute
| Clock | What happens | Compute state |
|---|---|---|
| 02:00:00 | The scheduler reaches the trigger time and submits the job | Starting: a machine is requested |
| 02:01:00 | The job's process is up and connects to the signups database | Running |
| 02:01:30 | It reads every signup dated April 24 | Running: reading |
| 02:06:00 | It counts the rows per country in memory | Running: computing |
| 02:07:30 | It overwrites `daily_counts` with the new rows | Running: writing |
| 02:08:00 | The job reports success and its process exits | Shut down |
| 02:08 until 02:00 next day | Nothing runs. The scheduler watches the clock | Off |
Three points on that timeline matter more than the rest. Compute starts at 02:00, when the scheduler submits the job and a machine is allocated. Compute runs from 02:01 to 02:08, and this is the only period in which data is touched. Compute shuts down at 02:08, and from then until the next trigger the pipeline costs nothing and does nothing. A batch pipeline is defined as much by the 23 hours and 52 minutes it is off as by the 8 minutes it is on.
- 0.6%
- 99.4%
The window is the boundary
SELECT
country,
COUNT(*) AS signups
FROM signups
WHERE signup_date = DATE '2026-04-24'
GROUP BY countryThe WHERE clause draws the window. The run at 02:00 on April 25 owns signups dated April 24 and nothing else. A signup at 23:59 on April 24 is counted 2 hours and 9 minutes after it happened. A signup at 00:01 on April 25 belongs to the next window and waits until 02:08 on April 26, which is 26 hours and 7 minutes. Two records created 2 minutes apart show up in the output 24 hours apart, and that gap is the price of a daily window.
Every run replaces its result
The animation wrote daily_counts in overwrite mode, and that is the usual batch choice. Each run produces the complete answer for its window and replaces whatever the previous run wrote for it. If the job fails at 02:05 and someone restarts it at 02:20, the second attempt reads the same April 24 rows, computes the same counts, and leaves the table exactly as a successful first attempt would have. A batch run is a unit you can repeat, because its window closed when the day ended.
- Starts: the scheduler fires at the trigger time and a machine is allocated to the job.
- Runs: the job reads its window, computes, and writes, all inside one process.
- Shuts down: the process exits, and compute cost drops to zero until the next trigger.
- Repeats: the next run owns the next window and replaces its own output.
A user signs up at 02:03 on April 25, while the nightly job is already running. When does that signup first appear in daily_counts?
- At 02:08 on April 25, because the running job picks up every row present at the moment it writes.
- At 02:08 on April 26, because the row is dated April 25 and belongs to the next run's window.
- Within 2 seconds, because the table is updated as soon as each record arrives in the source.
- Never, because rows created while a run is in progress fall between windows and are skipped.
About this lesson
This section is part of the Batch vs Streaming: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.