Batch: Picture, Rhythm, Example

Concepts covered: Batch Processing

Batch processing is the older of the 2 rhythms and still the dominant pattern in production. Most analytical work in most companies runs as a batch job, often nightly, sometimes hourly. The pattern is so common that the word pipeline used without qualification almost always means a batch pipeline. Knowing the shape of a batch run cold is the foundation for everything else, because streaming is largely defined by what it changes about that shape.

The shape of a batch run

The nightly run

The canonical example is the nightly run. Sometime between 1am and 4am Pacific, when production traffic is at its lowest, an orchestrator wakes up dozens or hundreds of jobs in dependency order. Each job reads its inputs, applies its transforms, and writes its outputs. The work finishes by 6am or 7am, in time for the morning consumers to read fresh data. By 9am the dashboards show the previous day's numbers, and the cycle repeats the next night.

-- A typical nightly batch transform
-- Runs once at 2am Pacific, computes yesterday's signups by country
INSERT INTO mart.daily_signups (signup_date, country, signup_count)
SELECT
  DATE(signup_timestamp) AS signup_date,
  country_code,
  COUNT(*) AS signup_count
FROM raw.signups
WHERE signup_timestamp >= :run_date::date
  AND signup_timestamp <  :run_date::date + INTERVAL '1 day'
GROUP BY 1, 2;

The query reads exactly one day of data, aggregates it, and writes the result. The next day's run does the same for the next day. This pattern has run nearly unchanged in data warehouses since the 1990s because the basic shape is right: read a window, transform it, write a partition, stop.

Batch pays only while it runs

Batch jobs spin compute up at the start of the run and shut it down at the end. A Snowflake warehouse charges by the second of compute time. A nightly job that runs for 40 minutes pays for 40 minutes of compute, period. The other 23 hours and 20 minutes of the day cost nothing. Cloud compute is elastic specifically so that batch workloads can ramp up to large clusters when work is happening and ramp down to zero between runs. The cost-per-unit-of-data of a well-designed batch pipeline is the lowest of any architecture.

23h 20min Idle time per day for a 40-minute nightly batch: Compute is paid for only during the 40-minute run; the rest of the day costs nothing

Common batch cadences

The freshness floor of batch

Batch cannot beat its own schedule. A daily pipeline can't tell anyone what happened in the last hour. An hourly pipeline can't tell anyone what happened in the last minute. The freshness floor is the schedule itself. For a finance team that reads the dashboard once a morning, that floor is fine. For a fraud team that needs to react to a suspicious transaction within seconds, the floor is unusable. The mismatch between consumer freshness needs and batch cadence is the most common reason streaming gets pulled into a system.

Do

• Use batch as the default unless a specific consumer cannot wait for the next scheduled run
• Match the cadence to the consumer's freshness need; don't over-run if hourly is enough
• Partition outputs by the run window so failed runs replay one partition cleanly

Avoid

• Reach for streaming because it sounds more modern; the cost difference is real
• Run a batch job continuously by scheduling it every minute; that is just expensive streaming
• Confuse a slow batch with a streaming need; profiling first beats redesigning later

About this lesson

This section is part of the Batch vs Streaming: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.