Batch Processing

The question arrives as 'walk me through how you would produce yesterday's sales numbers every morning,' or in its bluntest form, 'what is batch processing?' Either way the interviewer is checking the same thing: whether you can describe a job that takes a bounded set of data and runs until it has written a result someone reads later. The scoring sheet for this question is short, and most candidates lose points by skipping an item on it rather than by saying anything wrong.

What batch means

A batch is a bounded input. The job knows, before it starts, which data it will read, and it stops once that data is processed. That boundedness is what makes a batch job reliable to reason about: you can count the input rows, count the output rows, and compare the 2 numbers. Latency is measured in hours, and that's acceptable when the consumer is a report or a monthly invoice, anything that reads data on a schedule rather than reacting to it. A daily sales table rebuilt at 02:00 and read by analysts at 09:00 is the canonical example, and it is the one to reach for when the interviewer asks for one.

Each arrow carries 1 day of orders, selected by the run's date boundary.

orders database (source, transactional DB)
nightly extract (transform, orchestrator); freshness: complete by 06:00 UTC; retries: 2; backoff: 10 min wait
aggregate by store (transform, SQL step)
daily_sales (storage, warehouse table); backfill: rerun one date
morning dashboard (consumer, analysts at 09:00)

orders database -> nightly extract
nightly extract -> aggregate by store
aggregate by store -> daily_sales
daily_sales -> morning dashboard

Saying the boundary out loud

'Yesterday' means 2 different things to 2 different people. State it as a half-open interval: ordered_at >= '2026-10-18 00:00:00' and ordered_at < '2026-10-19 00:00:00', in UTC. The half-open shape means an order placed at exactly midnight lands in 1 day and never in both, and 2 adjacent runs cover every second without overlap. The query the job runs looks like this:

INSERT INTO daily_sales
SELECT
  CAST(ordered_at AS DATE) AS sale_date,
  store_id,
  COUNT(*) AS order_count,
  SUM(total_cents) AS revenue_cents
FROM orders
WHERE
  ordered_at >= TIMESTAMP '2026-10-18 00:00:00'
  AND ordered_at < TIMESTAMP '2026-10-19 00:00:00'
GROUP BY
  1,
  2

Before that insert runs, the job deletes any rows already sitting in daily_sales for October 18, and both statements run in 1 transaction, so a crash between them leaves the old rows and no reader sees the day empty or half-written. That pair is what makes the rerun answer correct: run the job once and the day has 1 set of rows, run it 3 times and it still has 1 set. Appending without the delete means a retry doubles revenue for that date, and the interviewer will ask what happens on retry whether you raise it first.

DELETE FROM daily_sales
WHERE
  sale_date = DATE '2026-10-18'

Where the daily job breaks

Break	What the dashboard shows	Property that prevents it
Retry after a partial failure	Revenue for the day doubled	Replace the date's rows instead of appending
Late data: an order committed at 23:59:58 reaches the extract after the run	Yesterday short by a few orders	Extract from a snapshot taken after a short lag, and allow a rerun for the date
Schema change: a column added to orders	The job fails, or `SELECT *` shifts values into the wrong columns	An explicit column list and a row count check against the source
Source unavailable at 02:00	No rows for the date at all	A retry after a wait, then an alert if it still fails before the 06:00 deadline

The answer that scores

Put together, the answer sounds like this. 'I would schedule a job at 02:00 UTC that selects orders with ordered_at inside the previous UTC day, aggregates them by store, deletes any existing rows for that date in the target table, and inserts the new ones. If it fails it retries twice with a 10 minute wait, and if it misses the 06:00 deadline it pages someone. Because the job replaces the day rather than appending, I can rerun any date safely, which is also how I would backfill a month.' That is 3 sentences, every item on the sheet covered, and the interviewer moves to the next question with nothing left to probe.

About this lesson

This section is part of the How Data Moves: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.