Micro-Batch: The Middle Ground

Micro-batch is batch run on a short, fixed timer against a queue. Every trigger interval, say 60 seconds, the job wakes, reads everything that arrived on the queue since the previous trigger, transforms it, writes one output, records how far it read, and sleeps until the next tick.

Kept from batch

• Discrete runs, each with a start, an end and one output
• Fixed per-run overhead shared across many rows
• A run can be repeated by rereading the same queue range

Taken from streaming

• A queue as the source rather than a daily file or table snapshot
• A job that never exits and never waits for a scheduler
• Freshness measured in seconds or minutes rather than hours

The interval sets the latency

Worst-case latency for a row is the full interval plus the processing time of the batch it lands in. A row that arrives 1 second after a tick waits 59 seconds for the next one, then waits for that batch to finish. Processing time grows with the batch, so the table below assumes 6 seconds of fixed overhead per run plus 200 rows a second, on 4M rows a day. At a 60 second trigger each batch takes around 20 seconds, so latency ranges from 20 to 80 seconds.

Shrinking the interval hits a floor: near 6 seconds, the fixed overhead alone fills each slot and runs fire back to back. Processing rate is the other limit. This job processes 200 rows a second against an average input of 46. If the busiest minute brings 250 rows a second, each batch takes longer than the time it covers, every tick starts later than the last, and the backlog grows until the peak passes. Size processing for the busiest minute of the day, not the average one.

import time

TRIGGER_SECONDS = 60
offset = load_committed_offset()

while True:
    tick = time.monotonic()
    events, new_offset = read_queue(after=offset)
    if events:
        rows = transform(events)
        write_batch(rows)
        offset = new_offset
        commit_offset(offset)
    elapsed = time.monotonic() - tick
    time.sleep(max(0, TRIGGER_SECONDS - elapsed))

The offset is committed only after the batch is written, so a crash between the 2 writes replays the same events on restart rather than skipping them. The sleep is computed from the elapsed time, so a 20 second batch is followed by a 40 second sleep and the ticks land on the minute. An empty read still advances the clock: a quiet queue produces no output but costs a wake-up.

Fixed write overhead, shared across a batch

Every write to the curated layer carries fixed overhead that is the same for 1 row or 1M: opening a connection, creating a file, committing a transaction, updating a listing. Pure streaming pays that overhead once per event or once per tiny buffer. Micro-batch pays it once per interval and spreads it across every row in the batch. At 1 minute and 2,778 rows per run, the fixed cost per row is 2,778 times smaller than paying it per event. That amortization is why micro-batch holds most of streaming's freshness at a fraction of its cost, as long as the consumer tolerates a minute.

A micro-batch job on a 30 second trigger processes a typical batch in 12 seconds, but between 18:00 and 19:00 the input quadruples and each batch takes 45 seconds. The consumer needs rows within 2 minutes. What happens during that hour, and what is the smallest fix?

• Latency rises to 75 seconds and holds there; the 2 minute need is met and no change is needed.
• The backlog grows all hour; lengthen the interval to 60 seconds so each run has more time.
• The backlog grows all hour and latency climbs past 2 minutes; add parallel workers so the peak batch finishes in under 30 seconds.
• Nothing changes, because micro-batch reads everything since the last tick regardless of size.

About this lesson

This section is part of the Batch vs Streaming: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.