Micro-Batch: The Middle Ground

Concepts covered: Micro-Batch vs True Streaming

Most production pipelines that look like streaming are not pure streaming. They are micro-batch: very small batches, often every few seconds or every minute, processed by an engine that exposes a streaming API on top. Spark Structured Streaming is the largest example. Flink can run in batch or streaming mode with a tunable trigger interval. The pattern exists because pure streaming is expensive and pure batch cannot meet sub-15-minute freshness. Micro-batch sits in the middle: latency low enough to feel real-time, throughput high enough to amortize per-batch overhead, complexity simpler than full streaming.

How Micro-Batch Works

Aspect	Pure Streaming	Micro-Batch
Trigger	Every event arrival	A configured interval (10 sec, 1 min, 5 min)
Per-event work	Paid per event	Paid per micro-batch; amortized within the batch
End-to-end latency	Bounded by single-event processing time (100ms to a few seconds)	Bounded by the trigger interval (10 sec to 5 min)
Engine examples	Flink, Kafka Streams, custom	Spark Structured Streaming, Flink in batch mode, dbt incremental
State handling	Continuous in-memory state	State checkpointed each batch; smaller surface for bugs

The Spark Structured Streaming Pattern

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, count

spark = SparkSession.builder.getOrCreate()

events = (
    spark.readStream
         .format('kafka')
         .option('subscribe', 'raw.events')
         .load()
         .select(parse_json(col('value')).alias('event'))
)

running = (
    events.groupBy('event.country')
          .agg(count('*').alias('events'))
)

query = (
    running.writeStream
           .outputMode('complete')
           .format('delta')
           .option('checkpointLocation', 's3://chk/country_counts')
           .trigger(processingTime='1 minute')  # micro-batch interval
           .start('s3://lake/country_counts')
)

The trigger argument is the key knob. Setting processingTime to one minute tells Spark to process whatever has accumulated in the last minute and write the result, then wait for the next minute. The application code does not change. The latency-throughput tradeoff is exposed as a single number. A trigger of one second behaves more like streaming and costs more. A trigger of ten minutes behaves more like batch and costs less. Production pipelines tune the trigger to the actual freshness requirement and stop there.

Why Micro-Batch Exists

Operational simplicity: A failed micro-batch reruns from a checkpoint. There is no in-flight event hanging in the system. Recovery has the same shape as a batch job that failed and restarted.
Throughput at low latency: A 1-minute micro-batch processing 60K events per minute amortizes the per-batch overhead 60K times. The throughput-per-dollar approaches batch while latency stays sub-2-minute.
Tooling continuity: Spark and Flink let the same SQL or DataFrame code run as batch or as micro-batch. Migration between modes does not require a rewrite, only a config change.

When Micro-Batch Is the Right Answer

Micro-batch is right when the freshness target is between one minute and fifteen minutes, the volume is high enough to amortize per-batch overhead, and the team prefers operational simplicity over the absolute lowest latency. Most live operational dashboards, fraud retrospective pipelines, and CDC consumers fit this description. Setting up Spark Structured Streaming or Flink in batch mode with a tunable trigger gives most of the latency win of streaming for most of the cost win of batch. For a freshness target of one second or below, micro-batch is too slow; pure streaming is required. For a freshness target of one hour or above, hourly batch is cheaper; micro-batch is overkill.

When Micro-Batch Is the Wrong Answer

Situation	Why Micro-Batch Fails	Better Choice
Sub-second latency target	Trigger interval cannot go below the per-batch overhead	Pure streaming with Flink, Kafka Streams, or custom
Very low volume	Per-batch overhead dominates; cost is wasteful	Hourly batch is cheaper and simpler
Per-event side effects	Batches can fail and reprocess; side effects must be idempotent	Pure streaming with at-least-once semantics and idempotent sinks
Unbounded state	Each batch must checkpoint state; growth is not sustainable	True streaming with key-managed state and TTLs

Tuning the Trigger Interval

The trigger interval is the most important knob in a micro-batch pipeline. Setting it too low forces per-batch overhead to dominate cost. Setting it too high pushes latency above the consumer's threshold. The right interval is usually the largest value the consumer can tolerate, because larger intervals cost less. A consumer asking for under 5 minutes is best served by a 4-minute trigger, not a 30-second trigger. The 30-second version is technically faster and costs five to ten times more. Naming the consumer's actual freshness floor pushes the interval up to the right place.

Spark Structured Streaming defaults to a 'micro-batch ASAP' mode if no trigger is set. The default is rarely right; it produces tiny batches with high per-batch overhead. Always set processingTime explicitly.

Do

Use micro-batch as the default for tier 2 freshness (under 15 minutes)
Set the trigger interval to the largest value the consumer accepts
Treat checkpointed state as recoverable; design transforms to survive a restart

Avoid

Default to a sub-second trigger because lower sounds better; cost compounds fast
Use micro-batch when the freshness target is sub-second; the architecture cannot reach it
Mix per-event side effects with micro-batch without idempotent sinks; replays will duplicate

About this lesson

This section is part of the Batch vs Streaming: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.