When Batch Outgrows Itself

Concepts covered: Batch vs Streaming

The exercise below walks through a real-shaped scenario: a pipeline that started as nightly batch, grew, and stopped meeting its freshness target. The redesign is not a wholesale switch to streaming. The redesign is a careful examination of which dimension is failing and the smallest change that fixes it. Most batch-to-streaming migrations in production look like this exercise, not like a rewrite.

The Starting Pipeline

An e-commerce company's nightly pipeline reads orders from a Postgres database, joins with a product catalog, computes daily aggregates by category and region, and writes a fact_daily_orders table consumed by the executive dashboard. The pipeline runs at 2am, finishes by 5am, and the dashboard is fresh by 6am Pacific. Volume is 4 million orders per day. Compute cost is roughly $40 per night. The architecture has worked unchanged for two years.

-- The original nightly transform
INSERT INTO mart.fact_daily_orders
SELECT
  DATE(o.order_timestamp) AS order_date,
  p.category,
  o.region,
  COUNT(*) AS orders,
  SUM(o.amount_cents) / 100.0 AS revenue
FROM raw.orders o
JOIN dim.product p ON o.product_id = p.product_id
WHERE DATE(o.order_timestamp) = :run_date
GROUP BY 1, 2, 3;

The Symptoms

The Wrong Instinct

The product team's first proposal is to migrate everything to a Flink streaming pipeline. The proposal sounds reasonable: streaming is real-time, batch is slow, the freshness problem is obvious. The proposal is wrong. Migrating a billion-row-per-day join to streaming costs roughly 20x the current spend, takes nine months of engineering, and requires building state management for a stateful join that does not currently exist. The freshness problem is real but the diagnosis is incomplete. The actual problem is two separable issues that look like one.

The Right Diagnosis

The Redesign

The redesign keeps batch where it works and adds streaming only where the consumer requires it. The executive dashboard stays tier 4 (daily) and uses an hourly micro-batch instead of a single nightly run. The marketing team gets a tier-2 path that uses Spark Structured Streaming with a 1-minute trigger reading from a CDC stream of the orders table. The two paths share the curated layer concept from Lesson 1: both write to the same fact_daily_orders table family, partitioned in a way that lets each path update only its own partitions. The redesign respects the freshness tiers; it does not impose one rhythm on consumers who want different things.

# Path 1: Hourly micro-batch (executive dashboard, tier 4)
# Spark batch job, runs at the top of every hour
# Reads only the last hour, writes hourly partition
spark.sql('''
  INSERT OVERWRITE mart.fact_hourly_orders PARTITION (hour_ts = :run_hour_ts)
  SELECT
    DATE_TRUNC('hour', o.order_timestamp) AS hour_ts,
    p.category,
    o.region,
    COUNT(*) AS orders,
    SUM(o.amount_cents) / 100.0 AS revenue
  FROM raw.orders o
  JOIN dim.product p ON o.product_id = p.product_id
  WHERE o.order_timestamp >= :run_hour_ts
    AND o.order_timestamp <  :run_hour_ts + INTERVAL '1 hour'
  GROUP BY 1, 2, 3
''')

# Path 2: Streaming for marketing (tier 2, micro-batch every 1 min)
# Reads CDC stream of the orders table
stream = spark.readStream.format('delta').load('s3://cdc/raw_orders')
live = (
    stream.join(broadcast(product_dim), 'product_id')
          .groupBy(window('order_timestamp', '1 minute'),
                   'category', 'region')
          .agg(count('*').alias('orders'),
               sum('amount_cents').alias('cents'))
)
live.writeStream.outputMode('update') \
    .trigger(processingTime='1 minute') \
    .toTable('mart.live_minute_orders')

The Result

Before

• One nightly batch, 11 hours, $40 per run
• Tier 4 for everyone; tier 2 needs unmet
• Marketing built shadow pipeline that drifts from canon
• On-call paged 3-4 mornings a week

After

• Hourly micro-batch (5 min/run, $4) plus streaming micro-batch (1-min trigger, $200/day)
• Tier 4 for executive dashboard; tier 2 for marketing dashboard; both consumers happy
• One canonical fact table family; shadow pipeline retired
• On-call paged less than once a week; both paths run independently

What This Example Teaches

The example is not a story about streaming winning. It is a story about diagnosing which dimension is failing. Volume outgrew cadence; the fix was a tighter cadence, not a wholesale rhythm change. The non-incremental transform read a full day every time; the fix was incrementality, not a new engine. Different consumers had different freshness needs; the fix was to give each consumer the rhythm it actually needs, not a single shared rhythm. Senior engineers fix freshness problems by naming the dimension that is failing. The wrong instinct is to swap the architecture; the right one is to swap the smallest piece that solves the actual problem.

About this lesson

This section is part of the Batch vs Streaming: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.