Hybrid Architectures

The hybrid question is always a follow-up. You propose a streaming job for the live dashboard, the interviewer nods, and then: 'Your stream processor shipped a bug on Tuesday and has been emitting wrong totals for 3 days. How do you fix history, and what does the dashboard show while you do?' The grading is about the seam: which path owns which data, and who wins when the 2 disagree.

The 2 shapes a fix can take

Both shapes answer the Tuesday bug, with different bills. In the 2-path shape the seam has already contained the damage. The batch job is an independent implementation, and at 02:00 each night it replaced the stream's Tuesday and Wednesday with its own output from the raw store, so the closed days are correct and only today's stream output is wrong. The correction is to fix and redeploy the stream code and recompute today; rerunning batch over the affected days matters only when the bug lives in the batch job or in logic both paths share. The replay shape has 1 implementation, so all 3 days carry the bug. It resets the stream job to Tuesday morning's offset, writes into a fresh output table, and swaps it in once it has caught up to the present. Replay requires that the log still holds Tuesday, and that's a storage decision someone made months earlier.

• 400 GB
• 12 TB

The seam

The seam in that diagram is a date boundary. The stream job writes continuously into its own table, and the batch job owns every closed day: at 02:00 it recomputes yesterday from the raw store into the batch table, and from then on the view serves that day from batch. Batch wins on disagreement because it runs later, sees the late events that arrived after midnight, and runs the reviewed code path. Events that land after 02:00 for a day already closed are the late-data case. The batch job reprocesses 1 extra day of lookback on every run, which corrects events up to about 1 day late and no later. The 02:00 run on 20 October recomputes the 19th and the 18th; an event for the 18th that lands at 09:00 on the 20th is never picked up, because the next run covers the 20th and the 19th. Older stragglers need a wider lookback or a backfill triggered for their date, and that width is the number to defend, as with the high-water mark: each extra day rereads another 400 GB from the raw store on every run.

WITH boundary AS (
  SELECT
    max(event_date) AS last_batch_date
  FROM gold.daily_plays_batch
)
SELECT
  event_date,
  plays
FROM gold.daily_plays_batch
UNION ALL
SELECT
  s.event_date,
  s.plays
FROM gold.daily_plays_stream AS s
CROSS JOIN boundary AS b
WHERE s.event_date > b.last_batch_date

The serving view is where the paths touch, and the interviewer checks the filters. If both branches could return the same date, totals double for that day. A view that hard-codes current_date as the boundary has the opposite bug: between midnight and the 02:00 batch, yesterday belongs to neither branch and drops off the dashboard, and a batch job 3 hours late stretches the hole past 5 hours. Keying the boundary on the latest date the batch table holds lets the stream serve every day batch has not closed, and a day changes owner the moment batch writes it. That answers 'what does the dashboard show while you fix it': the stream's number until batch replaces it, never a blank day and never a doubled one.

• A named boundary: which partition, date range or table each path owns.
• A tiebreak with a reason: batch wins because it runs later and sees late data.
• A correction route with a duration: rerun 3 days of batch, or replay from an offset.
• A storage bill: the raw store the batch reads, or the log retention the replay needs.

About this lesson

This section is part of the How Data Moves: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.