Designing for Reprocessability

Concepts covered: Late-Arriving Data

The widest form of this question drops the symptom entirely. 'The billing provider is resending the whole of Q3 with corrected tax. What do you do?' Every technique so far is a special case of one capability: can you recompute any derived partition from raw whenever you choose and get the same answer the original run would have produced from the same inputs? A pipeline that can is reprocessable, and reprocessability is designed in, in 5 places, long before the resend arrives.

5 design decisions

• Raw is append-only and partitioned by arrival: Every row keeps its original payload and its processed_ts. Reprocessing reads raw. It never reads a derived table, because a derived table may already carry a past mistake.
• Every derived partition is a function of raw rows in a declared window: The job takes a run date and a window as parameters. Rerun for 2026-07-14 with the same raw rows and it produces the identical partition, byte for byte.
• Keys are deterministic: A surrogate key issued by a counter differs on every rerun, and every fact pointing at it breaks. Derive keys from the natural key so a rebuilt dimension row gets the key it had before.
• Both clocks live on every row: Without processed_ts you can rebuild the present but never a past belief, and the past belief is what an audit asks for.
• A run log records what each run read: The log is how you answer 'what was in the 14 July partition on 15 July?' without guessing, and how you find every rollup that read a partition you are about to change.

Reading the table as it was

Because every row carries processed_ts, the same fact_playback partition answers 2 different questions. 'How many plays happened on 14 October?' reads the partition as it stands today. 'How many plays did the 15 October report show for 14 October?' adds a single predicate on the arrival clock.

SELECT
  event_date,
  count(*) AS plays_as_known
FROM fact_playback
WHERE event_date = date '2026-10-14'
  AND processed_ts < timestamp '2026-10-15 06:00:00'
GROUP BY event_date

The second query reproduces a published number from a table the lookback has since rewritten 3 times. That holds because the rewrite keeps the first arrival per event_id, with its original processed_ts, and because corrections arrive as reversal rows with their own arrival clock rather than as edits. Remove either property and the as-of read is gone. A table with this property is bitemporal: one axis for when it happened, one for when you knew, and a report is a cut along the second.

Scoping the reprocessing request

Back to the Q3 resend. The answer is a blast radius computed from the lineage, and then a conversation with the people inside it.

Each step leans on a design choice above. Raw makes step 1 an append. Reversal rows guarded by source_version make step 2 a loop over 92 dates that is safe to rerun. The run log names the rollups that read those partitions. Step 4 works because each original amount stays in fact_charge beside its reversal, so an as-of cutoff on processed_date still returns the old total that a rebuild in place would erase. The compute takes hours. Without the old totals, the restatement conversation with finance takes weeks.

About this lesson

This section is part of the Late-Arriving Data: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.