Idempotent Pipelines

Concepts: paBatchProcessing

What They Want to Hear 'An idempotent pipeline produces the same result whether it runs once or five times on the same input. I achieve this with MERGE statements that upsert on a primary key, or by replacing entire partitions on each run. This means every retry, every backfill, and every re-run is safe.' This is the answer that shows production experience. Candidates who say 'just make it transactional' are missing the point. Three Idempotent Patterns