Make It Reliable

The production test - idempotency is the #1 senior signal

Challenges

0 hands-on challenges

Lesson Sections

1. How Do You Ensure Data Quality?

   Data SLOs, Cost-of-Error Analysis, and Quality Regression At intermediate, quality gates are pass/fail. The interviewer wants to hear you talk about SLOs and cost models. The trap is saying 'we check for nulls.' The senior signal is: 'We defined a completeness SLO of 99.95% for revenue-critical tables, with a cost model that estimates $12K per hour of underreported revenue, which determines our alerting threshold and on-call response time.' A Data SLO is a measurable quality target: 'fewer than

2. Can You Re-run This Safely?

   Distributed Idempotency and Cross-System Consistency At intermediate, idempotency is a single MERGE statement. The interviewer then asks: 'How do you coordinate idempotency across Kafka, Spark, and your warehouse when each can fail independently?' This is the distributed systems question that separates good from great candidates. The core problem you need to articulate: in a distributed pipeline, you can't use a single database transaction to span multiple systems. A Kafka consumer might commit

3. How Do You Handle Duplicates?

   Probabilistic Dedup at Scale At intermediate, dedup is ROW_NUMBER over a table that fits in memory. The next question the interviewer asks: 'How do you deduplicate 10 billion events per day when holding all keys in memory is impossible?' This is where you need to talk about probabilistic data structures. The trap is saying 'just use a bigger cluster.' The senior signal is naming Bloom filters and explaining the false-positive tradeoff. A Bloom filter answers one question: 'Have I seen this key b

4. How Do You Know It's Broken?

   ML Anomaly Detection and Automated Runbooks At intermediate, monitoring is static thresholds. The follow-up: 'Static thresholds break on seasonal data, holiday spikes, and organic growth. How do you handle that?' The trap is saying 'we adjust the thresholds manually.' The senior answer is ML-driven anomaly detection that adapts to patterns humans can't manually encode. The simplest ML anomaly detector that actually works in production is seasonal decomposition (STL): separate the time series int

5. What Happens to Bad Records?

   Poison Pill Isolation and Failure-Mode-Driven Architecture At intermediate, bad records go to a DLQ. The interviewer then asks: 'How do you architect the entire pipeline around failure modes?' The trap is treating error handling as an afterthought. The senior signal: 'Before I write the first line of transformation code, I enumerate the failure modes - schema drift, poison pills, volume spikes, upstream delays - and design a mitigation for each.' A poison pill is a record that doesn't just fail