Before the Batch Is Lost
A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.
- Domain
- Pipeline Design
- Difficulty
- hard
- Seniority
- senior
- Asked in
- Anheuser-Busch
Problem
What pipeline would you design for an operations lead running a few hundred breweries whose bottling and canning lines stream fill level, temperature and capper torque at about 4 billion readings a day? Plants on flaky links go dark for hours, then dump their backlog in a burst when they reconnect.
Business requirements
- As a plant operator I need to know a line is drifting out of spec within seconds so I can stop it before we scrap a whole batch.
- As supply chain I need an exact daily count of good units produced per SKU that reconciles cleanly for finance.
- As supply chain I need the daily numbers to include readings from plants that were offline and reported late.
- As the data engineering team I need a stuck or malformed sensor to not corrupt the numbers or halt the line-monitoring pipeline.
Worked solution and explanation
What this really is
This is a lambda architecture dressed up as a bottling line. One sensor stream feeds two consumers with opposite correctness budgets. Operators want drift flagged in seconds and can live with an approximate count. Finance wants an exact per-SKU daily count and can wait until tomorrow. Anyone can draw Kafka and Flink. What separates candidates is refusing to let the fast aggregate become the financial number. That aggregate buckets by arrival time, so a plant that reconnects after 4 hours either lands its production in the wrong day or double-counts it on replay. Then the count never reconciles.
The second trap sits upstream. A stuck sensor repeating a constant torque, or one spraying out-of-range fill levels, trips false pages or wedges the monitor, and it quietly inflates the count. Quarantine it before it reaches either path, and never let the check block the line monitors.
Walk the requirements
Step 1: Quarantine bad readings before either path
One validation step sits right after the queue, ahead of both paths. Range and stuck-value checks send garbage down an error path to a dead-letter store, and good readings keep flowing. A rising dead-letter rate is itself the signal that a sensor is failing.
Step 2: Put ops on a sub-minute streaming path
Flink reads the validated readings, scores fill level, temperature and capper torque per line against spec, and pages within seconds. Approximate and at-least-once are fine here. A scheduled job is not, however elegant the rest of the design is.
Step 3: Reconcile finance on event time
The daily count keys on the reading's PLC timestamp, with a watermark and a grace window sized to the longest offline gap. Key on arrival time instead, and the plants with the worst links report less than they made, with no error raised anywhere.
Step 4: Make exactness a property of the daily job
Deduplicate on a stable unit id (line id plus PLC sequence number) inside the daily job, so a gateway replaying its whole buffer cannot inflate yield. The transport stays at-least-once, because only finance ever needed exactness.
The reference design
- line_sensors (source, API)
- ingest_queue (queue, Kafka)
- reading_validation (quality gate, Great Expectations)
- dead_letter_store (storage, object storage)
- line_drift_monitor (transform, Flink); freshness: < 1min
- ops_alerts (consumer, alerting)
- event_time_dedup_reconciliation (transform, Spark); freshness: < 24h
- production_warehouse (storage, warehouse)
- supply_chain_dashboard (consumer, dashboard)
- line_sensors -> ingest_queue
- ingest_queue -> reading_validation
- reading_validation -> dead_letter_store
- reading_validation -> line_drift_monitor
- line_drift_monitor -> ops_alerts
- reading_validation -> event_time_dedup_reconciliation
- event_time_dedup_reconciliation -> production_warehouse
- production_warehouse -> supply_chain_dashboard
Common follow-up questions
- A plant replays 4 hours of backlog. What stops double-counting, and what puts each reading in the right day? (Whether the dedup key and the watermark are wired in, or just named.)
- Finance closed last month on the 3rd, and a plant reports late on the 4th. How do you restate a closed day? (Versioned daily partitions and idempotent overwrite, not a window left open forever.)
- Volume grows 5x and the anomaly job misses its sub-minute budget. Where do you look first? (Queue partitioning, skewed hot lines, and isolating the ops path from batch spikes.)