Before the Batch Is Lost

A hard Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain
Pipeline Design
Difficulty
hard
Seniority
senior
Asked in
Anheuser-Busch

Problem

What pipeline would you design for an operations lead running a few hundred breweries whose bottling and canning lines stream fill level, temperature and capper torque at about 4 billion readings a day? Plants on flaky links go dark for hours, then dump their backlog in a burst when they reconnect.

Business requirements

Worked solution and explanation

What this really is

This is a lambda architecture dressed up as a bottling line. One sensor stream feeds two consumers with opposite correctness budgets. Operators want drift flagged in seconds and can live with an approximate count. Finance wants an exact per-SKU daily count and can wait until tomorrow. Anyone can draw Kafka and Flink. What separates candidates is refusing to let the fast aggregate become the financial number. That aggregate buckets by arrival time, so a plant that reconnects after 4 hours either lands its production in the wrong day or double-counts it on replay. Then the count never reconciles.

The second trap sits upstream. A stuck sensor repeating a constant torque, or one spraying out-of-range fill levels, trips false pages or wedges the monitor, and it quietly inflates the count. Quarantine it before it reaches either path, and never let the check block the line monitors.

Walk the requirements

Step 1: Quarantine bad readings before either path

One validation step sits right after the queue, ahead of both paths. Range and stuck-value checks send garbage down an error path to a dead-letter store, and good readings keep flowing. A rising dead-letter rate is itself the signal that a sensor is failing.

Step 2: Put ops on a sub-minute streaming path

Flink reads the validated readings, scores fill level, temperature and capper torque per line against spec, and pages within seconds. Approximate and at-least-once are fine here. A scheduled job is not, however elegant the rest of the design is.

Step 3: Reconcile finance on event time

The daily count keys on the reading's PLC timestamp, with a watermark and a grace window sized to the longest offline gap. Key on arrival time instead, and the plants with the worst links report less than they made, with no error raised anywhere.

Step 4: Make exactness a property of the daily job

Deduplicate on a stable unit id (line id plus PLC sequence number) inside the daily job, so a gateway replaying its whole buffer cannot inflate yield. The transport stays at-least-once, because only finance ever needed exactness.

The reference design

One buffer, one validation step, two paths. Flink at '< 1min' drives the ops alerts. A '< 24h' Spark job buckets readings by event time, deduplicates them, and lands exact per-SKU counts in the warehouse the supply-chain dashboard reads.
  • line_sensors (source, API)
  • ingest_queue (queue, Kafka)
  • reading_validation (quality gate, Great Expectations)
  • dead_letter_store (storage, object storage)
  • line_drift_monitor (transform, Flink); freshness: < 1min
  • ops_alerts (consumer, alerting)
  • event_time_dedup_reconciliation (transform, Spark); freshness: < 24h
  • production_warehouse (storage, warehouse)
  • supply_chain_dashboard (consumer, dashboard)
  • line_sensors -> ingest_queue
  • ingest_queue -> reading_validation
  • reading_validation -> dead_letter_store
  • reading_validation -> line_drift_monitor
  • line_drift_monitor -> ops_alerts
  • reading_validation -> event_time_dedup_reconciliation
  • event_time_dedup_reconciliation -> production_warehouse
  • production_warehouse -> supply_chain_dashboard

Common follow-up questions