The Firehose and the Ledger

A hard Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant grading.

Domain

Pipeline Design

Difficulty

hard

Seniority

senior

Problem

We ingest payment events from a few hundred partner APIs, some pushing webhooks and some we poll on a schedule, totaling around 2 billion events a day. Operations needs a near-real-time view of transaction volume and error rates within seconds, while finance needs an exactly-once daily ledger that reconciles to the cent. Partners go down and resend the same events when they recover, so the platform has to retry, deduplicate, and never lose or double-count a transaction.

Practice This Problem

Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and grades it automatically.