End of Day Is Too Late

A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain: Pipeline Design
Difficulty: medium
Seniority: L6

Problem

What pipeline would put card transactions in front of a fraud analyst as they happen, at roughly 5,000 authorizations a second and triple that on paydays? Today she sees them only after the end-of-day batch.

Every swipe tells a story.

Business requirements

The fraud team needs to act on suspicious card activity while it's happening, not after a customer has been hit.
A lost transaction means a missed fraud signal; a duplicated one inflates volume reports finance signs off on.
We can't fail a PCI audit on what's stored in our systems.
Bad events do come in, and one of them can't stop the thousands of valid transactions behind it.

Worked solution and explanation

What this really is

This is a latency problem with three correctness constraints hidden in the statement. Everyone draws Kafka and a stream processor. What separates candidates is whether every hop on the path to fraud is continuous, and whether that same path is also exactly-once, PCI-clean and immune to a poison pill. Leave one scheduled stage on the path and fraud waits for the next run. Miss one of the other three and you ship duplicate volume, raw card numbers in storage, or a stream that halts on one malformed event.

Walk the requirements

Step 1: Keep the whole fraud path continuous

Authorizations arrive from an API, a streaming job tokenizes them, they land on a Kafka topic, pass validation, and a Flink job updates the fraud store within seconds. A batch stage anywhere upstream of the dashboard brings end of day back in smaller chunks.

Step 2: Tokenize before anything persists

Swap the card number for a token before the first durable write, and that includes the Kafka topic. A topic is storage with a retention window. Masking inside the stream processor is too late: the raw number already sits in the broker, and the auditor will find it there.

Step 3: Make exactly-once an end-to-end property

Three hops can lose or double a transaction, and each needs its own guard. The tokenizer retries its write to Kafka and acknowledges the gateway only after the write lands, so nothing is dropped at the edge. The fraud job checkpoints its Kafka offsets and runs exactly-once, so a restart resumes instead of skipping or replaying into a count. The gateway's timeout resends still arrive as new copies, and both sinks absorb them by upserting on transaction_id.

Step 4: Route poison pills aside and keep draining

A validation gate sits ahead of the fraud processor. Records that fail parsing or schema checks leave through the error path to a dead letter queue, and valid traffic behind them never waits. The alerts consumer tells someone when that queue fills.

The reference design

Every hop to fraud runs on Flink. The card number is tokenized before the first durable write, the tokenizer retries until Kafka holds the event, the fraud job is exactly-once, both sinks upsert on `transaction_id`, and invalid events leave through the validation gate to a watched dead letter queue.

card authorizations (source, API)
tokenizer (transform, Flink); on error: retry
tokenized events (queue, Kafka)
transaction validation (quality gate)
exactly-once fraud stream (transform, Flink)
dead letter queue (queue)
fraud store (storage, PostgreSQL); idempotency: upsert
fraud dashboard (consumer, Grafana); freshness: real-time
event lake (storage, Delta Lake); idempotency: upsert
finance reporting (consumer)
pipeline alerts (consumer)

card authorizations -> tokenizer
tokenizer -> tokenized events
tokenized events -> transaction validation
transaction validation -> exactly-once fraud stream
transaction validation -> dead letter queue
exactly-once fraud stream -> fraud store
exactly-once fraud stream -> event lake
fraud store -> fraud dashboard
event lake -> finance reporting
dead letter queue -> pipeline alerts

Ingest and tokenize: card authorizations, tokenizer, tokenized events
Stream processing: transaction validation, exactly-once fraud stream, dead letter queue
Sinks: fraud store, fraud dashboard, event lake, finance reporting
Monitoring: pipeline alerts

The first cut: Raw events land in Kafka, the card number is masked inside the processor, the sinks append blindly, and a parse error crashes the consumer. Every restart replays offsets into sinks that count them twice.
What survives review: Tokens only past the edge. A retried write into Kafka, checkpointed offsets and sinks keyed on transaction_id make a crash or a resend harmless. Parse errors go to the dead letter queue and the consumer never stops.

Common follow-up questions

An issuer resends a transaction under the same `transaction_id` with a different amount. What protects you? (Keyed upserts absorb exact duplicates silently. A mutated duplicate needs a version or a conflict rule, not just a key.)
One merchant floods a single Kafka partition during a sale. What happens to fraud latency? (Tests skew awareness: partition on the card token instead of the merchant, or salt the key.)
The tokenization service goes down. What does the pipeline do? (Tokenization is a hard dependency before the first write. Tests whether you accept ingestion backpressure over letting raw numbers through.)