The Binding and the Claim

A medium Pipeline Design interview practice problem on DataDriven. Design the pipeline on the canvas: sources, transforms, storage and checks, then get feedback on it.

Domain
Pipeline Design
Difficulty
medium
Seniority
L5

Problem

Policy transactions (bindings, endorsements, cancellations) need immediate confirmation, claim events are processed in the background, and both now share database tables that no analytical query can read without hitting production. What event-driven pipeline handles both classes correctly?

Policies are instant. Claims take their time.

Business requirements

Worked solution and explanation

What this problem really is

This is a split-by-latency problem in an insurance costume, and the split is the easy part. Everyone draws a fast lane for bindings and a slow lane for claims. What separates candidates is what each lane carries. The policy lane must carry the moment of binding as the record's timestamp. A malformed claim must land somewhere it can be replayed from. Get the first wrong and every regulator audit reports your pipeline's clock. Get the second wrong and a customer's claim stops existing.

The quieter trap is reading 'immediate confirmation' as 'policies only go to the dashboard.' The statement also says analysts cannot query either class without hitting production. If policy events feed only the dashboard store, policy analytics is still unsolved, and the bind time never reaches the warehouse where the regulator asks for it.

Step 1: Put a log between the app and everything else

Policy transactions go onto a policy event stream and claims onto a claim event queue, both on Kafka. The production database stops being the integration point, and every downstream reader consumes from the log instead of querying the tables agents write to.

Step 2: Size each lane to its reader

A Flink processor consumes policy events in real time and upserts the dashboard store, so a binding or cancellation shows up within a minute. Claims run through an hourly Spark job. The claims lane only has to be slower than the policy lane, and an hourly cadence is plenty for analytics.

Step 3: Land both classes in staging, then the warehouse

The policy processor writes to the dashboard store and to the staging area. Claims land in the same staging area. The analytics warehouse loads from staging, and analysts read only the warehouse. Production never sees an analytical query again.

Step 4: Stamp event_time at the source and never overwrite it

The application sets event_time when the policy binds. Every hop passes it through untouched, and staging and the warehouse partition on it. Keep processed_at as a separate lineage column if you like, but it never becomes the record's time.

Step 5: Park bad claims, replay them on their own

A claim that fails parsing goes to the dead letter queue with its rejection reason, and the rest of the batch keeps moving. After the parser is fixed, a replay job reads only the parked events and writes them to staging. Events that already loaded are untouched.

The reference design

Policies stream in real time to the dashboard store and also land in staging. Claims load in a slower batch. Both reach the warehouse carrying the application's `event_time`. Malformed claims wait in the dead letter queue and are replayed on their own.
  • policy service (source, API)
  • claims service (source, API)
  • policy event stream (queue, Kafka)
  • claim event queue (queue, Kafka)
  • policy stream processor (transform, Flink); freshness: real-time
  • claim batch job (transform, Spark); freshness: < 1h; on error: dlq
  • dead letter queue (object storage) (storage)
  • claim replay job (transform, Spark)
  • staging area (event time) (storage)
  • agent dashboard store (storage, PostgreSQL); freshness: < 1min
  • analytics warehouse (storage)
  • agent dashboard (consumer, Grafana)
  • analytics team (consumer, Metabase)
  • policy service -> policy event stream
  • claims service -> claim event queue
  • policy event stream -> policy stream processor
  • claim event queue -> claim batch job
  • policy stream processor -> agent dashboard store
  • policy stream processor -> staging area (event time)
  • claim batch job -> staging area (event time)
  • claim batch job -> dead letter queue (object storage)
  • dead letter queue (object storage) -> claim replay job
  • claim replay job -> staging area (event time)
  • staging area (event time) -> analytics warehouse
  • agent dashboard store -> agent dashboard
  • analytics warehouse -> analytics team
Processing time
Rows carry processed_at. A consumer lag or a replay shifts every binding by minutes or days, and the warehouse reports when the pipeline woke up.
Event time
Rows carry the application's event_time, and partitions key on it. A replayed claim lands in the day it actually happened, and the regulator sees the bind moment.

Common follow-up questions