One Source, Two Consumers

Concepts covered: Medallion Architecture, Batch vs Streaming

Two teams read the retailer's orders. The finance dashboard wants hourly revenue by category, complete for yesterday by 08:00 and refreshed through the day. The recommendations team wants per-customer features in Feast: orders in the last 7 days, average basket, most recent category, refreshed every 15 minutes, and computed point-in-time so that a training set built for March 12 sees only what was known on March 12. Same source, 2 consumers, and 2 places the pipeline can fork.

Fork at ingestion

The first design gives each consumer its own reader on orders.v1. The dashboard path is the hourly batch already built. The feature path is a 15-minute Spark Structured Streaming job that reads the topic, cleans, aggregates per customer and writes to Feast's offline store, with its own test-account filter, deduplication and cast of amount. It ships fast, because neither team waits on the other, and it recreates the drift section 0 opened with.

Fork at the curated layer

The second design keeps 1 raw landing and 1 curated orders table, and forks after it: 1 serving job aggregates curated orders into the finance mart, and 1 computes features from the same rows. The cost is that the curated layer must run at the faster consumer's cadence. An hourly curated table cannot feed a 15-minute feature refresh, so the curated load moves to a 15-minute micro-batch, and the finance mart, still hourly, reads 4 curated slices per run. A 15-minute MERGE over 10,000 orders completes in under 30 seconds on an X-Small Snowflake warehouse.

The dashboard and the feature store fork after curated, which now runs every 15 minutes. Fraud scoring forks at the topic because a 200-millisecond budget allows nothing else, and a nightly job reconciles its decisions against curated orders.

orders.v1 (source, Kafka)
raw landing (bronze, S3 Parquet, 15 min)
curated orders (silver, Snowflake + dbt, 15 min); idempotency: merge on order_id
finance mart (gold, Snowflake + Looker, hourly)
customer features (consumer, Feast, 15 min)
fraud scoring (consumer, Flink, 200 ms)

orders.v1 -> raw landing
raw landing -> curated orders
curated orders -> finance mart
curated orders -> customer features
orders.v1 -> fraud scoring

Sources: orders.v1
Raw: raw landing
Curated: curated orders
Serving: finance mart, customer features, fraud scoring

Concern	Fork at ingestion	Fork at curated
Definition of an order	2 implementations that drift	1 dbt model, both read it
Fix to the test-account filter	2 pull requests, 2 deploys	1 change, both consumers next run
Freshness	each path at its own cadence	curated at the fastest consumer's cadence
Reconciliation	numbers disagree, a meeting picks one	both derive from the same rows

Point-in-time correctness decides it. A training set asks, for each order on March 12, what the customer's 7-day order count was at that moment. From curated orders with order_hour as event time, that is a window over 1 table. From a streaming path with its own deduplication, the answer depends on which late orders the stream happened to see, and the model trains on features the dashboard cannot reproduce. Forking at curated makes the model's orders match finance's by construction.

When the fork belongs at ingestion

A consumer that needs latency the curated layer cannot provide reads the source directly. Fraud scoring at checkout needs a decision within 200 milliseconds of the order event, and a 15-minute micro-batch is 4,500 times too slow. It gets a Flink job on orders.v1 with minimal cleaning, plus a nightly job reconciling its decisions against curated orders. The early fork is deliberate and bounded, and its reconciliation cost is written into the design.

A fourth consumer appears: fraud scoring that must accept or hold an order within 200 milliseconds of the event. The curated layer runs every 15 minutes. Where does this consumer attach?

To the curated orders table, polling every minute
To orders.v1 directly, through a Flink job with minimal cleaning, plus a nightly job reconciling its decisions against curated orders
To the raw S3 landing, since it is the earliest batch copy
To the finance mart, which already has revenue by hour

Do

Split a pipeline into layers at the second consumer or the second source, before cleaning gets copied
Write each layer's promise down: raw is complete and untouched, curated is the one definition
Pin every input of a merge step to the run's interval with an as-of rule
Run the curated layer at the fastest consumer's cadence and let slower consumers aggregate
Put a count and freshness check between curated and serving, and make it block

Avoid

Don't let a consumer read another consumer's serving table as if it were curated
Don't inner join a late-arriving dimension; keep the rows and make the nulls visible
Don't mask PII in a model after the raw rows have landed in the warehouse
Don't fork at ingestion for a consumer a 15-minute curated layer would satisfy

About this lesson

This section is part of the Pipeline Anatomy: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.