Many Sources, One Curated Layer
Concepts covered: Medallion Architecture
At a mid-size retailer the order pipeline begins as 1 chain. A Kafka topic orders.v1 feeds an hourly Spark job that writes Parquet to S3, and a Snowflake task loads fct_revenue_hourly for the finance dashboard. The first change request comes from marketing, which wants orders joined to ad spend, and the quickest path is a second job reading the same topic. Three months later the recommendations team builds a third. Each job drops test orders, casts amount from string to NUMERIC and deduplicates on order_id, and each does it a little differently.
The cost arrives as disagreement. For Tuesday hour 14 finance counts 40,000 orders, marketing counts 40,210 because its job never filtered the staff-discount test accounts, and the recommendations model trains on 39,600 because its deduplication keys on order_id plus updated_at and drops legitimate resubmissions. Three teams hold 3 numbers, and a meeting decides which one is real. New sources multiply the problem: once the Postgres product catalog, the Zendesk ticket API and the Google Ads spend export arrive as sources, 4 sources and 3 consumers wired point to point need 12 jobs, and the cleaning of each source is written in 3 of them.
The trigger is the second consumer
One source and 1 consumer need no middle layer; the transform can shape data for its only reader, which is the beginner pipeline. The second consumer of the same source is the moment to split, because the cleaning either gets copied or gets shared, and copied cleaning drifts within weeks. The second source feeding the same consumer is the other trigger: conforming keys, such as the catalog's sku against the order line's product_code, is work every later consumer needs too. Either event turns N x M jobs into N + M, with 1 landing job per source and 1 serving job per consumer.
What each layer promises
The medallion names, bronze, silver and gold, already describe the 3 layers. Designing them means writing down what each one promises to the layer beside it, because that promise lets finance and recommendations stop reading the Kafka topic themselves.
| Layer | Holds | Promise to the next layer | Owner |
|---|---|---|---|
| Raw (bronze) | Every record as the source sent it, Parquet on S3 partitioned by arrival hour, never updated | Nothing lost, nothing changed, replayable from any hour | Data platform |
| Curated (silver) | 1 row per order, typed, deduplicated, test accounts removed, catalog keys conformed | The company's single definition of an order | Data platform with domain owners |
| Serving (gold) | Aggregates and joins shaped for 1 consumer, derived only from curated | Fast for its reader, rebuildable from curated at any time | The consuming team |
- orders.v1 (source, Kafka)
- product catalog (source, Postgres via Debezium)
- ad spend (source, Google Ads export)
- raw landing (bronze, S3 Parquet)
- curated orders (silver, Snowflake + dbt)
- finance mart (gold, Snowflake, Looker)
- customer features (gold, Feast)
- orders.v1 -> raw landing
- product catalog -> raw landing
- ad spend -> raw landing
- raw landing -> curated orders
- curated orders -> finance mart
- curated orders -> customer features
- Sources: orders.v1, product catalog, ad spend
- Raw: raw landing
- Curated: curated orders
- Serving: finance mart, customer features
The layer is paid for in latency and storage. Each extra hop runs on its own schedule, so an hourly curated layer bounds every serving table at hourly freshness or slower. Storage roughly triples, but at around $23 per compressed TB a month, 3 copies of the retailer's 2TB order history cost about $140 a month, less than 1 hour of 3 teams reconciling revenue numbers.
The layer also moves where a bug gets fixed. When the test-account filter missed a new pattern of staff accounts in March, the fix landed in 1 dbt model and every serving table picked it up on its next run. Point to point, the same fix is 3 pull requests to 3 repositories, and the third ships a month late.
The recommendations team needs order data for a model. Today 1 Spark job reads orders.v1, filters test accounts, deduplicates and writes fct_revenue_hourly, an hourly aggregate finance reads. What should the team read?
- A new consumer on orders.v1 with its own filtering and deduplication
- fct_revenue_hourly, since the cleaning is already done
- Split the existing job: land raw to S3, build a curated orders table that holds the cleaning, and have both the finance aggregate and the model read it
- A nightly export of fct_revenue_hourly to S3 for the model to load
About this lesson
This section is part of the Pipeline Anatomy: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.