Per-Node Freshness Tier Analysis

Concepts covered: Batch vs Streaming

Four tiers, one rule

A freshness tier is the delay a node promises between an event happening and that event showing in its output. Four tiers cover most pipelines: seconds, minutes, hours and daily. Writing the tier beside every node exposes a rule an unannotated diagram hides: a node is only as fresh as its slowest input, however often it runs.

tier	typical cadence	usually built as
seconds	under 1 minute	stream job over a log
minutes	1 to 15 minutes	micro-batch
hours	hourly to every 6 hours	scheduled job
daily	once a day or slower	nightly job or snapshot export

Annotating a real graph

An online store runs the pipeline below. Order events arrive on a log. A product catalog is exported from the operations database every night at 02:00. A micro-batch every 5 minutes joins recent orders to the catalog to attach each order's category and margin, writing enriched_orders. A second micro-batch every 5 minutes sums enriched_orders into revenue_by_category. Two consumers read the results: a finance dashboard that refreshes hourly, and a fraud model that polls enriched_orders every 30 seconds to score each new order against its category's typical basket.

Each node carries its own cadence; the analysis asks whether that cadence is one its inputs can support and its consumers need.

order_events (source, event log)
product_catalog (storage, nightly export)
enrich_orders (transform, micro-batch)
enriched_orders (storage, orders with category, margin)
aggregate_revenue (transform, micro-batch)
revenue_by_category (storage, sums per category)
finance_dashboard (consumer, refreshes hourly)
fraud_model (consumer, polls every 30 s)

order_events -> enrich_orders
product_catalog -> enrich_orders
enrich_orders -> enriched_orders
enriched_orders -> aggregate_revenue
aggregate_revenue -> revenue_by_category
revenue_by_category -> finance_dashboard
enriched_orders -> fraud_model

node	own cadence	slowest input	effective freshness	consumer expects	finding
order_events	seconds	none	seconds		fine
product_catalog	daily	operations database, daily export	daily		fine
enriched_orders	5 min	product_catalog, daily	order columns 5 min; category and margin up to 24 h	fraud_model, 30 s	unmet: polled 10 times per run, margin a day stale
revenue_by_category	5 min	enriched_orders, 5 min	amounts 5 min; category daily	finance_dashboard, hourly	hidden cost: 12 runs per refresh

The 2 mismatches point in opposite directions. The fraud model reads a 5-minute table every 30 seconds and believes it scores fresh orders; it scores orders up to 5 minutes old, and when the catalog reprices a product at 10:00 the margin it sees stays wrong until the next 02:00 export. That is an unmet expectation, and no alarm fired because the table did update every 5 minutes. The revenue table is rebuilt every 5 minutes for a dashboard that reads it hourly: hidden cost, as the aggregate job starts, reads, writes and releases workers around the clock for a consumer that looks once an hour.

The 5-step analysis

Write each node's own cadence beside it, including sources and snapshot exports
Propagate downward: a node's effective freshness is the slowest of its inputs, per column where inputs differ
Write each consumer's expected freshness beside it, taken from how the data is used, never from how often it polls
Flag as hidden cost runs outpacing all their inputs or the fastest consumer's need, like 288 revenue runs for 24 reads
Flag every consumer whose expectation is faster than its source's effective freshness as unmet

Fixes move a tier boundary rather than nudge a cadence. If the fraud model needs only the order columns at 30 seconds, it should read order_events through a stream job and skip the enrichment. If it needs margin at 30 seconds, the catalog must publish each price change as an event the enrichment consumes, moving product_catalog to the seconds tier: the expensive fix. For finance, revenue_by_category moves to hourly or becomes a query the dashboard runs on refresh, while the 5-minute enrichment stays if the fraud path still wants it.

A pipeline streams clickstream events (seconds) into a sessions table rebuilt by a micro-batch every 10 minutes, which joins a user_segments table refreshed weekly. A marketing tool reads sessions every 15 minutes to pick a banner by segment. Which finding is correct?

The micro-batch should run every 5 minutes so the marketing tool reads fresher sessions on each 15-minute poll and the banner reflects the latest clicks.
The marketing tool's 15-minute read is served, but segment is weekly-fresh, so a user who changed segment on Monday gets the old banner all week; the fix is on user_segments.
The sessions micro-batch is hidden cost, because clickstream arrives in seconds and the batch only runs every 10 minutes, wasting the source's freshness.
The weekly user_segments refresh is hidden cost, because the micro-batch reads it 1,008 times a week for a table that only changes once.

About this lesson

This section is part of the Batch vs Streaming: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.