Cost of Freshness
The question is a budget question
It arrives as enthusiasm from the product side, repeated by the interviewer: 'the executives want the revenue dashboard in real time, how would you build it?' The scored answer begins with a question back: what decision does anyone make from this number, and how fast do they make it? If the dashboard is read at the 09:30 standup, 15-minute freshness is indistinguishable from 1-second freshness, and the difference in cost is where the judgement shows. The interviewer wants you to price the ladder, place the use case on the right rung and defend the placement.
The freshness ladder and what each rung costs
| Cadence | Runs per day | Compute shape | Hidden cost |
|---|---|---|---|
| Daily batch | 1 | 1 cluster, 45 min | worst-case staleness of 25 h |
| Hourly batch | 24 | 1 cluster, 5 min each | 24x the scheduling surface, late data straddles hours |
| 5-minute micro-batch | 288 | warm cluster or always-on pool | small files, a compaction job, lag alerts |
| Streaming | continuous | always on, sized for peak | 2 code paths if batch backfill stays, checkpoint operations |
The shape of the cost curve matters more than any single figure. Daily and hourly batch cost compute roughly in proportion to data processed, because the cluster runs and then stops. Below about 5 minutes the cluster stops stopping: it stays warm for the next run, so you pay for peak capacity all day. A streaming job sized for a Black Friday peak of 10,000 events per second runs at that size on a Tuesday in February at 400 per second. The number to carry is how much of the day the capacity is busy: a batch job that runs 45 minutes a day uses its capacity about 3 percent of the time, and the same work as a stream uses 100 percent of an always-on allocation, so the streaming version can cost 10x to 30x more for the same bytes when the capacity is fixed.
- 1,440
- 12x
The cost of small files
Micro-batches carry a cost that shows up 3 months later: each run writes a file per partition, so 288 runs across 50 partitions is 14,400 files a day, and a month in, a query over that table opens 432,000 files before reading a byte. Then a compaction job is added to merge them, which is another pipeline to monitor, and the freshness promise now has an asterisk because compaction and the next micro-batch race for the same partition. Interviewers reward candidates who raise this unprompted, because it shows they have run the 5-minute pipeline rather than drawn it.
Placing the use case
| Use case | Who acts, how fast | Right cadence | Why |
|---|---|---|---|
| Fraud scoring at checkout | a model, inside the request | streaming | the decision happens inside the latency window |
| Live sales globe on a marketing page | nobody acts, it is a display | streaming, sampling allowed | the value is the motion, so dropped events are acceptable |
| Executive revenue dashboard | humans, at a daily standup | hourly or daily | the decision cadence is daily |
| Inventory reorder | a planner, each morning | daily, plus a hot path for stockouts | split the 1 urgent signal from the bulk |
| Experiment metrics | a PM reading results after a week | daily | nothing changes faster than the experiment |
The row to study is inventory, because it shows the pattern interviewers reward most: split the workload. The bulk recomputation runs daily and cheaply, and a narrow streaming path carries only the stockout signal, which is small and high value, and where latency matters. You get real time where it pays and batch everywhere else, and the streaming cluster is sized for a trickle rather than the whole feed.
Saying it in the room
Lead with the question back, then state the rung, then state the number. 'Who reads this and what do they do within 5 minutes of reading it? If nobody acts inside an hour, hourly batch at roughly a twelfth of an always-on cluster the same size; if the fraud model consumes it, streaming, sized for peak with the batch path kept for backfill.' Then name what the design costs: 2 code paths are a maintenance cost you are choosing, and a late-data policy is required at every cadence, because an hourly job still receives events stamped in the previous hour. Close by naming the freshness SLA as a contract with a number and an owner, since 'real time' without a number is a request for an unbounded budget.
About this lesson
This section is part of the How Data Moves: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.