Allowed Lateness Versus Accuracy
Concepts covered: Late-Arriving Data
Allowed lateness L keeps a window's state alive after the watermark passes its end. An event that arrives within L still updates the window, and the engine emits a corrected result; one that arrives later is dropped or sent to a side output. Together, D and L decide 3 things at once: how complete the streaming result is, how much state the engine holds, and how often a consumer sees a number change.
Coverage comes from the lateness distribution
An event updates its window while the watermark, newest event time minus D, is still below the window's end plus L. Lateness is how far an event's time trails the newest one seen on arrival. Lateness up to D + L always counts, and an event stamped early in its 1-minute window gets almost a minute more. For the trades feed with D = 30 seconds, a month of measured lateness gives:
| Allowed lateness L | Counted if lateness up to | Events covered |
|---|---|---|
| 0 | 30 s | 96.0% |
| 5 min | 5 min 30 s | 99.2% |
| 1 hour | 1 h 0 min 30 s | 99.93% |
| 24 hours | 24 h 0 min 30 s | 99.995% |
State grows with the budget
With tumbling windows of size S, a key holds at most (D + L) / S open windows, rounded up, plus the one being filled. The job keys 1-minute windows by 500,000 active accounts at 400 bytes each. A window exists only once its account trades, so the table is a ceiling: every account trading every minute.
| L | Open windows per key | Engine state | Events covered |
|---|---|---|---|
| 0 | 2 | 400 MB | 96.0% |
| 5 min | 7 | 1.4 GB | 99.2% |
| 1 hour | 62 | 12.4 GB | 99.93% |
| 24 hours | 1,442 | 288.4 GB | 99.995% |
Read the table as marginal cost: 0 to 5 minutes buys 3.2 points for at most 1 GB more state, while 1 hour to 24 hours buys 0.065 points for up to 276 GB, reloaded on every restart. That flattening makes a large L poor value. No L reaches 100%, because some events arrive later than any finite budget.
Count coverage in the unit the consumer uses
Coverage by event count can flatter a stream. On the trades feed, the 0.07% of events that land after a 1-hour budget are mostly corrections and cancellations sent from back-office systems, and each one moves far more money than a typical fill. Weighted by notional value, the same budget covers less.
- 98.6%
- 1.4%
Choosing a percentile target
- Start from the consumer: what coverage, in its own unit, makes the live number usable?
- Find the smallest L that reaches that percentile on a month of measured lateness.
- Check the state at that L against the job's memory and restart-time budget.
- Send events past L to a side output so the tail stays measured.
- Hand everything beyond L to a reconciliation pass rather than raising L further.
For the trades job that lands on L = 1 hour: 99.93% of events, 62 windows per key, 12.4 GB of state, and a tail small enough for a nightly pass to correct. Each late event inside the hour re-emits its window, so consumers upsert by account and window start, the idempotent write from earlier lessons.
A dashboard needs 99% of events inside its live number, and the job has a 2 GB state budget. Using the tables above, which L fits?
- 0, since 96.0% is close enough and state is smallest
- 5 minutes: 99.2% of events for at most 1.4 GB of state
- 1 hour, since a larger margin over 99% is always safer
- 24 hours, so the dashboard never needs reconciliation
- Raise L
- Live number closer to final
- State grows with every key and window
- Restarts reload more state
- More re-emissions downstream
- Keep L, reconcile the tail
- Live number labeled provisional
- State sized to the knee of the curve
- Tail corrected in batch from raw events
- Exactness arrives on the pass's schedule
Each re-emission replaces the window's earlier result, so a consumer that appends results counts that window twice.
A team raises L from 1 hour to 24 hours so finance stops reporting missing late trades. Using the tables above, what changes?
- Coverage reaches 100%, so the nightly reconciliation can be retired
- State stays near 12.4 GB, since only late events add windows
- Coverage rises 3.2 points, matching the step from 0 to 5 minutes
- State can grow to 288.4 GB for 0.065 more points of event coverage
About this lesson
This section is part of the Schema Drift: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.