Allowed Lateness Versus Accuracy

Concepts covered: Late-Arriving Data

Allowed lateness L keeps a window's state alive after the watermark passes its end. An event that arrives within L still updates the window, and the engine emits a corrected result; one that arrives later is dropped or sent to a side output. Together, D and L decide 3 things at once: how complete the streaming result is, how much state the engine holds, and how often a consumer sees a number change.

Coverage comes from the lateness distribution

An event updates its window while the watermark, newest event time minus D, is still below the window's end plus L. Lateness is how far an event's time trails the newest one seen on arrival. Lateness up to D + L always counts, and an event stamped early in its 1-minute window gets almost a minute more. For the trades feed with D = 30 seconds, a month of measured lateness gives:

Allowed lateness LCounted if lateness up toEvents covered
030 s96.0%
5 min5 min 30 s99.2%
1 hour1 h 0 min 30 s99.93%
24 hours24 h 0 min 30 s99.995%

State grows with the budget

With tumbling windows of size S, a key holds at most (D + L) / S open windows, rounded up, plus the one being filled. The job keys 1-minute windows by 500,000 active accounts at 400 bytes each. A window exists only once its account trades, so the table is a ceiling: every account trading every minute.

LOpen windows per keyEngine stateEvents covered
02400 MB96.0%
5 min71.4 GB99.2%
1 hour6212.4 GB99.93%
24 hours1,442288.4 GB99.995%

Read the table as marginal cost: 0 to 5 minutes buys 3.2 points for at most 1 GB more state, while 1 hour to 24 hours buys 0.065 points for up to 276 GB, reloaded on every restart. That flattening makes a large L poor value. No L reaches 100%, because some events arrive later than any finite budget.

Count coverage in the unit the consumer uses

Coverage by event count can flatter a stream. On the trades feed, the 0.07% of events that land after a 1-hour budget are mostly corrections and cancellations sent from back-office systems, and each one moves far more money than a typical fill. Weighted by notional value, the same budget covers less.

Choosing a percentile target

For the trades job that lands on L = 1 hour: 99.93% of events, 62 windows per key, 12.4 GB of state, and a tail small enough for a nightly pass to correct. Each late event inside the hour re-emits its window, so consumers upsert by account and window start, the idempotent write from earlier lessons.

A dashboard needs 99% of events inside its live number, and the job has a 2 GB state budget. Using the tables above, which L fits?

Raise L
  • Live number closer to final
  • State grows with every key and window
  • Restarts reload more state
  • More re-emissions downstream
Keep L, reconcile the tail
  • Live number labeled provisional
  • State sized to the knee of the curve
  • Tail corrected in batch from raw events
  • Exactness arrives on the pass's schedule

Each re-emission replaces the window's earlier result, so a consumer that appends results counts that window twice.

A team raises L from 1 hour to 24 hours so finance stops reporting missing late trades. Using the tables above, what changes?

About this lesson

This section is part of the Schema Drift: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.