1-Hour Allowed Lateness

Concepts covered: Late-Arriving Data

Allowed lateness is the second setting on a window. After the watermark fires a window, the engine keeps that window's state for a further stretch of event time, 60 minutes in this job. A late event that lands inside the stretch updates the total, and the window fires again with the corrected figure. An event that turns up after the stretch finds no state left to update and goes to late_events, a dead letter table where a person can find it.

ClassArrives while the watermark isExampleResult for [09:00, 09:05)
On-timebefore 09:05E1, E2In the first firing: 300
Slightly latefrom 09:05 up to 10:05E3, at 09:34Total becomes 350, window fires again
Very lateat or past 10:05E4, at 11:15State already discarded; E4 goes to late_events

The animation replays all 6 events through the revenue_5m job in arrival order. Run the 60-minute variant, then the variant with no allowed lateness, and watch where E3 ends up each time.

One window, three kinds of late
  • mobile_events (queue, Kafka)
  • revenue_5m (transform, Beam)
  • revenue_by_window (storage, BigQuery)
  • late_events (queue, dead letter)
  • mobile_events -> revenue_5m
  • revenue_5m -> revenue_by_window
  • revenue_5m -> late_events
  • Allowed lateness 60 min: Result: 09:00–09:05 = 350 and 09:05–09:10 = 40; E4 is in late_events
  • No allowed lateness: Result: 09:00–09:05 = 300 and 09:05–09:10 = 40; E3 and E4 are in late_events

Lateness runs on the watermark

E6 decided E4's fate. Stamped 10:06:30, E6 moved the watermark to 10:05:30, past the window's end plus 60 minutes, so the engine discarded the window's state and E4 found nothing to join at 11:15. Had E6 never arrived, the watermark would still read 09:05:01 and E4 would have been accepted, however late the wall clock said it was. With allowed lateness at 0, the state goes the moment the window fires, so E3 and E4 both land in late_events and the total stays at 300.

The window logic in plain Python

The function below holds the whole policy: a 5-minute window, a 60-second lag and 60 minutes of allowed lateness. For each event it checks whether the window's lateness has run out, updates the total, re-fires a window that already fired, then advances the watermark and fires or discards windows.

from datetime import datetime, timedelta

WINDOW = timedelta(minutes=5)
WATERMARK_LAG = timedelta(seconds=60)
ALLOWED_LATENESS = timedelta(minutes=60)


def at(hms):
    return datetime.strptime(hms, "%H:%M:%S")


def window_start(event_time):
    minute = event_time.minute - event_time.minute % 5
    return event_time.replace(minute=minute, second=0)


def run(events):
    totals, fires, output, late = {}, {}, {}, []
    watermark = datetime.min
    for event_id, event_time, amount in events:
        start = window_start(event_time)
        if watermark >= start + WINDOW + ALLOWED_LATENESS:
            late.append(event_id)
            continue
        totals[start] = totals.get(start, 0) + amount
        if start in fires:
            fires[start] += 1
            output[start] = totals[start]
        watermark = max(watermark, event_time - WATERMARK_LAG)
        for open_start in list(totals):
            end = open_start + WINDOW
            if watermark >= end and open_start not in fires:
                fires[open_start] = 1
                output[open_start] = totals[open_start]
            if watermark >= end + ALLOWED_LATENESS:
                del totals[open_start]
    return output, fires, late


events = [
    ("E1", at("09:02:14"), 100),
    ("E2", at("09:03:55"), 200),
    ("E5", at("09:06:01"), 40),
    ("E3", at("09:02:14"), 50),
    ("E6", at("10:06:30"), 60),
    ("E4", at("09:02:14"), 75),
]
output, fires, late = run(events)
windowtotalfires
[09:00, 09:05)3502
[09:05, 09:10)401

The function returns late as ['E4']. Window [10:05, 10:10) holds E6's 60 and has not fired, because the watermark stands at 10:05:30.

Every re-fire replaces the last one

Each firing emits the window's full current total, and a later firing supersedes an earlier one. A sink that appends keeps both rows for 09:00, and a dashboard that sums the table reports 650. The sink has to upsert on the window key: the replace-not-append rule from idempotent reruns, applied to streaming output.

A team's sink appended every firing to revenue_firings, one row per firing with a fired_at timestamp. Late refunds can lower a total on re-fire. Which aggregation returns the current total per window?

What an hour of lateness costs

Every window's state now outlives its firing by an hour of watermark progress, so each key carries the open window plus every window that fired in the previous hour. A revenue job keyed by city across 600 cities multiplies that by 600. Readers also see a number that may still change: the 09:00 total is final only once the watermark moves an hour past 09:05.

A job uses 5-minute windows, a 60-second lag and 60 minutes of allowed lateness. The watermark reads 14:40:00. Three events then arrive, stamped 14:03:00, then 15:06:00, then 14:04:00. What happens to window [14:00, 14:05)?

About this lesson

This section is part of the Schema Drift: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.