Watermarks: The Engine's Promise

Concepts covered: Late-Arriving Data

A daily batch job knows yesterday is finished when the last file lands. A streaming job that sums revenue in 5-minute event-time windows gets no such signal, and T6 showed that a trip can surface 67 minutes after it ended. Before the job publishes a total for 09:00 to 09:05, it needs a rule for deciding it has waited long enough. A watermark is that rule.

A watermark is a timestamp in event time that the engine carries alongside the stream. When it reads W, the engine is asserting: no further events with event time earlier than W will arrive. A window covering [09:00, 09:05) counts as complete once the watermark reaches 09:05, and at that moment the engine emits, or fires, the window's total.

Max event time seen, minus a lag

Most engines compute a heuristic watermark from the data itself: the largest event time seen so far, minus a fixed lag that absorbs ordinary out-of-order delivery. With a 60-second lag, an event stamped 09:06:01 moves the watermark to 09:05:01. It never moves backward, and it moves forward only when an event newer than every earlier one arrives.

arriveseventevent_timeamountwatermark afterwhat happens
09:02:18E109:02:1410009:01:14joins [09:00, 09:05), total 100
09:04:30E209:03:5520009:02:55joins [09:00, 09:05), total 300
09:06:02E509:06:014009:05:01opens [09:05, 09:10); [09:00, 09:05) fires with 300
09:34:00E309:02:145009:05:01event time sits behind the watermark: late

Read the last row closely. E3 ended at 09:02:14, before the watermark of 09:05:01, so the engine's assertion turned out false. A heuristic watermark is a bet about the network, and a phone in a garage loses that bet. The watermark held at 09:05:01 as E3 arrived, since E3 is older than events the engine had already seen. What the engine does with E3 is a separate setting, the subject of the next section.

Because this watermark is built from event times, a quiet stream holds it still. Window [09:05, 09:10) fires only when an event stamped at or after 09:11:00 arrives, however much wall-clock time passes first.

A job sums 5-minute windows with a 60-second watermark lag. Events arrive in this order, stamped 10:03:10, 10:05:30, 10:06:20, then 10:04:50. What happens to window [10:00, 10:05)?

Choosing the lag

10-second lag
  • Totals publish about 10 seconds after a window ends
  • Any event delayed past 10 seconds counts as late
  • Open windows sit in memory briefly
  • Suits alerting, where a fast rough answer beats a slow exact one
30-minute lag
  • Totals publish about 30 minutes after a window ends
  • Only events delayed past 30 minutes count as late
  • Every key holds 30 minutes of open windows in memory
  • Suits billing, where completeness outranks speed

The lag trades latency for completeness, so set it from measured skew. If the p99 of processed_at minus ended_at is 40 seconds, a 60-second lag leaves fewer than 1 event in 100 late in a steady stream, and every total waits about a minute past its window's end.

A team raises the watermark lag on its revenue job from 60 seconds to 10 minutes. What should it expect?

About this lesson

This section is part of the Schema Drift: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.