Watermarks: The Engine's Promise
Concepts covered: Late-Arriving Data
A daily batch job knows yesterday is finished when the last file lands. A streaming job that sums revenue in 5-minute event-time windows gets no such signal, and T6 showed that a trip can surface 67 minutes after it ended. Before the job publishes a total for 09:00 to 09:05, it needs a rule for deciding it has waited long enough. A watermark is that rule.
A watermark is a timestamp in event time that the engine carries alongside the stream. When it reads W, the engine is asserting: no further events with event time earlier than W will arrive. A window covering [09:00, 09:05) counts as complete once the watermark reaches 09:05, and at that moment the engine emits, or fires, the window's total.
Max event time seen, minus a lag
Most engines compute a heuristic watermark from the data itself: the largest event time seen so far, minus a fixed lag that absorbs ordinary out-of-order delivery. With a 60-second lag, an event stamped 09:06:01 moves the watermark to 09:05:01. It never moves backward, and it moves forward only when an event newer than every earlier one arrives.
| arrives | event | event_time | amount | watermark after | what happens |
|---|---|---|---|---|---|
| 09:02:18 | E1 | 09:02:14 | 100 | 09:01:14 | joins [09:00, 09:05), total 100 |
| 09:04:30 | E2 | 09:03:55 | 200 | 09:02:55 | joins [09:00, 09:05), total 300 |
| 09:06:02 | E5 | 09:06:01 | 40 | 09:05:01 | opens [09:05, 09:10); [09:00, 09:05) fires with 300 |
| 09:34:00 | E3 | 09:02:14 | 50 | 09:05:01 | event time sits behind the watermark: late |
Read the last row closely. E3 ended at 09:02:14, before the watermark of 09:05:01, so the engine's assertion turned out false. A heuristic watermark is a bet about the network, and a phone in a garage loses that bet. The watermark held at 09:05:01 as E3 arrived, since E3 is older than events the engine had already seen. What the engine does with E3 is a separate setting, the subject of the next section.
Because this watermark is built from event times, a quiet stream holds it still. Window [09:05, 09:10) fires only when an event stamped at or after 09:11:00 arrives, however much wall-clock time passes first.
A job sums 5-minute windows with a 60-second watermark lag. Events arrive in this order, stamped 10:03:10, 10:05:30, 10:06:20, then 10:04:50. What happens to window [10:00, 10:05)?
- It fires when the 10:05:30 event arrives, and the 10:04:50 event is late.
- It fires when the 10:06:20 event arrives, holding both 10:03:10 and 10:04:50.
- It fires when the 10:06:20 event arrives, holding only the 10:03:10 event.
- It fires when the 10:04:50 event arrives, because that event completes the window.
Choosing the lag
- 10-second lag
- Totals publish about 10 seconds after a window ends
- Any event delayed past 10 seconds counts as late
- Open windows sit in memory briefly
- Suits alerting, where a fast rough answer beats a slow exact one
- 30-minute lag
- Totals publish about 30 minutes after a window ends
- Only events delayed past 30 minutes count as late
- Every key holds 30 minutes of open windows in memory
- Suits billing, where completeness outranks speed
The lag trades latency for completeness, so set it from measured skew. If the p99 of processed_at minus ended_at is 40 seconds, a 60-second lag leaves fewer than 1 event in 100 late in a steady stream, and every total waits about a minute past its window's end.
A team raises the watermark lag on its revenue job from 60 seconds to 10 minutes. What should it expect?
- Fewer late events and earlier totals, since the engine waits for stragglers.
- Fewer late events, totals about 9 minutes later, and more open windows in memory.
- The same late events, since lateness depends on allowed lateness alone.
- More late events, because a later watermark leaves more events behind it.
About this lesson
This section is part of the Schema Drift: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.