Watermarks for Irregular Streams
Concepts covered: Stream Processing
Every watermark strategy assumes something about how late events arrive. The common default, bounded out-of-orderness, sets the watermark to the newest event time seen minus a fixed delay D. It works when lateness is steady and capped, as on a feed of order fills from a trading engine. Real sources break that assumption in 3 ways: a long tail of stragglers, partitions that fall silent, and clocks that run fast.
Profile the source before picking a strategy
| Source | Lateness profile | Idleness profile | Strategy that fits |
|---|---|---|---|
| Order fills from the trading engine | p99 under 5 minutes, steady | Quiet overnight | Fixed delay plus idle timeout |
| Mobile app events | Long tail: offline phones replay hours later | Rarely idle | Percentile delay plus late-event handling |
| Clearing-house settlement messages | Arrive in batches per cycle | Silent for hours between cycles | Producer markers |
| Field sensors | Small, but some clocks drift ahead | Devices drop off for days | Future-time cap plus idle timeout |
The 4 strategies and the late data each one misses
- Fixed delay: Predictable and cheap. Fails on a tail longer than D: those events arrive behind the watermark and miss their window.
- Percentile delay: D is recomputed from a trailing percentile, such as p99 lateness over the last hour. A watermark may never move backward, so a sudden rise in D freezes it, and a sudden fall lets it race ahead of stragglers.
- Idle timeout: A partition with no events for T is dropped from the minimum. When it wakes, its first events may sit behind a watermark that moved on without it.
- Producer markers: The producer sends a marker meaning nothing older than time X is coming. Exact for sparse batch sources, and useless when the producer cannot make that promise.
Settlement messages arrive in batches up to 3 hours long, with hours of silence between cycles. Which watermark strategy fits?
- Fixed delay of 30 seconds with a 2-minute idle timeout
- Percentile delay set from the trailing hour's p99 lateness
- A marker from the producer at the end of each cycle
- Fixed delay of 3 hours with no idle timeout
One quiet partition stalls the whole job
An engine reading several partitions takes the minimum of the per-partition watermarks, because the slowest partition bounds what it can promise. A partition that stops receiving events stops its own watermark, and the job's watermark stops with it. No window closes and no result is emitted, while every other partition keeps filling state. The table below is the trades job at 10:05:00, with D = 30 seconds.
| Partition | Newest event time | Last event received | Partition watermark |
|---|---|---|---|
| p0 | 10:04:58 | 10:05:00 | 10:04:28 |
| p1 | 10:04:51 | 10:04:59 | 10:04:21 |
| p2 | 09:58:10 | 09:58:12 | 09:57:40 |
Without an idle timeout the job watermark is 09:57:40, and no window after 09:57 can close. With a 2-minute idle timeout, p2 has been silent for nearly 7 minutes, so it drops out and the job watermark becomes 10:04:21. The price comes later: if p2 resumes with events stamped 09:58:30, they arrive nearly 6 minutes behind the watermark. They are late, and whether they still count depends on the allowed-lateness budget the next section sizes.
A fast clock is worse than a slow partition
A sender whose clock runs 3 hours ahead stamps an event 13:05 when real time is 10:05. Its partition's newest event time leaps to 13:05 and that partition's watermark to 13:04:30. The minimum across partitions hides this only while the others stay active. Once idle timeouts drop them, the fast partition sets the job watermark alone, it jumps 3 hours, and every honest event after it is late. The defense is a cap: clamp event time to processing time plus 5 seconds, and quarantine anything stamped beyond that.
from datetime import timedelta
DELAY = timedelta(seconds=30)
IDLE_TIMEOUT = timedelta(minutes=2)
FUTURE_TOLERANCE = timedelta(seconds=5)
def job_watermark(partitions, now, previous):
candidates = []
for p in partitions:
if now - p["last_received"] > IDLE_TIMEOUT:
continue
newest = min(p["newest_event_time"], now + FUTURE_TOLERANCE)
candidates.append(newest - DELAY)
if not candidates:
return previous
if previous is None:
return min(candidates)
return max(previous, min(candidates))Start from the partition table above at 10:05:00 with a 2-minute idle timeout. p2 sends nothing more, and p1 goes silent after 10:04:59. At 10:07:30, p0's newest event time is 10:07:28. What is the job watermark?
- 10:06:58, from p0 alone, because p1 and p2 are both past the idle timeout
- 10:04:21, from p1, because a silent partition holds its last watermark
- 09:57:40, from p2, because the minimum includes every partition ever seen
- 10:07:28, from p0, because idle partitions also remove the 30-second delay
About this lesson
This section is part of the Schema Drift: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.