Watermarks for Irregular Streams

Concepts covered: Stream Processing

Every watermark strategy assumes something about how late events arrive. The common default, bounded out-of-orderness, sets the watermark to the newest event time seen minus a fixed delay D. It works when lateness is steady and capped, as on a feed of order fills from a trading engine. Real sources break that assumption in 3 ways: a long tail of stragglers, partitions that fall silent, and clocks that run fast.

Profile the source before picking a strategy

SourceLateness profileIdleness profileStrategy that fits
Order fills from the trading enginep99 under 5 minutes, steadyQuiet overnightFixed delay plus idle timeout
Mobile app eventsLong tail: offline phones replay hours laterRarely idlePercentile delay plus late-event handling
Clearing-house settlement messagesArrive in batches per cycleSilent for hours between cyclesProducer markers
Field sensorsSmall, but some clocks drift aheadDevices drop off for daysFuture-time cap plus idle timeout

The 4 strategies and the late data each one misses

Settlement messages arrive in batches up to 3 hours long, with hours of silence between cycles. Which watermark strategy fits?

One quiet partition stalls the whole job

An engine reading several partitions takes the minimum of the per-partition watermarks, because the slowest partition bounds what it can promise. A partition that stops receiving events stops its own watermark, and the job's watermark stops with it. No window closes and no result is emitted, while every other partition keeps filling state. The table below is the trades job at 10:05:00, with D = 30 seconds.

PartitionNewest event timeLast event receivedPartition watermark
p010:04:5810:05:0010:04:28
p110:04:5110:04:5910:04:21
p209:58:1009:58:1209:57:40

Without an idle timeout the job watermark is 09:57:40, and no window after 09:57 can close. With a 2-minute idle timeout, p2 has been silent for nearly 7 minutes, so it drops out and the job watermark becomes 10:04:21. The price comes later: if p2 resumes with events stamped 09:58:30, they arrive nearly 6 minutes behind the watermark. They are late, and whether they still count depends on the allowed-lateness budget the next section sizes.

A fast clock is worse than a slow partition

A sender whose clock runs 3 hours ahead stamps an event 13:05 when real time is 10:05. Its partition's newest event time leaps to 13:05 and that partition's watermark to 13:04:30. The minimum across partitions hides this only while the others stay active. Once idle timeouts drop them, the fast partition sets the job watermark alone, it jumps 3 hours, and every honest event after it is late. The defense is a cap: clamp event time to processing time plus 5 seconds, and quarantine anything stamped beyond that.

from datetime import timedelta

DELAY = timedelta(seconds=30)
IDLE_TIMEOUT = timedelta(minutes=2)
FUTURE_TOLERANCE = timedelta(seconds=5)


def job_watermark(partitions, now, previous):
    candidates = []
    for p in partitions:
        if now - p["last_received"] > IDLE_TIMEOUT:
            continue
        newest = min(p["newest_event_time"], now + FUTURE_TOLERANCE)
        candidates.append(newest - DELAY)
    if not candidates:
        return previous
    if previous is None:
        return min(candidates)
    return max(previous, min(candidates))

Start from the partition table above at 10:05:00 with a 2-minute idle timeout. p2 sends nothing more, and p1 goes silent after 10:04:59. At 10:07:30, p0's newest event time is 10:07:28. What is the job watermark?

About this lesson

This section is part of the Schema Drift: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.