Event Time Versus Processing Time

Concepts covered: Stream Processing

Every trip event carries 2 timestamps that answer different questions. Event time is when the thing happened, stamped by the phone: the trip ended at 08:47. Processing time is when the pipeline handled the event: the streaming job saw it at 09:12. The gap between them is skew. On a good network skew is a few seconds; for a phone in a garage or on a flight it runs to hours.

tripended_at (event time)processed_at (processing time)skew
T108:58:0008:58:033s
T208:59:4009:00:0525s
T308:47:0009:12:0025m
T409:10:0009:10:022s
T509:20:0009:20:011s
T608:55:0010:02:0067m

Count those 6 trips per hour and the 2 clocks disagree about every hour. T3 and T6 both ended in the 08:00 hour, yet T3 reached the pipeline at 09:12 and T6 at 10:02.

The 2 queries differ only in the column inside date_trunc. This one buckets by event time; putting processed_at in place of ended_at produces the right-hand table.

SELECT
  date_trunc('hour', ended_at) AS event_hour,
  count(*) AS trips
FROM trips
WHERE trip_date = DATE '2026-10-24'
GROUP BY date_trunc('hour', ended_at)
ORDER BY event_hour

The partition column inherits the same choice, and the query above assumes trip_date comes from ended_at. If trip_date comes from processed_at, a trip that ends at 23:50 and uploads at 00:40 lands in the next day's partition, and an event-time query for one day must scan 2 partitions to find all its trips.

Which clock answers which question

Ask whether the question is about the world or about the pipeline. Questions about what riders and drivers did use event time. Questions about load, delay and cost inside the system use processing time.

Consumer questionClockReason
Driver earnings for 08:00 to 09:00event timePay follows when the trip happened
Events ingested per minute, for capacity planningprocessing timeLoad lands when events arrive
Fraud rule: 5 trips on one card within 10 minutesevent timeThe rule describes rider behaviour
Delay from phone to table, p95bothDelay is processed_at minus ended_at
Alert when the job falls behindprocessing timeFalling behind is a property of the job

This query feeds the driver earnings page, which shows pay per hour of driving. What is wrong with it?

SELECT
  driver_id,
  date_trunc('hour', processed_at) AS pay_hour,
  sum(fare_cents) AS earned_cents
FROM trips
WHERE trip_date = DATE '2026-10-24'
GROUP BY driver_id, date_trunc('hour', processed_at)

During a concert surge, the on-call engineer wants a chart showing whether the streaming job keeps up with incoming events. Which timestamp should the chart bucket by?

About this lesson

This section is part of the Schema Drift: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.