Event Time Versus Processing Time
Concepts covered: Stream Processing
Every trip event carries 2 timestamps that answer different questions. Event time is when the thing happened, stamped by the phone: the trip ended at 08:47. Processing time is when the pipeline handled the event: the streaming job saw it at 09:12. The gap between them is skew. On a good network skew is a few seconds; for a phone in a garage or on a flight it runs to hours.
| trip | ended_at (event time) | processed_at (processing time) | skew |
|---|---|---|---|
| T1 | 08:58:00 | 08:58:03 | 3s |
| T2 | 08:59:40 | 09:00:05 | 25s |
| T3 | 08:47:00 | 09:12:00 | 25m |
| T4 | 09:10:00 | 09:10:02 | 2s |
| T5 | 09:20:00 | 09:20:01 | 1s |
| T6 | 08:55:00 | 10:02:00 | 67m |
Count those 6 trips per hour and the 2 clocks disagree about every hour. T3 and T6 both ended in the 08:00 hour, yet T3 reached the pipeline at 09:12 and T6 at 10:02.
The 2 queries differ only in the column inside date_trunc. This one buckets by event time; putting processed_at in place of ended_at produces the right-hand table.
SELECT
date_trunc('hour', ended_at) AS event_hour,
count(*) AS trips
FROM trips
WHERE trip_date = DATE '2026-10-24'
GROUP BY date_trunc('hour', ended_at)
ORDER BY event_hourThe partition column inherits the same choice, and the query above assumes trip_date comes from ended_at. If trip_date comes from processed_at, a trip that ends at 23:50 and uploads at 00:40 lands in the next day's partition, and an event-time query for one day must scan 2 partitions to find all its trips.
Which clock answers which question
Ask whether the question is about the world or about the pipeline. Questions about what riders and drivers did use event time. Questions about load, delay and cost inside the system use processing time.
| Consumer question | Clock | Reason |
|---|---|---|
| Driver earnings for 08:00 to 09:00 | event time | Pay follows when the trip happened |
| Events ingested per minute, for capacity planning | processing time | Load lands when events arrive |
| Fraud rule: 5 trips on one card within 10 minutes | event time | The rule describes rider behaviour |
| Delay from phone to table, p95 | both | Delay is processed_at minus ended_at |
| Alert when the job falls behind | processing time | Falling behind is a property of the job |
This query feeds the driver earnings page, which shows pay per hour of driving. What is wrong with it?
SELECT
driver_id,
date_trunc('hour', processed_at) AS pay_hour,
sum(fare_cents) AS earned_cents
FROM trips
WHERE trip_date = DATE '2026-10-24'
GROUP BY driver_id, date_trunc('hour', processed_at)- Nothing, because processed_at is the more reliable of the 2 clocks for money.
- It should group by trip_date alone, since hourly pay needs no timestamp.
- It should sum fare, because fare_cents may still be null on old rows.
- It buckets by processed_at, so a trip that ended at 08:47 is paid in the 09:00 hour.
During a concert surge, the on-call engineer wants a chart showing whether the streaming job keeps up with incoming events. Which timestamp should the chart bucket by?
- processed_at, because keeping up is about when the job handled each event.
- ended_at, because the surge is defined by when riders finished their trips.
- Either one, because both timestamps fall in the same hour for most trips.
- ended_at on the axis, with rows filtered to processed_at in the last hour.
About this lesson
This section is part of the Schema Drift: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.