Temporal Types & DST

Concepts: dmDataTypes

Timestamps are the most dangerous data type in production systems because they look correct until daylight saving time happens, or until your pipeline runs in a different timezone than the system that generated the data. The most common bug: a pipeline running in UTC processes events timestamped in local time and shifts an entire day of data into the wrong partition. The DST Trap On the second Sunday of March in the US, clocks spring forward from 2:00 AM to 3:00 AM. The time 2:30 AM does not exist. On the first Sunday of November, clocks fall back from 2:00 AM to 1:00 AM. The time 1:30 AM occurs twice. If your timestamp column is TIMESTAMP (naive), and someone records an event at '2024-03-10 02:30:00' in Eastern time, that time literally did not happen. Your system now has impossible data.