"Why Is My Query So Slow?"

What They're Really Testing The Unlock Every file has a fixed overhead: metadata, file handle, S3/HDFS listing entry, Spark task scheduling. When you have 1 million 1 KB files instead of 1,000 1 MB files, you have 1,000x the overhead for the same data. The query engine spends more time opening files than reading data. 150 bytes The 60-Second Framework How Small Files Are Created