Preventing Small Files at the Source

The best compaction strategy is not needing compaction. Preventing small files at the source is cheaper, simpler, and more reliable than cleaning them up after the fact. Prevention Strategies The Spark Repartition Trap