File Parsing for Data Engineers: Staff+ Level

Row group statistics, bloom filters, and the small file problem that ate the cluster.

Row group statistics, bloom filters, and the small file problem that ate the cluster.

Category

Python

Difficulty

advanced

Duration

25 minutes

Challenges

0 hands-on challenges

Topics covered: Parquet Column Encoding: Beyond 'It Uses Compression', Bloom Filters and Late Materialization, Delta Lake vs Apache Iceberg: The Real Differences, The Small File Problem: Diagnosis and Compaction, Designing a File Format Strategy for a New Data Platform

Lesson Sections

1. Parquet Column Encoding: Beyond 'It Uses Compression'

2. Bloom Filters and Late Materialization

3. Delta Lake vs Apache Iceberg: The Real Differences

4. The Small File Problem: Diagnosis and Compaction

5. Designing a File Format Strategy for a New Data Platform