Too Many Small Files
An easy Spark mock interview question on DataDriven, with an AI interviewer who asks follow-up questions and a hire or no-hire decision at the end.
- Domain
- Spark
- Difficulty
- easy
- Seniority
- L5
Interview Prompt
A daily export pipeline reads 200 GB from the transactions table and writes the flagged high-value records over $500 as Parquet to S3. Downstream Athena queries take 45 seconds for a simple COUNT(*), and S3 holds 2,000 Parquet files averaging 1 MB each even though the job already sets spark.sql.shuffle.partitions to 2000. Fix the export so it lands as a handful of properly sized files.
Summary
Two thousand files. One megabyte each. Athena says no.
How This Interview Works
- Read the vague prompt (just like a real interview)
- Ask clarifying questions to the AI interviewer
- Write and run your Spark code against the tables
- Answer the interviewer's follow-up questions, then get a hire or no-hire decision