An IoT pipeline writes hourly batch output as GZIP-compressed CSV files in a partitioned curated zon
A medium Pipeline Design mock interview question on DataDriven. Practice with AI-powered feedback, real code execution, and a hire/no-hire decision.
- Domain
- Pipeline Design
- Difficulty
- medium
Interview Prompt
An IoT pipeline writes hourly batch output as GZIP-compressed CSV files in a partitioned curated zone. Each 5GB file is not splittable, so Spark collapses to one task per partition and the dashboard takes 40 minutes. Apply the section's compression-and-splittability framing and replace the curated GZIP-CSV format with the splittable columnar format the section names as the de facto choice.
How This Interview Works
- Read the vague prompt (just like a real interview)
- Ask clarifying questions to the AI interviewer
- Write your pipeline design solution with real code execution
- Get instant feedback and a hire/no-hire decision