The Word Count Shuffle Trap
An easy Spark mock interview question on DataDriven, with an AI interviewer who asks follow-up questions and a hire or no-hire decision at the end.
- Domain
- Spark
- Difficulty
- easy
- Seniority
- L5
Interview Prompt
Ever since the input tripled, the nightly tally of calls per response status that an on-call engineer owns has died with 48 GB of shuffle write and three executors out of memory, because it collects every single call for a status code onto one machine before counting them. She needs the same per-status call totals from `api_calls`, produced in a way that stays small no matter how many calls pile onto the busiest status code.
Summary
groupByKey works. Your cluster disagrees.
How This Interview Works
- Read the vague prompt (just like a real interview)
- Ask clarifying questions to the AI interviewer
- Write and run your Spark code against the tables
- Answer the interviewer's follow-up questions, then get a hire or no-hire decision