Fix Skewed Viewing Events Pipeline

A hard Spark mock interview question on DataDriven, with an AI interviewer who asks follow-up questions and a hire or no-hire decision at the end.

Domain
Spark
Difficulty
hard
Seniority
L5

Interview Prompt

A streaming company's nightly `viewing_engagement` Spark job, which combines a large `event_data` table with a small `users` dimension on `user_id` to produce daily engagement counts by event type and account status, has blown its 60-minute SLA and is still running after 2 hours. Read the Spark UI evidence to find the root cause and fix the job so it meets SLA.

Summary

Your nightly Spark job just paged you. One task has 40% of the data.

How This Interview Works

  1. Read the vague prompt (just like a real interview)
  2. Ask clarifying questions to the AI interviewer
  3. Write and run your Spark code against the tables
  4. Answer the interviewer's follow-up questions, then get a hire or no-hire decision