Kill the UDF
A hard Spark mock interview question on DataDriven, with an AI interviewer who asks follow-up questions and a hire or no-hire decision at the end.
- Domain
- Spark
- Difficulty
- hard
- Seniority
- L5
Interview Prompt
Each row of `payments` is one card transaction, and the starter's Python `risk_score` UDF scores it from ten of its columns through thresholds, a night-hours check and weighted sums, flagging it against `FLAG_CUTOFF`. Return each transaction's `txn_id`, `merchant_id`, `risk_score` and `flagged`, scoring exactly as the UDF does but with no Python left in the plan, since at two billion rows a day the JVM-to-Python round trip is 70% of the job's 4 hours.
Summary
2 billion rows through Python. One UDF at a time.
How This Interview Works
- Read the vague prompt (just like a real interview)
- Ask clarifying questions to the AI interviewer
- Write and run your Spark code against the tables
- Answer the interviewer's follow-up questions, then get a hire or no-hire decision