Kill the UDF

A hard Spark mock interview question on DataDriven, with an AI interviewer who asks follow-up questions and a hire or no-hire decision at the end.

Domain
Spark
Difficulty
hard
Seniority
L5

Interview Prompt

Each row of `payments` is one card transaction, and the starter's Python `risk_score` UDF scores it from ten of its columns through thresholds, a night-hours check and weighted sums, flagging it against `FLAG_CUTOFF`. Return each transaction's `txn_id`, `merchant_id`, `risk_score` and `flagged`, scoring exactly as the UDF does but with no Python left in the plan, since at two billion rows a day the JVM-to-Python round trip is 70% of the job's 4 hours.

Summary

2 billion rows through Python. One UDF at a time.

How This Interview Works

  1. Read the vague prompt (just like a real interview)
  2. Ask clarifying questions to the AI interviewer
  3. Write and run your Spark code against the tables
  4. Answer the interviewer's follow-up questions, then get a hire or no-hire decision