Spark Sorting — The Question Every Senior DE Interview Asks

If you're interviewing for a senior DE role at any company that uses Spark — which is most of them — you will be asked about Spark sorting. The specific question: 'What's the difference between orderBy and sortWithinPartitions, and when do you use each?' Getting this wrong signals you haven't worked with Spark in production. Getting it right with the right vocabulary is a strong positive signal. orderBy vs sortWithinPartitions AQE — Adaptive Query Execution