Data Engineering Challenges

SQL challenges reward the habit of asking what 1 row means before writing a line. Hidden failures mostly come from duplicates that inflate a count, NULLs that silently drop rows from a filter or a NOT IN, and ties that make a ranking ambiguous.

Ties are the cleanest example. Take scores of 90, 90, 80. ROW_NUMBER() returns 1, 2, 3, because it numbers rows and breaks the tie arbitrarily. RANK() returns 1, 1, 3 and skips a value after the tie. DENSE_RANK() returns 1, 1, 2. "The top 2 scores" has a different answer under each, so the first move is to ask which one the brief means.

Work more of these in the SQL practice problems, and the SQL interview questions show how the same patterns are asked out loud.

A data engineering Python challenge is rarely an algorithm puzzle. It hands you a log, a list of events or a half-broken feed and asks for counts, with the duplicates removed or the gaps filled. The tools are plain: dictionaries, sets, sorting and string methods.

The traps are the same each time. An empty input should return an empty result, not raise. Counting by id needs a decision about ids that differ only in case or whitespace. When you deduplicate events and must keep the first, order matters. A dict keeps insertion order, which the language has guaranteed since Python 3.7, so writing an id only when it's absent preserves the original timeline.

Solve them against real inputs in the Python practice problems, and see how interviewers phrase them in the Python interview questions.

A modeling challenge gives you a business and asks for tables. Every other decision depends on the grain. Kimball's technique page on declaring the grain says it establishes exactly what a single fact table row represents.

The cost of skipping it is arithmetic. Suppose 3 orders carry a $5 shipping fee each, spread over 3, 2, 2 order lines, 7 lines in all. At order grain the fee totals $15. Put the fee on the line-grain table and sum it, and every line repeats it: $35, which is $20 too much. Hidden data catches this because the sample often has 1 line per order.

History is the other recurring test. When a customer moves, a type 2 slowly changing dimension keeps 1 row per period, with validity dates, so old facts still point at where the customer lived then. The data modeling practice problems include both, and the data modeling interview questions cover how to defend a grain choice.

DataDriven runs a Weekly, 1 community data engineering challenge a week. Each comes with a brief, a small sample dataset and a starter file, and is scored blind on hidden data.

Blind scoring changes how you work. The sample is small, so it can only show the shape of the data, never every case in it. A solution tuned to the sample passes the sample and nothing else, so the habit that pays is building your own cases: a duplicate, a null, a tie, an empty day. The discussion forum is where members talk through approaches.

Duplicates come from a retried load or a doubled event, and the practice is idempotent writes and deduplication with a window function. Late data arrives after the day was reported, and the practice is reprocessing a closed partition without double counting. Schema drift is an upstream team renaming or dropping a column, and the practice is defensive parsing plus a check on the shape of each load.

A wrong grain shows up as a fee or a total that repeats after a join, and the practice is declaring grain before you model or join. Slow jobs usually mean 1 value holds most of the data or thousands of tiny files slow the reads, and the practice is reading a Spark plan and fixing skew by salting or by compacting files. Each of these has a challenge that trains it, so the problems of the field and the exercises are the same list.

A challenge has a check, so you know in minutes whether you're right. A project has no check and runs for days, but it teaches how a stack fits together. A take-home sits between them: a fixed brief like a challenge, with the scope of a small project and a reviewer who reads your reasoning as well as your output. The sensible order is challenges first for precision, then a project for breadth.

To choose a project, see the data engineering projects list. For more problems to work now, the data engineer practice problems page groups them by domain, and the full practice problems catalogue lets you filter by skill and difficulty.