AI Coding Interview Questions for Data Engineers

1. The Late Arrival

Asked in a Data Engineer interview by ShopifyEasy~30 minOpen the workspace

Finance reports the daily orders dashboard is missing roughly 3% of revenue versus the source system of record. Reproduce by running tests/test_late_events.py. Fix the pipeline so the test passes; keep any currently-passing tests green. Be ready to walk an interviewer through what changed and how a re-run on the next hour does or does not double-count.

Why this matters. A schedule-time versus event-time bug: the DAG aggregates whatever arrived by the run, and 3% of orders land after the window closes. The fix is a lookback that reprocesses recent windows, and the interviewer wants the explanation as much as the patch: why late data makes any on-schedule aggregate provisional, and what bounds the lookback.

2. One in Five

Asked in a Data Engineer interview by StripeEasy~25 minOpen the workspace

Analytics ships fct_payments through dbt nightly. The unique test on payment_id passes locally every time, fails in CI ~20% of runs. Same input fixtures, same model SQL. Run tests/test_uniqueness.py to reproduce, then fix the model so the test is deterministic. The fix is in the SQL, not the test.

Why this matters. Green locally, red in 20% of CI runs is the fingerprint of nondeterminism, here a dedup whose ordering ties on identical timestamps and picks a different survivor per run. The fix is a complete tiebreaker in the model's ordering, and the discipline it demonstrates, deterministic transforms, is what the round is checking.

3. Ghosts of Snapshots Past

Asked in a Data Engineer interview by NetflixEasy~30 minOpen the workspace

Yesterday a bad backfill wrote 2M corrupt rows into events_iceberg. We rolled back to the prior snapshot. Now an analyst's AS OF (timestamp) query for a window two days before the rollback is also returning the corrupt rows. Reproduce by running tests/test_time_travel.py. The bug is in how our reader resolves a snapshot for a timestamp; fix it so AS OF queries reflect what was visible at that time.

Why this matters. Rollback does not erase history; it appends a new snapshot pointing at old data. The reader resolves AS OF by wall-clock timestamp, which still lands inside the corrupt lineage. Understanding snapshot ids versus timestamp resolution, and why time travel needs the former after a rollback, is the entire bug.

4. Counted Twice

Asked in a Data Engineer interview by DoorDashEasy~30 minOpen the workspace

Restaurant partners are seeing ~12% inflated revenue in our partner-ping job. The job builds daily_restaurant_revenue from the orders + dropoffs tables. Run tests/test_no_double_count.py to reproduce: rows where one order has multiple dropoffs (multi-stop deliveries) get counted once per dropoff. Fix the SQL in queries/daily_restaurant_revenue.sql so revenue is per order, not per (order, dropoff).

Why this matters. Join fanout wearing a pager: one order, multiple dropoffs, revenue counted once per dropoff. Aggregate to order grain before joining, or dedupe the join key. The same fanout that inflates SQL screens inflates production revenue, and the agentic twist is proving it with the failing check before touching the code.

5. Four Million Echoes

Asked in a Data Engineer interview by ConfluentMedium~40 minOpen the workspace

Our orders enrichment consumer reprocesses ~4M messages every time we deploy. The team added enable_auto_commit=False and explicit commit() calls weeks ago, so this should have stopped. Run tests/test_offsets_persist.py to reproduce. The reprocess window correlates with rebalance events. Find why the explicit commits aren't sticking and fix it. Bonus credit: explain what 'cooperative-sticky' would have changed.

Why this matters. Manual commits alone do not survive rebalances: without a rebalance listener committing on partition revoke, the new assignment resumes from stale offsets and replays 4M messages. The correlation with deploys is the tell. Commit on revoke, and idempotent downstream writes make whatever slips through boring.

6. The Silent Pipeline

Asked in a Data Engineer interview by AirbnbMedium~35 minOpen the workspace

Last week the upstream booking_events feed broke for 3 days, so fct_bookings emerged empty each morning. Our data-quality suite reported all expectations passing. The on-call only caught it when finance asked why the dashboard was flat. Run tests/test_quality_suite.py to reproduce. Fix the suite (in dq/expectations.py) so an empty input table fails LOUDLY instead of vacuously passing every row-level check.

Why this matters. Vacuous truth: every row-level expectation passes when there are no rows. A quality suite needs existence assertions, volume against a rolling baseline and freshness against the clock, or it is a rubber stamp. 3 days of empty tables with a green suite is the exact failure this question replays.

7. The Skew

Asked in a Data Engineer interview by UberMedium~45 minOpen the workspace

ML team's fraud model trains at 0.84 AUC offline and tanks online. We isolated it to feature `user_orders_last_24h` , offline backfills compute it correctly, but the online value at scoring time is consistently higher. Run tests/test_feature_parity.py to see the divergence on a known prediction. Find the time-leak in the offline pipeline (features/orders_24h.py) and fix it. Defend your fix: what's the upper bound on staleness you've now committed the team to?

Why this matters. Train-serve skew in one feature: the offline backfill computes over closed windows while the online path counts in-flight events, so serving values run hot. The fix pins both paths to one definition, and the parity check on a known prediction is the pattern worth naming: features are contracts, not code in 2 places.

8. History Erased

Asked in a Data Engineer interview by SnowflakeMedium~40 minOpen the workspace

dim_customer is supposed to be SCD-2: every change to address/email creates a new row with effective_from / effective_to and is_current. After today's incremental run, the table has exactly one row per customer , history wiped. Run tests/test_scd2_history.py to reproduce. Fix sql/merge_dim_customer.sql so changes append a new version and close the old one, instead of overwriting.

Why this matters. A MERGE whose matched clause updates in place is Type 1 behavior wearing an SCD2 costume: one run and the history is gone. The correct shape matches on business key plus is_current, expires the old version, inserts the new one. Idempotence under rerun is the follow-up, and the answer is keying on content change, not run date.

9. One Bad Executor

Asked in a Data Engineer interview by UberMedium~35 minOpen the workspace

On-call paged at 02:14 last night: the nightly orders enrichment job ran 3h 40m instead of its usual 22 minutes and one executor OOMed on stage 4. Cluster spend tripled. Volumes only grew about 8% week-over-week. The job lives in jobs/. Reproduce by running tests/test_skew_observability.py. Fix it so the test passes. Be ready to defend the trade-offs of your fix when the interviewer asks what happens with a different data shape.

Why this matters. A 22-minute job at 3h40m with one OOMing executor and only 8% volume growth is skew, not scale: one hot key concentrated a stage on one task. The agentic discipline is reading the stage evidence before prompting, then directing the fix (broadcast or salt, with the tradeoff said aloud) rather than accepting a memory-bump patch that hides it for a week.

10. Too Good To Be True

Asked in a Data Engineer interview by AnthropicHard~45 minOpen the workspace

Last week's RAG eval reported recall@5 = 0.97 (up from 0.62) with no documented index or model change. Finance wants to expand the agent rollout to all enterprise tenants based on this number. The data team is suspicious. Reproduce by running tests/test_eval_sanity.py. Fix what the test surfaces and whatever else you find while tracing the data flow. Be ready to defend why the new recall number is trustworthy when the interviewer asks 'what would you check next?'

Why this matters. A metric that jumps from 0.62 to 0.97 with no change log is not good news, it is leakage: eval queries bleeding into the index. The fix is provenance and dedup between corpus and eval set, plus a sanity gate that flags too-good deltas. Staff-level judgment here is distrusting improvement you cannot attribute.

11. The Materialization Storm

Asked in a Data Engineer interview by AnthropicHard~60 minOpen the workspace

We added a new dataset to our Dagster repo last week. It declared 50 dynamic-partition assets that depend on shared upstream assets. Dagster scheduler CPU pegged at 100% within an hour and we're now seeing 12,000 materialization runs per hour for what should be ~50. Run tests/test_no_runaway_runs.py to reproduce. Find the cycle/fanout pattern in dagster_repo/assets.py and fix it so the partition fan-out is bounded and the run count is sane. Be ready to defend the trade-off your fix makes.

Why this matters. 50 dynamic partitions multiplying through shared upstream dependencies into 12,000 runs an hour is a graph-shape bug: the declaration created a materialization edge per partition per upstream. The fix collapses the fan-in and scopes the automation policy, and the judgment being scored is recognizing an orchestration graph as code you review, not config you trust.

12. The Quiet Partition

Asked in a Data Engineer interview by LinkedInHard~55 minOpen the workspace

Our Flink job aggregates user sessions and writes hourly rollups. Each midnight one source partition (the EU one) falls quiet for ~2 hours during low-traffic. During that window the global watermark stops advancing, no session windows close, and state size grows unbounded until the JM kills the job. Run tests/test_quiet_partition.py to reproduce. Find the watermarking pattern in jobs/sessions.py and fix it. Defend the new behavior: what's the maximum end-to-end latency you've now committed to?

Why this matters. The global watermark is the minimum across partitions, so one quiet partition freezes every window and state grows until the job dies. Idleness handling (withIdleness or per-partition idle timeouts) lets the watermark advance past silence. Knowing watermark-as-minimum is the crux; the config is a detail.

13. Drift

Asked in a Data Engineer interview by OpenAIHard~60 minOpen the workspace

Our RAG pipeline calls a hosted embedding endpoint. Last week the vendor silently upgraded the model under the same endpoint URL; cosine distances on a held-out probe set shifted by 18% mean, and product saw a precision drop on a fixed eval set. We have no alert for this. Run tests/test_drift_alarm.py to reproduce. Build a drift-detection check in monitoring/drift.py that runs daily and fires when the embedding distribution shifts beyond a defensible threshold. Write the threshold logic so an interviewer can defend the false-positive rate.

Why this matters. A vendor swapping the model under a stable endpoint is a schema change nobody announced. The defense is treating external models as versioned dependencies: a fixed probe set, distance-distribution monitoring, an alert on shift, and pinned versions where the API allows. The feature to build is the alarm, not the outrage.