Last week the upstream booking_events feed broke for 3 days, so fct_bookings emerged empty each morning. Our data-quality suite reported all expectations passing. The on-call only caught it when finance asked why the dashboard was flat. Run tests/test_quality_suite.py to reproduce. Fix the suite (in dq/expectations.py) so an empty input table fails LOUDLY instead of vacuously passing every row-level check.
dq/expectations.py
# dq/expectations.py # Great-Expectations-style suite for fct_bookings. Each expectation # returns dict(success=bool, observed=...). from typing import List, Dict, Any def expect_column_values_to_be_unique(rows: List[Dict[str, Any]], column: str): seen, dups = set(), 0 for r in rows: v = r.get(column) if v in seen: dups += 1 seen.add(v) return {'success': dups == 0, 'observed_dups': dups} def expect_column_values_to_not_be_null(rows: List[Dict[str, Any]], column: str): nulls = sum(1 for r in rows if r.get(column) is None) return {'success': nulls == 0, 'observed_nulls': nulls} def expect_column_values_to_be_between(rows: List[Dict[str, Any]], column: str, low: float, high: float): bad = sum(1 for r in rows if not (low <= r.get(column, 0) <= high)) return {'success': bad == 0, 'observed_out_of_range': bad} def run_suite(rows: List[Dict[str, Any]]) -> Dict[str, Any]: return { 'unique_booking_id': expect_column_values_to_be_unique(rows, 'booking_id'), 'guest_id_not_null': expect_column_values_to_not_be_null(rows, 'guest_id'), 'amount_in_range': expect_column_values_to_be_between(rows, 'amount_cents', 100, 10_000_000), }
Active Now|Sr. Data Engineer (L5)|||3.7k Attempts|1.7k Solves|
Data Quality Take-Home Exercise: The Silent Pipeline
An AI-assisted Data Quality coding round for data engineers at senior level. Work in a real IDE with an AI agent, then defend your changes to an interviewer.
- Stack
- Data Quality
- Format
- Take-Home Exercise
- Seniority
- Senior
- Estimated time
- 35 minutes
- Files in the repo
- 6
The Task
Last week the upstream booking_events feed broke for 3 days, so fct_bookings emerged empty each morning. Our data-quality suite reported all expectations passing. The on-call only caught it when finance asked why the dashboard was flat. Run tests/test_quality_suite.py to reproduce. Fix the suite (in dq/expectations.py) so an empty input table fails LOUDLY instead of vacuously passing every row-level check.
Summary
Silence, mistaken for success.
Repository Files
- dq/expectations.py (python)
- dq/runner.py (python)
- dq/io.py (python)
- dq/constraints.sql (sql)
- tests/test_quality_suite.py (python)
- requirements.txt (text)