Our RAG pipeline calls a hosted embedding endpoint. Last week the vendor silently upgraded the model under the same endpoint URL; cosine distances on a held-out probe set shifted by 18% mean, and product saw a precision drop on a fixed eval set. We have no alert for this. Run tests/test_drift_alarm.py to reproduce. Build a drift-detection check in monitoring/drift.py that runs daily and fires when the embedding distribution shifts beyond a defensible threshold. Write the threshold logic so an interviewer can defend the false-positive rate.
monitoring/drift.py
# monitoring/drift.py # Daily drift check on the embedding distribution. We embed a fixed # probe set (monitoring/probes.json) every day, store the result, and # compare today's vs. the rolling baseline. # # Implement detect_drift() so it returns dict(alarmed: bool, reason: str). # The grader will call it on (a) a stable history, (b) a sudden shift, and # (c) a slow walk that should NOT alarm in any single day. from typing import List, Dict, Any from monitoring.history import load_history, append_today from monitoring.embed import embed_probes def detect_drift() -> Dict[str, Any]: history = load_history() # list of daily summaries, oldest first today = embed_probes() append_today(today) # TODO: implement. return {'alarmed': False, 'reason': 'not implemented'}
Active Now|Staff Data Engineer (L6)|||5.7k Attempts|2.7k Solves|
RAG Take-Home Exercise: Drift
An AI-assisted RAG coding round for data engineers at staff level. Work in a real IDE with an AI agent, then defend your changes to an interviewer.
- Stack
- RAG
- Format
- Take-Home Exercise
- Seniority
- Staff
- Estimated time
- 60 minutes
- Files in the repo
- 8
The Task
Our RAG pipeline calls a hosted embedding endpoint. Last week the vendor silently upgraded the model under the same endpoint URL; cosine distances on a held-out probe set shifted by 18% mean, and product saw a precision drop on a fixed eval set. We have no alert for this. Run tests/test_drift_alarm.py to reproduce. Build a drift-detection check in monitoring/drift.py that runs daily and fires when the embedding distribution shifts beyond a defensible threshold. Write the threshold logic so an interviewer can defend the false-positive rate.
Summary
Nothing changed. Everything moved.
Repository Files
- monitoring/drift.py (python)
- monitoring/probes.py (python)
- monitoring/constraints.sql (sql)
- monitoring/embed.py (python)
- monitoring/history.py (python)
- monitoring/stats.py (python)
- tests/test_drift_alarm.py (python)
- requirements.txt (text)