Drift

A hard ai_coding interview practice problem on DataDriven. Write and execute real ai_coding code with instant grading.

Domain

ai_coding

Difficulty

hard

Seniority

L6

Problem

Our RAG pipeline calls a hosted embedding endpoint. Last week the vendor silently upgraded the model under the same endpoint URL; cosine distances on a held-out probe set shifted by 18% mean, and product saw a precision drop on a fixed eval set. We have no alert for this. Run tests/test_drift_alarm.py to reproduce. Build a drift-detection check in monitoring/drift.py that runs daily and fires when the embedding distribution shifts beyond a defensible threshold. Write the threshold logic so an interviewer can defend the false-positive rate.

Summary

Nothing changed. Everything moved.

Practice This Problem

Solve this ai_coding problem with real code execution. DataDriven runs your solution and grades it automatically.