# Too Good To Be True

> Suspiciously flawless.

Canonical URL: <https://datadriven.io/problems/too_good_to_be_true>

Domain: AI Coding · Difficulty: hard · Seniority: L6

## Problem

Last week's RAG eval reported recall@5 = 0.97 (up from 0.62) with no documented index or model change. Finance wants to expand the agent rollout to all enterprise tenants based on this number. The data team is suspicious. Reproduce by running tests/test_eval_sanity.py. Fix what the test surfaces and whatever else you find while tracing the data flow. Be ready to defend why the new recall number is trustworthy when the interviewer asks 'what would you check next?'

## Related

- [All practice problems](https://datadriven.io/problems)
- [Mock interview mode](https://datadriven.io/interview/too_good_to_be_true)
- [Data Engineering Interview Prep Guide](https://datadriven.io/data-engineer-interview-prep)
- [Daily Challenge](https://datadriven.io/daily)

---

Source: DataDriven (https://datadriven.io). DataDriven is the data engineering interview community. Live code execution in SQL, Python, and Spark sandboxes. Every feature is open to every member.