RAG Debug: Too Good To Be True

Last week's RAG eval reported recall@5 = 0.97 (up from 0.62) with no documented index or model change. Finance wants to expand the agent rollout to all enterprise tenants based on this number. The data team is suspicious. Reproduce by running tests/test_eval_sanity.py. Fix what the test surfaces and whatever else you find while tracing the data flow. Be ready to defend why the new recall number is trustworthy when the interviewer asks 'what would you check next?'

eval/run.py

# eval/run.py
# Eval harness for our customer-support RAG. Loads a labeled set of
# (question, gold_doc_id) pairs, retrieves top-k chunks per question,
# reports recall@k. Recall jumped from 0.62 to 0.97 last week with no
# index changes. Finance is asking us to extend the agent rollout to
# all enterprise tenants based on these numbers. Something is wrong.

from eval.dataset import load_eval_set
from eval.retriever import Retriever
from eval.metrics import recall_at_k

def main():
    eval_set = load_eval_set("data/support_eval.jsonl")
    retriever = Retriever.from_index("indexes/support.faiss")

    hits = 0
    for ex in eval_set:
        chunks = retriever.search(ex["question"], k=5)
        gold = ex["gold_doc_id"]
        if any(c["doc_id"] == gold for c in chunks):
            hits += 1
    print(f"recall@5 = {hits / len(eval_set):.3f} on n={len(eval_set)}")

if __name__ == "__main__":
    main()

# eval/run.py
# Eval harness for our customer-support RAG. Loads a labeled set of
# (question, gold_doc_id) pairs, retrieves top-k chunks per question,
# reports recall@k. Recall jumped from 0.62 to 0.97 last week with no
# index changes. Finance is asking us to extend the agent rollout to
# all enterprise tenants based on these numbers. Something is wrong.

from eval.dataset import load_eval_set
from eval.retriever import Retriever
from eval.metrics import recall_at_k

def main():
    eval_set = load_eval_set("data/support_eval.jsonl")
    retriever = Retriever.from_index("indexes/support.faiss")

    hits = 0
    for ex in eval_set:
        chunks = retriever.search(ex["question"], k=5)
        gold = ex["gold_doc_id"]
        if any(c["doc_id"] == gold for c in chunks):
            hits += 1
    print(f"recall@5 = {hits / len(eval_set):.3f} on n={len(eval_set)}")

if __name__ == "__main__":
    main()

Active Now|Staff Data Engineer (L6)|||4.3k Attempts|1.2k Solves|