Why is Recall@K the right offline metric for RAG?

Category: Database & RAG

Answer

RAG quality = how often the right chunk lands in the top-K. Recall@K = fraction of queries where the ground-truth-correct chunk appears in the top-K. Optimize for Recall@K offline, then track end-to-end answer quality online with golden-set evals. Precision@K matters too if the LLM context window is tight.

Concrete examples from the fca project context

Example 1

Build a golden set: 50 questions with hand-labeled correct chunks.

Example 2

Measure Recall@5, Recall@10, MRR; pick the chunking that maximizes Recall@10 at acceptable p95.

Example 3

Track drift: per-week eval regressions catch schema or embedding-model changes.

Data flow / flow chart

golden_q -> ANN top-K -> check if ground_truth is in top-K
  Report mean recall@K over the set

Takeaway

Online A/B is too slow for retrofit. Recall@K on a golden set is the cadence you can run daily.