AI & Automation

Evaluating Retrieval Quality

Jabane Mohamed Ayoub4 min read

Evaluating Retrieval Quality title card on a pink-to-purple gradient, reading: Measure whether the right chunk was retrieved, separately
Table of contents
  1. Separate retrieval quality from answer quality
  2. Build a labeled eval set from real queries
  3. Change one thing at a time, and re-run the eval set
  4. Human spot-checks still matter
  5. Step by step
  6. Pitfalls
  7. FAQ
  8. Related resources

A fluent, confident answer tells you almost nothing about whether retrieval found the right chunk. A synthesis model can write a reasonable-sounding response from a mediocre source, a partially relevant one, or occasionally the wrong one entirely, and the fluency hides all of it. Evaluating a RAG system by reading its final answers measures the model’s writing, not the retrieval it’s writing from.

Those are two different things, and they need two different kinds of evaluation.

Separate retrieval quality from answer quality

Retrieval quality asks one question: given this query, did the right chunk show up near the top of the ranked list? Answer quality asks a different one: given the retrieved chunks, did the model write a good response from them? A system can score well on one and badly on the other. A great synthesis model papering over bad retrieval will still produce answers that read fine and are quietly wrong. A mediocre synthesis model given perfect retrieval will still produce something useful, just less polished.

Measure retrieval on its own, with its own metrics, before trusting any judgment about the system as a whole.

Build a labeled eval set from real queries

Precision and recall at a fixed rank, usually written precision@k and recall@k, need a set of real questions paired with the chunk or chunks that actually answer each one. Pull these from real traffic, not invented examples: the questions people actually ask tend to be messier, more ambiguous, and more revealing than the clean ones a team writes for a test.

Twenty to fifty labeled questions is enough to start catching regressions. It doesn’t need to be exhaustive to be useful, it just needs to be real.

flowchart LR
  accTitle: Evaluating retrieval quality on purpose
  accDescr: Real queries get labeled with their correct source chunks to build an eval set. A retrieval change is tested against that set to measure precision and recall before it ships, instead of being judged only by whether the final generated answers sound reasonable.

  Q[Real queries from traffic]
  LABEL[Labeled with correct<br/>source chunks]
  CHANGE[A retrieval change:<br/>new reranker, new weighting]
  SCORE[Precision@k, recall@k<br/>measured against the eval set]
  SHIP{Did it improve<br/>or regress?}

  Q --> LABEL --> SCORE
  CHANGE --> SCORE
  SCORE --> SHIP

Change one thing at a time, and re-run the eval set

A retrieval pipeline usually has several moving parts: embedding model, reranker, fusion weights, age decay, chunk size. Change more than one before re-running the eval set and a regression in one gets masked, or credited to the wrong change entirely.

Nexus ships a built-in eval runner specifically for this: it A/B tests the optional graph retriever against plain hybrid search on the same query set, so the decision to turn it on for a given project is based on a measured difference, not a feeling that the answers seemed better. The same idea applies to any single change: swap the reranker, keep everything else fixed, measure the delta.

Human spot-checks still matter

Automated metrics catch regressions at scale, but they don’t catch everything a labeled set can’t anticipate: a chunk that’s technically the “correct” one but has gone stale since it was labeled, or a new kind of query the eval set doesn’t cover yet. Spot-check a sample of real answers against their cited sources on a schedule, not just when a metric looks suspicious.

Step by step

  1. Pull 20 to 50 real questions from actual usage, not invented ones, and label each with the chunk or chunks that actually answer it.
  2. Measure precision@k and recall@k for the current pipeline before changing anything, so there’s a baseline to compare against.
  3. Change exactly one component (reranker, fusion weights, chunk size, embedding model) and re-run the same eval set.
  4. Compare against the baseline, not against intuition. A change that feels like an improvement but scores worse on the eval set is a change worth reverting.
  5. Add newly discovered failure cases to the eval set as you find them, so the set grows more representative over time instead of staying frozen at whatever it looked like on day one.

Pitfalls

  • Judging retrieval only by reading final answers. This is the single most common mistake, and it’s the one that lets a bad retriever hide behind a good writer for months.
  • An eval set that never grows. A fixed set from six months ago stops reflecting the questions people actually ask today.
  • Testing only “easy” queries the current system already handles well. Include queries you know are hard: ones with ambiguous wording, ones where the right answer is short or buried, ones where a near-miss chunk exists and shouldn’t win.

FAQ

How big does an eval set need to be before it's useful?

Useful starts earlier than most people expect: 20 to 50 real, labeled questions will catch most meaningful regressions. The value comes from the questions being real and the labels being accurate, not from the set being large.

What's the difference between precision@k and recall@k?

Precision@k asks what fraction of the top k retrieved results are actually relevant. Recall@k asks what fraction of all relevant results made it into the top k. A system can have high precision and low recall if it returns a few good results but misses others, or the reverse if it returns everything relevant buried among a lot of noise.

Can you evaluate retrieval without a labeled eval set at all?

Only loosely, by spot-checking outputs by hand, which doesn’t scale and doesn’t catch regressions reliably. A labeled set is what turns “the answers seem fine” into a number you can compare before and after a change, which is the entire point of evaluating on purpose instead of by feel.