Interview Bank

From Frontend Engineer to Agent Engineer in 30 Days

D24 RAG, Level Up: Hybrid Search, Reranking, Citations, Recall Evaluation

• How do you evaluate retrieval quality in a RAG system, and how should the evaluation set be built?怎么评估一个 RAG 系统的检索效果？评估集应该怎么构造？

  Common in ChinaCommon overseasDeep dive#rag#evaluation#recall

  How to reason about it · think before answering

  1. This is a very common question in the Chinese market and the fastest way to expose someone who has assembled RAG but never tuned it. The test: does your answer contain concrete metric names and an annotation granularity?
  2. First separate what is being evaluated — the step people most often conflate. Retrieval evaluation asks 'was it found'; generation evaluation asks 'was the answer right'. Keep two separate sets. Merge them and, when the score drops, you cannot tell whether retrieval missed or the model fumbled — and those have completely different fixes.
  3. Shape of the set: about 20 queries, each annotated with 1-3 chunk ids that must be retrieved. Annotate at chunk level, not document level — chunks are the retrieval unit, and document-level labels inflate the numbers. Cover the real query mix, especially the types you know break: codes, paraphrase, cross-document.
  4. Three metrics, three questions. recall@5 is what actually reaches the model, so it is the number you care about. recall@20 is the ceiling — if it does not move, the problem is on the recall side and no reranker will save you. MRR is sensitive to ordering and breaks ties when recall is equal.
  5. Production view: freeze the set once agreed, because changing samples destroys comparability — the same reason a factory keeps fixed reference samples. Pair it with online counterparts (empty-citation rate, hallucinated-citation rate, escalation rate), since passing offline does not mean passing in production.
  6. Expected follow-up: is 20 enough given the labelling cost? Not for statistical significance, but enough for regression — its job is to stop retrieval silently getting worse. Scale up before you settle an A/B, and grow it from failure cases rather than random additions.

  分析过程 · 先想清楚再作答

  1. 这题是国内面试的极高频题，也是最容易暴露「只搭过没调过」的一题。判据很简单：你的回答里有没有出现**具体的指标名和标注粒度**，没有就是没做过。
  2. 先把评估对象分清楚——这是最容易混的一步：**检索评估问「找得到找不到」，生成评估问「答得对不对」**。两套评估集要分开维护。混成一套的后果是分数掉了你分不清是检索漏了还是模型答砸了，而这两件事的修法完全不同。
  3. 评估集的形状：20 条左右的 query，每条**人工标注 1 到 3 个必须召回的 chunkId**。注意标注粒度是**块**不是文档——检索的单位就是块，标到文档级会让指标虚高。query 要覆盖真实分布，尤其要包含那些你知道会翻车的类型（编号、同义改写、跨文档）。
  4. 三个指标各回答一个问题：recall@5 是「进上下文的那几条覆盖了多少」，也就是你真正关心的数；recall@20 是天花板，它上不去说明问题在召回侧、重排再强也没用；MRR 对排序质量敏感，recall 打平时用它分高下。
  5. 生产视角：评估集一旦定下来就要冻结，换了样本分数就没有可比性——这和产线质检必须用固定的标准样品是同一个道理。同时线上要有对照指标（引用为空率、幻觉引用率、转人工率），因为离线过了不等于线上没事。
  6. 可预期的追问：标注成本这么高，20 条够吗？答：20 条不够做统计显著性，但足够做**回归**——它的作用是「改了检索之后别悄悄变差」。要做 A/B 定论再上规模，而且优先扩充失败案例，不是随机加样本。

  Key points

  • Retrieval and generation evaluation are two separate sets: 'was it found' versus 'was the answer right'.
  • Around 20 queries, each labelled with 1-3 chunk ids that must be retrieved — chunk level, not document level.
  • recall@5 is what the model actually sees, recall@20 is the ceiling, MRR measures ordering quality.
  • Freeze the set once agreed or scores stop being comparable; pair it with online empty-citation and hallucinated-citation rates.
  • Twenty cases is a regression guard, not a significance test; grow it from failure cases, not random samples.

  答题要点

  • 检索评估和生成评估是两套：前者问「找得到找不到」，后者问「答得对不对」，分开维护。
  • 评估集是 20 条左右的 query，每条人工标 1 到 3 个必须召回的 chunkId——标到块级，不是文档级。
  • recall@5 是真正关心的数（模型只看得到这几条），recall@20 是天花板，MRR 衡量排序质量。
  • 评估集一旦定下来就冻结，否则分数没有可比性；线上再配引用为空率、幻觉引用率做对照。
  • 20 条不够做显著性但够做回归；扩充时优先补失败案例，不是随机加样本。