Interview Bank

RAG in 14 Days: From Retrieval to Trustworthy Answers

D8 Evaluation First: Building a Golden Set, Computing Recall and Ranking Metrics, Using a Model as Judge for Faithfulness

• Recall, mean reciprocal rank, and normalized discounted cumulative gain - which failure mode does each one catch first, and what do you miss by watching only one?召回率、平均倒数排名、归一化折损累计增益，这三个检索指标分别在什么故障下会先掉下来？只盯一个会漏掉什么？

  Common in ChinaCommon overseasIntermediate#retrieval-metrics#evaluation#ranking

  How to reason about it · think before answering

  1. This tests whether you know each metric's blind spot, not whether you can recite definitions. Layer them as 'did it show up / how high / how good overall' and you are halfway there.
  2. Recall is boolean: is the answer document in the final context. It catches 'never retrieved', but it does not move when the answer slips from rank 1 to rank 8, as long as it still fits the budget.
  3. MRR looks only at the rank of the first relevant hit, so ranking degradation shows up immediately. Its blind spot: one relevant item in the top ten scores exactly the same as five.
  4. nDCG discounts every relevant hit in the top k by its position, so it tracks overall ranking quality and is the direct optimization target for reranking. Its blind spot is existence - it is zero both when nothing was retrieved and when ranking is terrible.
  5. Conclusion: together they localize the failure. Recall drops means retrieval or chunking; recall flat but MRR down means ranking degraded, reach for a reranker; both stable but nDCG down means more noise crept into the top results.
  6. Expected follow-up: what if a metric saturates? Make the questions harder - a saturated metric means the eval set lost its discriminative power, and further tuning is blind.

  分析过程 · 先想清楚再作答

  1. 这题考的是「知不知道指标之间的盲区」，不是背定义。能把三者按「有没有 / 靠不靠前 / 整体好不好」分层的，基本就答对了一半。
  2. 推导链是这样的：召回率是布尔的——答案文档在不在最终上下文里。它对「压根没捞到」最敏感，但答案从第 1 名掉到第 8 名它一动不动，只要还在预算内。
  3. 倒数排名只看第一条相关结果的名次，所以「答案还在但被挤到后面」它立刻掉。反过来它有个盲区：前十条里有一条命中还是五条命中，它给的分完全一样。
  4. 归一化折损累计增益把前 k 名里每一条相关结果都按名次折算再累加，所以它对「整体排序质量」敏感，是重排最直接的优化目标。它的盲区是不告诉你「有没有」——召回率为零时它也是零，看不出是没捞到还是排得差。
  5. 结论：三个一起看才能定位故障层。召回率掉说明检索或切块出了问题，要动召回策略；召回率不动而倒数排名掉，说明排序退化，该上重排；两者都稳而 nDCG 掉，说明前几名里混进了更多噪声。
  6. 可预期的追问是「指标顶格了怎么办」。真实答案是把题目做难：指标撞天花板说明评估集失去区分度，这时候继续优化系统是在瞎调。

  Key points

  • Recall answers 'did it make it into the context', sensitive to total misses, blind to rank shifts.
  • MRR answers 'how high is the first hit', sensitive to ranking degradation, blind to how many hits there are.
  • nDCG answers 'how good is the top k overall', the direct target for reranking, blind to existence.
  • Only the combination localizes the failure to retrieval, ranking, or noise.
  • State the hit criterion: context is packed against a token budget, not a fixed top-k.

  答题要点

  • 召回率管「有没有进上下文」，对完全没捞到最敏感，对名次变化不敏感。
  • 平均倒数排名管「第一条排第几」，对排序退化最敏感，但分不清命中一条还是五条。
  • 归一化折损累计增益管「前 k 名整体质量」，是重排的直接优化目标，但看不出有没有。
  • 三者组合才能定位故障在召回层、排序层还是噪声层。
  • 命中口径要说清：按 token 预算装上下文，不是按固定条数取前 k。