Interview Bank

RAG in 14 Days: From Retrieval to Trustworthy Answers

D1 Why Retrieve at All: Hallucination, Knowledge Cutoffs, and the Cost of Long Context; a Minimal Keyword-Only RAG

• In BM25, what problems do term-frequency saturation and document length normalisation each solve? What happens if you set both k1 and b to zero?BM25 里的词频饱和与文档长度归一化分别在解决什么问题？把 k1 和 b 都设成 0 会发生什么？

  Common in ChinaCommon overseasIntermediate#bm25#ranking#information-retrieval

  How to reason about it · think before answering

  1. This checks whether you have actually read the formula rather than merely called a library. The test is whether you can map k1 and b onto specific terms and name the failure each one prevents.
  2. Start with the two holes in raw term frequency: keyword stuffing lets one document dominate by repeating a word, and long documents win by accident because they contain more words overall.
  3. k1 closes the first hole. Term frequency appears in both numerator and denominator, so the ratio approaches a ceiling instead of growing linearly. Fifty mentions are more relevant than five, but not ten times more relevant. A smaller k1 saturates sooner.
  4. b closes the second. The normalisation factor is one minus b plus b times document length over average length: at b equal to zero length is ignored entirely, at one it is fully penalised, and 0.75 is the conventional compromise.
  5. Now the trap in the question: k1 equal to zero collapses the ratio to a constant, so one occurrence scores the same as a hundred and matching becomes boolean. b equal to zero removes length entirely. Set both to zero and BM25 degenerates into a plain sum of inverse document frequencies.
  6. Expected follow-up: can you drop the IDF term? No. Without it, ubiquitous words drown everything else, and it is precisely IDF that lets BM25 work without a stopword list.

  分析过程 · 先想清楚再作答

  1. 这题考的是你有没有真的读过公式，而不是有没有调过库。判据很明确：能不能把 k1 和 b 各自对应到公式里的哪一项，并说出去掉之后会被什么样的文档钻空子。
  2. 先说朴素词频的两个漏洞：一是重复刷词，一篇文章把关键词写五十遍就能霸榜；二是长文占便宜，文档越长越容易蒙中查询里的词。这两个漏洞正好对应两个修正。
  3. k1 管第一个漏洞。分子分母里都有词频 f，所以词频涨上去之后整个分式趋近一个上界而不是线性增长——写五十遍确实比写五遍相关，但绝不该相关十倍。k1 越小饱和越快。
  4. b 管第二个漏洞。归一化项是 1 减 b 加上 b 乘以本文长度除以平均长度，b 等于 0 时完全不看长度，b 等于 1 时完全按长度比例惩罚，0.75 是长期折中的默认值。
  5. 回到题干那个陷阱：k1 设成 0 会让分式退化成常数，词出现一次和一百次得分完全一样，等于只剩「有没有出现过」的布尔匹配；b 设成 0 则长度信息彻底消失。两个一起设成 0，BM25 就退化成对逆文档频率求和，跟词频再无关系。
  6. 可预期的追问：那逆文档频率去掉行不行？答案是不行，去掉之后「的」「我们」这类高频词会淹没一切——而且要顺带说明 BM25 因此天然不需要停用词表，这一句最能体现你读懂了公式。

  Key points

  • k1 controls saturation and prevents keyword stuffing: the score approaches a ceiling rather than growing linearly with frequency.
  • b controls length normalisation and stops long documents from winning by sheer word count.
  • Setting k1 to zero degenerates the scorer into boolean matching; one occurrence scores the same as a hundred.
  • Setting b to zero removes document length from the equation entirely; both at zero leaves only a sum of IDF terms.
  • IDF is the third component: it up-weights rare terms and removes the need for a stopword list.

  答题要点

  • 词频饱和由 k1 控制，防的是重复刷词：词频涨大后得分趋近上界而非线性增长。
  • 长度归一化由 b 控制，防的是长文档靠词多蒙中查询，用本文长度比平均长度把它压回去。
  • k1 设 0 会退化成布尔匹配，词出现一次和一百次同分；b 设 0 则完全不考虑文档长度。
  • 两者都设 0 时 BM25 只剩逆文档频率求和，等于放弃了词频信息。
  • 逆文档频率是第三块，让稀有词权重更高，也让 BM25 天然不需要停用词表。

D8 Evaluation First: Building a Golden Set, Computing Recall and Ranking Metrics, Using a Model as Judge for Faithfulness

• Recall, mean reciprocal rank, and normalized discounted cumulative gain - which failure mode does each one catch first, and what do you miss by watching only one?召回率、平均倒数排名、归一化折损累计增益，这三个检索指标分别在什么故障下会先掉下来？只盯一个会漏掉什么？

  Common in ChinaCommon overseasIntermediate#retrieval-metrics#evaluation#ranking

  How to reason about it · think before answering

  1. This tests whether you know each metric's blind spot, not whether you can recite definitions. Layer them as 'did it show up / how high / how good overall' and you are halfway there.
  2. Recall is boolean: is the answer document in the final context. It catches 'never retrieved', but it does not move when the answer slips from rank 1 to rank 8, as long as it still fits the budget.
  3. MRR looks only at the rank of the first relevant hit, so ranking degradation shows up immediately. Its blind spot: one relevant item in the top ten scores exactly the same as five.
  4. nDCG discounts every relevant hit in the top k by its position, so it tracks overall ranking quality and is the direct optimization target for reranking. Its blind spot is existence - it is zero both when nothing was retrieved and when ranking is terrible.
  5. Conclusion: together they localize the failure. Recall drops means retrieval or chunking; recall flat but MRR down means ranking degraded, reach for a reranker; both stable but nDCG down means more noise crept into the top results.
  6. Expected follow-up: what if a metric saturates? Make the questions harder - a saturated metric means the eval set lost its discriminative power, and further tuning is blind.

  分析过程 · 先想清楚再作答

  1. 这题考的是「知不知道指标之间的盲区」，不是背定义。能把三者按「有没有 / 靠不靠前 / 整体好不好」分层的，基本就答对了一半。
  2. 推导链是这样的：召回率是布尔的——答案文档在不在最终上下文里。它对「压根没捞到」最敏感，但答案从第 1 名掉到第 8 名它一动不动，只要还在预算内。
  3. 倒数排名只看第一条相关结果的名次，所以「答案还在但被挤到后面」它立刻掉。反过来它有个盲区：前十条里有一条命中还是五条命中，它给的分完全一样。
  4. 归一化折损累计增益把前 k 名里每一条相关结果都按名次折算再累加，所以它对「整体排序质量」敏感，是重排最直接的优化目标。它的盲区是不告诉你「有没有」——召回率为零时它也是零，看不出是没捞到还是排得差。
  5. 结论：三个一起看才能定位故障层。召回率掉说明检索或切块出了问题，要动召回策略；召回率不动而倒数排名掉，说明排序退化，该上重排；两者都稳而 nDCG 掉，说明前几名里混进了更多噪声。
  6. 可预期的追问是「指标顶格了怎么办」。真实答案是把题目做难：指标撞天花板说明评估集失去区分度，这时候继续优化系统是在瞎调。

  Key points

  • Recall answers 'did it make it into the context', sensitive to total misses, blind to rank shifts.
  • MRR answers 'how high is the first hit', sensitive to ranking degradation, blind to how many hits there are.
  • nDCG answers 'how good is the top k overall', the direct target for reranking, blind to existence.
  • Only the combination localizes the failure to retrieval, ranking, or noise.
  • State the hit criterion: context is packed against a token budget, not a fixed top-k.

  答题要点

  • 召回率管「有没有进上下文」，对完全没捞到最敏感，对名次变化不敏感。
  • 平均倒数排名管「第一条排第几」，对排序退化最敏感，但分不清命中一条还是五条。
  • 归一化折损累计增益管「前 k 名整体质量」，是重排的直接优化目标，但看不出有没有。
  • 三者组合才能定位故障在召回层、排序层还是噪声层。
  • 命中口径要说清：按 token 预算装上下文，不是按固定条数取前 k。