面试题库

14 天 RAG：从检索到可信回答

D6 生成这一侧：上下文怎么排、引用怎么标、什么时候必须拒答，以及流式回答

• 知识库问答的拒答阈值怎么定？定高了和定低了各自的代价是什么？How do you set the refusal threshold for a knowledge-base assistant, and what does it cost you when the threshold is too high or too low?

  国内高频海外高频深入#refusal#thresholds#evaluation

  分析过程 · 先想清楚再作答

  1. 这题真正在考的是：你有没有意识到拒答不是一个阈值，而是好几条判据；以及你定阈值靠不靠数据。只谈一个分数阈值的回答，说明只做过最浅的一层。
  2. 先把拒答拆成三条线，它们的触发时机完全不同。检索分数太低：生成之前就能判，省一次模型调用。材料互相矛盾：也在生成之前判，代码在块之间找同一件事的不同数字，检出后要么并列两种说法与各自的更新日期，要么在有权威信号（比如一份点破了这条不一致的会议纪要）时按更新日期择一——选哪条是产品决策，但无论如何不能让模型自己悄悄挑一个。问题超出材料覆盖范围：只能在生成之后判，判据是跑完引用校验一条有效引用都没有。
  3. 强调三种话术必须不同。第一种要说「库里没有相关材料，换个说法或确认资料是否入库」，第三种要说「找到了相关文档但里面没有能直接回答的内容」——用户的下一步动作完全不同，混成一句「抱歉我不知道」等于把信息扔了。
  4. 再答代价这一半。定高了：能答的问题被挡在门外，用户看到查不到而材料其实在库里，这是最伤信任的一种错，而且它在日志里几乎不可见。定低了：低分噪声材料进上下文，模型拿着不相关的东西硬答，错误反而更隐蔽，因为回答看起来还带着引用。
  5. 怎么定：拿一批已知有答案和已知没答案的问题跑一遍，看两组的分数分布在哪里分开，按你更怕哪种错来取点。分数是没有绝对量纲的，换语料、换检索方式都要重定，所以真正要交付的是这套定阈值的流程，不是那个数字。
  6. 可预期的追问：单一分数阈值不够怎么办？答案是加判据而不是调数字——最高分与次高分的差、命中块数、以及生成后的引用校验结果，都是比原始分数更稳的信号。

  How to reason about it · think before answering

  1. What is really being tested: do you know that refusal is several rules rather than one threshold, and do you set thresholds from data. An answer that mentions only a score cutoff shows you have only touched the surface.
  2. Break refusal into three rules with different timing. Score too low: decidable before generation, saving a model call. Sources conflict: also decidable before generation, by finding differing numbers about the same thing across blocks. You then either present both with their update dates, or pick the newer one when an authoritative signal backs it, such as meeting notes that flagged the discrepancy. Which of the two is a product decision, but silently letting the model pick is never an option. Question outside coverage: only decidable after generation, when citation verification leaves you with zero verified citations.
  3. Stress that the three responses must read differently. 'Nothing relevant in the knowledge base, try rephrasing or check whether the document was ingested' is a different instruction to the user than 'we found related documents but none of them answers this'. Collapsing both into 'sorry, I don't know' throws away information.
  4. Then the cost half. Too high: answerable questions get blocked, the user is told nothing was found while the material is in fact indexed. That is the most trust-damaging failure and it is nearly invisible in logs. Too low: weak passages enter the context and the model answers from irrelevant material, which is worse because the answer still looks cited.
  5. How to set it: run a set of questions with known answers and known non-answers, look at where the two score distributions separate, and pick a point according to which error you fear more. Scores have no absolute scale, so the deliverable is the procedure, not the number.
  6. Expected follow-up: what if one score threshold is not enough? Add signals rather than tuning the number: the gap between top and second score, the number of hits above threshold, and the post-generation verification result are all steadier than the raw score.

  答题要点

  • 拒答不是一条线而是三条：分数过低、材料冲突（都在生成前判）、超出材料覆盖范围（只能生成后按引用校验结果判）。
  • 冲突检出后并列两说还是按更新日期择一，是产品决策；只有在有权威信号背书时择一才站得住，否则老实并列。
  • 三种情况的话术必须不同，因为它们给用户的下一步动作不同。
  • 定高了会把能答的问题挡住，用户看到查不到而材料其实在库里，最伤信任且日志里看不见。
  • 定低了会让噪声材料进上下文，错误更隐蔽，因为回答看起来仍然带着引用。
  • 定法是拿已知有答案与已知没答案的两组问题跑分数分布，按更怕哪种错取点；换语料或换检索方式都要重定。

  Key points

  • Refusal is three rules, not one: low score and source conflict decided before generation, out-of-coverage decided after generation from the verification result.
  • On conflict, presenting both versions versus picking the newer one is a product decision; picking only holds up when an authoritative signal backs it.
  • The three responses must be worded differently because each implies a different next action for the user.
  • Too high blocks answerable questions; the user is told nothing exists while it does, which is the most damaging and least visible failure.
  • Too low lets weak passages in, producing errors that are harder to spot because the answer still carries citations.
  • Set it by comparing score distributions over answerable and unanswerable question sets, then choose based on which error is worse; re-tune whenever the corpus or retriever changes.