面试题库

14 天 RAG：从检索到可信回答

D6 生成这一侧：上下文怎么排、引用怎么标、什么时候必须拒答，以及流式回答

• 知识库问答的拒答阈值怎么定？定高了和定低了各自的代价是什么？How do you set the refusal threshold for a knowledge-base assistant, and what does it cost you when the threshold is too high or too low?

  国内高频海外高频深入#refusal#thresholds#evaluation

  分析过程 · 先想清楚再作答

  1. 这题真正在考的是：你有没有意识到拒答不是一个阈值，而是好几条判据；以及你定阈值靠不靠数据。只谈一个分数阈值的回答，说明只做过最浅的一层。
  2. 先把拒答拆成三条线，它们的触发时机完全不同。检索分数太低：生成之前就能判，省一次模型调用。材料互相矛盾：也在生成之前判，代码在块之间找同一件事的不同数字，检出后要么并列两种说法与各自的更新日期，要么在有权威信号（比如一份点破了这条不一致的会议纪要）时按更新日期择一——选哪条是产品决策，但无论如何不能让模型自己悄悄挑一个。问题超出材料覆盖范围：只能在生成之后判，判据是跑完引用校验一条有效引用都没有。
  3. 强调三种话术必须不同。第一种要说「库里没有相关材料，换个说法或确认资料是否入库」，第三种要说「找到了相关文档但里面没有能直接回答的内容」——用户的下一步动作完全不同，混成一句「抱歉我不知道」等于把信息扔了。
  4. 再答代价这一半。定高了：能答的问题被挡在门外，用户看到查不到而材料其实在库里，这是最伤信任的一种错，而且它在日志里几乎不可见。定低了：低分噪声材料进上下文，模型拿着不相关的东西硬答，错误反而更隐蔽，因为回答看起来还带着引用。
  5. 怎么定：拿一批已知有答案和已知没答案的问题跑一遍，看两组的分数分布在哪里分开，按你更怕哪种错来取点。分数是没有绝对量纲的，换语料、换检索方式都要重定，所以真正要交付的是这套定阈值的流程，不是那个数字。
  6. 可预期的追问：单一分数阈值不够怎么办？答案是加判据而不是调数字——最高分与次高分的差、命中块数、以及生成后的引用校验结果，都是比原始分数更稳的信号。

  How to reason about it · think before answering

  1. What is really being tested: do you know that refusal is several rules rather than one threshold, and do you set thresholds from data. An answer that mentions only a score cutoff shows you have only touched the surface.
  2. Break refusal into three rules with different timing. Score too low: decidable before generation, saving a model call. Sources conflict: also decidable before generation, by finding differing numbers about the same thing across blocks. You then either present both with their update dates, or pick the newer one when an authoritative signal backs it, such as meeting notes that flagged the discrepancy. Which of the two is a product decision, but silently letting the model pick is never an option. Question outside coverage: only decidable after generation, when citation verification leaves you with zero verified citations.
  3. Stress that the three responses must read differently. 'Nothing relevant in the knowledge base, try rephrasing or check whether the document was ingested' is a different instruction to the user than 'we found related documents but none of them answers this'. Collapsing both into 'sorry, I don't know' throws away information.
  4. Then the cost half. Too high: answerable questions get blocked, the user is told nothing was found while the material is in fact indexed. That is the most trust-damaging failure and it is nearly invisible in logs. Too low: weak passages enter the context and the model answers from irrelevant material, which is worse because the answer still looks cited.
  5. How to set it: run a set of questions with known answers and known non-answers, look at where the two score distributions separate, and pick a point according to which error you fear more. Scores have no absolute scale, so the deliverable is the procedure, not the number.
  6. Expected follow-up: what if one score threshold is not enough? Add signals rather than tuning the number: the gap between top and second score, the number of hits above threshold, and the post-generation verification result are all steadier than the raw score.

  答题要点

  • 拒答不是一条线而是三条：分数过低、材料冲突（都在生成前判）、超出材料覆盖范围（只能生成后按引用校验结果判）。
  • 冲突检出后并列两说还是按更新日期择一，是产品决策；只有在有权威信号背书时择一才站得住，否则老实并列。
  • 三种情况的话术必须不同，因为它们给用户的下一步动作不同。
  • 定高了会把能答的问题挡住，用户看到查不到而材料其实在库里，最伤信任且日志里看不见。
  • 定低了会让噪声材料进上下文，错误更隐蔽，因为回答看起来仍然带着引用。
  • 定法是拿已知有答案与已知没答案的两组问题跑分数分布，按更怕哪种错取点；换语料或换检索方式都要重定。

  Key points

  • Refusal is three rules, not one: low score and source conflict decided before generation, out-of-coverage decided after generation from the verification result.
  • On conflict, presenting both versions versus picking the newer one is a product decision; picking only holds up when an authoritative signal backs it.
  • The three responses must be worded differently because each implies a different next action for the user.
  • Too high blocks answerable questions; the user is told nothing exists while it does, which is the most damaging and least visible failure.
  • Too low lets weak passages in, producing errors that are harder to spot because the answer still carries citations.
  • Set it by comparing score distributions over answerable and unanswerable question sets, then choose based on which error is worse; re-tune whenever the corpus or retriever changes.

D7 第一周综合：把六天的零件装成一个可一键启动的检索问答服务并复盘

• 一个检索问答服务上线前你会做哪三项检查？为什么偏偏是这三项？What three checks would you run before shipping a retrieval QA service, and why those three?

  国内高频海外高频进阶#production-readiness#citations#refusal

  分析过程 · 先想清楚再作答

  1. 这题的区分度不在你能列几项，而在你能不能说清「为什么是这三项」。列十项而每项都不给理由，反而说明你没有排过优先级。
  2. 推导方式是按后果排序：哪种故障用户看不出来、又损失最大，哪一项就该排在前面。
  3. 第一项是引用可查证：每条引用的编号都能回查到真实存在的块，且那一块确实与该句有实质重合。这一项排第一是因为引用错了用户根本发现不了，而它恰恰是这类系统唯一的信任来源。
  4. 第二项是该拒答时真的拒答：构造一个语料里没有答案的问题，看它是回那句拒答话术还是开始编。这一项也属于用户看不出来的故障，且一旦编造被发现，整个系统的可信度归零。
  5. 第三项是摄取到检索的一致性：摄取完之后新文档立刻能被检索到，且关键词与向量两路的覆盖数量对得上。这一项防的是「一路能查一路查不到」这种最难排查的故障。
  6. 可预期的追问：为什么延迟和成本不在前三？因为它们是**看得见**的故障——慢了用户会抱怨，贵了账单会告诉你；而上面三项不检查就永远不会有人告诉你。

  How to reason about it · think before answering

  1. The discriminator is not how many checks you list but whether you can justify the three. Ten items with no ranking suggests you have never had to prioritise.
  2. Derive them by consequence: the failures that are invisible to users and most damaging go first.
  3. First, citations must be verifiable: every cited id resolves to a real chunk, and that chunk genuinely overlaps the sentence citing it. This ranks first because a wrong citation is undetectable by the user, and citations are the only source of trust this system has.
  4. Second, refusal must actually fire: ask a question the corpus cannot answer and confirm the system says so instead of inventing. Also invisible, and one discovered fabrication zeroes out trust in the whole product.
  5. Third, ingestion-to-retrieval consistency: freshly ingested documents are retrievable immediately, and the keyword and vector paths cover the same set. This guards against the 'one route finds it, the other does not' failure, which is the hardest to diagnose.
  6. Expected follow-up: why not latency and cost? Because those failures are visible. Users complain about slowness and the bill reports overspending; nobody will ever report the three above.

  答题要点

  • 先给排序依据：优先检查用户发现不了、但后果最重的故障。
  • 第一项引用可查证：编号能回查到真实的块，且该块与被引的那句话有实质重合。
  • 第二项拒答生效：用一个语料里没有答案的问题验证系统会说查不到，而不是开始编。
  • 第三项摄取与检索一致：新入库的文档立刻可检索，关键词与向量两路覆盖对得上。
  • 延迟和成本重要但排在后面，因为它们是看得见的故障，会自己找上门。

  Key points

  • State the ranking rule first: prioritise failures users cannot see but that cost the most.
  • Check one, verifiable citations: every id resolves to a real chunk that overlaps the sentence citing it.
  • Check two, refusal actually fires on a question the corpus cannot answer.
  • Check three, ingestion and retrieval agree: new documents are immediately retrievable on both routes.
  • Latency and cost matter but rank lower because those failures announce themselves.