面试题库

14 天 RAG：从检索到可信回答

D1 为什么要检索：幻觉、知识截止与长上下文的代价，以及一个纯关键词的最小 RAG

• 一个检索增强生成系统答错了，你怎么定位是检索的锅还是生成的锅？A retrieval-augmented generation system gave a wrong answer. How do you determine whether retrieval or generation is at fault?

  国内高频海外高频进阶#debugging#failure-modes#evaluation

  分析过程 · 先想清楚再作答

  1. 题眼在「怎么定位」，不在「有哪些原因」。答成一串可能原因的罗列就输了，面试官想听的是一个有先后顺序、能落到具体动作的排查流程。
  2. 先给最省时间的第一步：把这次检索出来的几段原文原样打印出来，自己读一遍。正确答案不在里面就是检索的锅，在里面而模型没用上才是生成的锅。这一步三十秒，能省掉大半天的瞎猜。
  3. 然后把链路展开成五个环节——切块、建索引、检索、组装上下文、生成——并给出「排查从右往左、修复从左往右」这条口径：从右往左是因为你最先看到的是生成结果，从左往右是因为左边的错会被右边放大。
  4. 补充几个能把环节钉死的症状：答案「半对」多半是切块把一条完整规则切断了；检索结果里混着一眼不相干的东西多半是解析没做干净；模型无视材料用先验知识作答，通常是提示词里少了「只能依据资料回答」；引用编号和内容对不上，那是生成侧漏读或串了行。
  5. 最后落到工程做法：这套排查要能重复做，就必须把每次请求的检索结果、进上下文的段落、最终回答一起记下来，否则线上出问题时你根本复现不了。到了要批量做的时候，就得换成一批固定问题加指标，而不是一条条人工看。
  6. 可预期的追问：如果检索确实没捞到，改提示词有没有用？答案是没用——材料里没有的东西，再好的指令也只能换一种编法。这句话最能证明你分清了两层。

  How to reason about it · think before answering

  1. The question asks how you localise the fault, not what the possible causes are. Listing causes loses; the interviewer wants an ordered procedure that ends in concrete actions.
  2. Give the cheapest first step: print the retrieved passages verbatim and read them. If the correct answer is not in there, retrieval is at fault. If it is in there and the model ignored it, generation is at fault. Thirty seconds, and it removes most of the guesswork.
  3. Then lay out the five stages — chunking, indexing, retrieval, context assembly, generation — with the rule: diagnose right to left, fix left to right. You see the generated answer first, but an error on the left is amplified by everything to its right.
  4. Add symptoms that pin down a stage: half-correct answers usually mean a rule was split across chunks; obviously irrelevant hits usually mean dirty parsing; the model ignoring the supplied material usually means the prompt never said it must; citation numbers that do not match their content point at generation.
  5. Land it in engineering terms: to run this procedure repeatedly you must log the retrieved hits, the passages that entered the context, and the final answer together, otherwise production issues are unreproducible. At scale this becomes a fixed question set with metrics rather than case-by-case reading.
  6. Expected follow-up: if retrieval missed the document, will prompt tuning help? No. Nothing in the prompt can conjure material that was never supplied.

  答题要点

  • 第一步永远是把检索出来的原文打印出来读一遍，判断正确答案在不在里面。
  • 把链路拆成切块、建索引、检索、组装上下文、生成五个环节，排查从右往左、修复从左往右。
  • 用症状钉环节：半对多半是切块问题，混入无关结果多半是解析问题，无视材料多半是提示词缺约束，引用与内容对不上是生成问题。
  • 检索没捞到时改提示词没有意义，材料里没有的东西模型只能编。
  • 要能重复排查就必须把检索结果、进上下文的段落和最终回答一起记录下来。

  Key points

  • Always start by printing the retrieved passages and checking whether the correct answer is present at all.
  • Split the pipeline into chunking, indexing, retrieval, context assembly and generation; diagnose right to left, fix left to right.
  • Use symptoms to pin the stage: half-correct answers point at chunking, irrelevant hits at parsing, ignored material at the prompt, mismatched citations at generation.
  • If retrieval missed the document, prompt changes cannot help; the material simply is not there.
  • Log retrieved hits, the passages that entered the context, and the final answer together, or production failures are unreproducible.

D2 embedding 与向量检索：相似度、维度与模型选型，把文本存进 pgvector

• 为什么有些 embedding 模型要求查询和文档加不同的前缀？不加会怎样，你怎么在上线前发现这个问题？Why do some embedding models require different prefixes for queries and documents? What happens if you skip them, and how would you catch it before shipping?

  国内高频海外高频进阶#embeddings#model-selection#evaluation

  分析过程 · 先想清楚再作答

  1. 这题的题眼是「静默失效」。会背「e5 要加 query 和 passage 前缀」只能拿基础分，能说清它为什么不报错、以及怎么在上线前抓住它，才是做过的人。
  2. 先讲原因：这一族模型是拿成对数据训练的，一侧是短问句、一侧是长段落，两者的分布本来就不一样。前缀是训练时给模型的角色标记，告诉它这一段该按查询编码还是按文档编码。推理时不给，模型就落在了训练分布之外。
  3. 再讲后果的性质：不加前缀模型照样输出向量、照样能算距离、名次照样有先后，只是整体质量下滑。**没有任何报错**——这跟忘了归一化是同一类问题：错误不会自己浮出来。
  4. 怎么发现：唯一可靠的办法是一小份标注问题集，用同一批文档跑两遍（加前缀与不加前缀），比命中率。这就是第 8 天要做的评估闸门，它的价值恰恰在于抓这类静默错误。上线前跑一遍，比读十遍文档管用。
  5. 补一个更容易踩的变体：**建库时加了前缀、查询时忘了加**，或者两边加成同一个前缀。这种情况下所有向量都在同一个坐标系里，看起来更「正常」，但查询与文档的对齐关系是错的，掉分同样查不出来。所以前缀应该封装在 embed 的调用约定里，而不是散在各处手拼。
  6. 可预期的追问：OpenAI 的模型要不要加前缀？不需要——它不属于这一族。所以这不是一条普遍规则，而是**每换一个模型都要重新读模型卡片确认**的事。

  How to reason about it · think before answering

  1. The core of this question is silent failure. Reciting 'e5 needs query: and passage: prefixes' is the baseline; explaining why nothing errors out and how you would catch it is what shows experience.
  2. The reason: these models are trained on pairs, short questions on one side and longer passages on the other, two genuinely different distributions. The prefix is a role marker learned during training. Omit it at inference and you are off-distribution.
  3. The consequence: the model still returns vectors, distances still compute, results still have an order, quality just degrades. Nothing throws, exactly like forgetting to normalise.
  4. How to catch it: run a small labelled question set against the same corpus twice, with and without prefixes, and compare hit rate. That is the evaluation gate built on day 8, and catching silent regressions is precisely what it is for.
  5. Mention the sneakier variant: prefixing at index time but not at query time, or using the same prefix on both sides. Everything sits in one coordinate space and looks healthier, yet the query-document alignment is wrong and the loss is just as invisible. Encapsulate prefixes in the embedding call convention rather than hand-writing them everywhere.
  6. Expected follow-up: do OpenAI models need prefixes? No, they are not in that family, so this is not a universal rule but a per-model detail you re-check on the model card every time you switch.

  答题要点

  • 这类模型用问句与段落的成对数据训练，前缀是区分两种角色的标记，缺了就落在训练分布之外。
  • 不加前缀不会报错，只会整体掉分，属于静默失效。
  • 唯一可靠的发现方式是拿一份标注问题集跑 A/B 对比命中率。
  • 更隐蔽的错法是两边前缀不一致或用了同一个前缀，看起来更正常但对齐是错的。
  • 前缀应封装在 embed 的调用约定里；换模型必须重读模型卡片，它不是普遍规则。

  Key points

  • These models are trained on question-passage pairs; the prefix marks which role a text plays, and omitting it puts you off-distribution.
  • Skipping prefixes never errors, it only degrades quality, so the failure is silent.
  • The reliable detection is an A/B run over a small labelled question set, comparing hit rate.
  • A subtler bug is mismatched or identical prefixes on both sides, which looks healthier but misaligns queries and documents.
  • Keep prefixes inside the embedding call convention, and re-read the model card whenever you switch models.

D4 切块策略：固定、递归、按结构、父子与语义五种切法，以及用评估而不是直觉来选

• 你怎么决定切块大小？说出你会看的两个指标和一个反例。How do you decide on chunk size? Name two metrics you would look at, and one counterexample.

  国内高频海外高频进阶#chunking#evaluation

  分析过程 · 先想清楚再作答

  1. 这题的题眼是「怎么决定」，不是「多大合适」。答一个具体数字（512 token、1000 字符）就已经输了——面试官想看的是你有没有一套定法，而不是你记得住哪个默认值。
  2. 先把矛盾摆出来：块大则信噪比低、上下文贵，块小则单块缺语境、模型答不出所以然。切块大小就是在这两头之间找位置，所以两个指标必须分别对应这两头。
  3. 第一个指标是检索侧的命中率——答案文档有没有进上下文。第二个是生成侧的可用性，最省事的代理指标是切碎率，也就是有多少块结尾停在半句话上；再往前一步就是忠实度和引用是否可定位。
  4. 关键补一句：两个指标必须在**同一个 token 预算**下比，不能按「取前 k 块」比。k 固定时块越大塞进去的字越多，大块切法会赢在买得多而不是切得准上。这一句往往是这道题的区分点。
  5. 反例要具体。最好用的一个是：把块从 400 字调到 1200 字，命中率不降反升——但那是因为一整篇短文档被当成一块塞了进去，检索其实什么都没做，等于退化成了全文投喂。指标涨了，系统更差了。
  6. 可预期的追问是「那你第一次上手时从哪个数字起步」。答：先按文档类型选切法（有标题层级就按结构切），块长从 300 到 500 字起步、重叠取一到两成，然后立刻建一组标准问题跑评估，用两三轮迭代把它调到位。起步值是起步值，不是结论。

  How to reason about it · think before answering

  1. The question is about method, not about a number. Answering with a specific default (512 tokens, 1000 characters) already loses it — the interviewer wants to hear that you have a procedure.
  2. State the tension first: large chunks dilute the signal and cost context; small chunks lose the surrounding meaning so the model cannot use them. The two metrics you name should map onto those two failure modes.
  3. Metric one is retrieval-side hit rate: did a document that actually answers the question make it into the context. Metric two is generation-side usability, cheaply proxied by the fraction of chunks that end mid-sentence, and more seriously by faithfulness and whether citations resolve.
  4. Add the point that separates candidates: both metrics must be compared under the same token budget, never under a fixed top-k. With fixed k, bigger chunks simply buy more text and win for the wrong reason.
  5. Make the counterexample concrete: raising chunk size from 400 to 1200 characters can lift hit rate purely because whole short documents now fit in one chunk, which means retrieval stopped doing anything and you are back to stuffing full documents. The metric improved while the system got worse.
  6. Expect the follow-up: where do you start on day one. Pick the strategy from the document type first (structural splitting whenever headings exist), start around 300 to 500 characters with 10 to 20 percent overlap, then build a golden set immediately and iterate. A starting point is not a conclusion.

  答题要点

  • 先按文档类型选切法，再调长度：有标题层级就按结构切，没有结构才谈固定长度或语义。
  • 看两个指标：检索侧的命中率，生成侧的切碎率（进一步是忠实度与引用可定位性）。
  • 两个指标必须在同一个 token 预算下比，不能按「取前 k 块」比，否则大块只是买得更多。
  • 反例：块调大后命中率上升，但那是因为整篇被当成一块，检索退化成全文投喂。
  • 起步值 300 到 500 字、重叠一到两成，然后靠一组固定问题迭代，不靠直觉定稿。

  Key points

  • Choose the strategy from the document type first, then tune length: split on headings whenever the structure survives parsing.
  • Watch two metrics: retrieval hit rate on one side, mid-sentence break rate (then faithfulness and citation resolvability) on the other.
  • Compare under an equal token budget, never a fixed top-k, or larger chunks win by buying more text.
  • Counterexample: hit rate rises after enlarging chunks because whole documents now fit in one chunk and retrieval has effectively stopped working.
  • Start near 300 to 500 characters with 10 to 20 percent overlap, then iterate against a fixed question set instead of guessing.
• 语义切分比递归切分贵不少，你怎么向团队证明这笔钱值得花？Semantic chunking costs considerably more than recursive splitting. How would you prove to your team that the money is well spent?

  国内高频海外高频深入#chunking#evaluation#cost

  分析过程 · 先想清楚再作答

  1. 这题表面问技术，实际考的是你会不会做一次带对照组的技术论证。上来就讲语义切分原理的人，答的是另一道题。
  2. 第一步是先承认它可能不值。语义切分的收益来自「文档没有可用的结构」；如果知识库是结构良好的文档，作者的标题层级已经免费替你做完了语义切分，这时候花的钱大概率打水漂。**先说清适用前提，再谈证明，这一步就把大多数候选人区分开了。**
  3. 第二步是把「值不值」翻译成可测的三笔账：指标涨了多少（同一批标准问题、同一个 token 预算下的命中率）、延迟涨了多少（切块是离线的，但更新链路的端到端时间会变）、钱涨了多少（首次全量 embedding 的费用，加上按更新频率折算的重算费用）。只报第一笔的论证不成立。
  4. 第三步是设计对照。递归切分是基线，语义切分是实验组，两组必须用同一份语料、同一批问题、同一个上下文预算、同一个检索器，只改切法这一个变量。改两个变量的实验，结论一文不值。
  5. 第四步是给决策一个门槛，而不是给一个感想。比如：命中率相对基线提升低于三个百分点就不上；提升超过五个百分点且重算成本在月度预算内就上；中间地带先在一类文档上灰度。**门槛要在跑数字之前定好**，否则你会不自觉地去迁就已经跑出来的结果。
  6. 可预期的追问是「有没有更便宜的办法拿到同样的收益」。答有：先试按结构切，它零成本且效果常常接近；结构确实不可用时，再考虑只对高价值的那一部分文档做语义切分，而不是全量上。

  How to reason about it · think before answering

  1. This looks like a technical question but it tests whether you can run a controlled technical argument. Launching into how semantic chunking works answers a different question.
  2. Step one is to concede that it may well not be worth it. The gain comes from documents that have no usable structure; if your knowledge base is well-formed documents, the authors' heading hierarchy already did the semantic split for free and the money is likely wasted.
  3. Step two is translating 'worth it' into three measurable numbers: how much the metric moved (hit rate on the same golden set under the same token budget), how much latency moved (chunking is offline, but the end-to-end update path changes), and how much it costs (the initial full embedding pass plus recomputation amortised over update frequency).
  4. Step three is the control. Recursive splitting is the baseline, semantic chunking the treatment, and they must share the corpus, the questions, the context budget and the retriever. Change one variable only; a two-variable experiment proves nothing.
  5. Step four is a decision threshold rather than an impression. For example: below three points of hit-rate gain, no; above five points with recomputation inside the monthly budget, yes; in between, roll it out on one document class first. Fix the threshold before you run the numbers, or you will quietly bend it to fit them.
  6. Expect the follow-up: is there a cheaper way to the same gain. Yes — try structural splitting first, since it is free and often nearly as good, and if the structure really is unusable, apply semantic chunking only to the high-value subset rather than the whole corpus.

  答题要点

  • 先讲适用前提：语义切分的收益来自文档没有可用结构，结构良好的文档上它大概率不值。
  • 把「值不值」翻译成三笔账：命中率涨多少、延迟涨多少、钱涨多少，只报第一笔不算论证。
  • 做对照实验：同语料、同问题集、同上下文预算、同检索器，只改切法一个变量。
  • 决策门槛必须在跑数字之前定好，避免事后迁就结果。
  • 先试零成本的按结构切；确需语义切分时也优先只覆盖高价值文档，而不是全量上。

  Key points

  • Start with the precondition: the gain comes from documents without usable structure, so on well-formed documents it usually is not worth it.
  • Translate 'worth it' into three numbers — hit rate, latency, and cost. Reporting only the first is not an argument.
  • Run a controlled comparison: same corpus, same golden set, same context budget, same retriever, with the splitting strategy as the only variable.
  • Fix the decision threshold before running the numbers so you cannot bend it to fit the result afterwards.
  • Try free structural splitting first, and if semantic chunking is genuinely needed, apply it to the high-value subset rather than the entire corpus.

D5 向量索引与库选型：HNSW 与倒排文件、量化省内存、带过滤的查询与多租户隔离

• 把向量从全精度换成半精度或二值量化，你会用什么方法确认召回没有明显下降？If you switch your vectors from full precision to half precision or binary quantisation, how do you verify that recall has not dropped materially?

  国内高频海外高频进阶#quantization#evaluation#recall

  分析过程 · 先想清楚再作答

  1. 这题表面问量化，实际问的是你会不会做评估。只回答「跑几个问题看看结果对不对」的人会被直接判为没做过——面试官想听的是一套可复现的量法。
  2. 先把真值这件事说死：真值必须来自暴力全量比对，也就是把索引关掉、全表算距离取前 k。拿索引结果当真值是最常见的自欺，因为那样量出来的召回永远接近 100%，你会以为量化无损。
  3. 然后给流程：固定一批查询（几十条起步，覆盖长短查询和不同主题），先用全精度算出真值，再换量化重跑，计算召回率@k。同时记录三件事——索引大小、建索引耗时、查询延迟的中位数与 p95，只报召回是不够的。
  4. 补一条判据：量化损失有多大取决于向量分布，别人的数字不能抄。稀疏向量对二值量化尤其不友好，因为二值化只保留符号位，零和负数会被压成同一个值，信息几乎被抹平。所以换方案必须在自己的数据上重新量一次。
  5. 结论要给可操作的建议：半精度通常近乎无损，还能把建索引维度上限从 2000 提到 4000，是默认可以先上的一档；二值量化损失明显，标准用法是拿它粗筛一批候选，再用原始向量在这一小批里精排，粗筛窗口越宽召回补得越多、延迟也越高。
  6. 可预期的追问：召回掉了多少算可以接受？答这取决于下游——后面还有重排时，粗排召回掉两三个点通常无感；如果检索结果直接进提示词，掉一个点就意味着每一百次回答里多一次缺材料。要把这个判断挂到业务指标上，而不是拍一个阈值。

  How to reason about it · think before answering

  1. The question looks like it is about quantisation, but it is really about whether you know how to evaluate. Answering 'try a few queries and eyeball it' fails immediately.
  2. Pin down ground truth first: it must come from an exhaustive scan with the index disabled. Using index results as ground truth is the classic self-deception, because recall then looks close to 100% no matter what you changed.
  3. Give the procedure: fix a query set of at least a few dozen covering short and long queries across topics, compute ground truth at full precision, rerun with the quantised representation, and report recall at k. Report index size, build time, and median plus p95 latency alongside it, because recall alone is not a decision.
  4. Add the judgement rule: quantisation loss depends on your vector distribution, so published numbers do not transfer. Sparse vectors suffer badly under binary quantisation because only the sign bit survives and zeros collapse together.
  5. Land on something actionable: half precision is usually near lossless and raises the indexable dimension ceiling from 2000 to 4000, so it is a safe first step. Binary quantisation loses real recall and should be used as a cheap first pass, re-ranked with the original vectors over a wider candidate window.
  6. Expected follow-up: how much loss is acceptable? It depends on what comes next. With a re-ranker downstream, a couple of points off first-stage recall is usually invisible; if retrieval feeds the prompt directly, one point means one more unanswerable question per hundred. Tie the threshold to a product metric, not to a number you made up.

  答题要点

  • 真值必须来自关掉索引的暴力全量比对，拿索引结果当真值会让召回永远接近 100%。
  • 固定一批查询，量化前后跑同一批，报召回率@k，同时报索引大小、建索引耗时和延迟分位数。
  • 量化损失取决于向量分布，别人的数字不能抄，必须在自己的数据上重新量。
  • 半精度通常近乎无损，还能把索引维度上限从 2000 提到 4000，可以作为默认第一档。
  • 二值量化损失明显，正确用法是粗筛加原始向量重排，粗筛窗口越宽召回补得越多、延迟越高。

  Key points

  • Ground truth must come from an exhaustive scan with indexes disabled; using index output as truth pins recall near 100%.
  • Run one fixed query set before and after, report recall at k together with index size, build time and latency percentiles.
  • Quantisation loss depends on your own vector distribution, so measure it on your data instead of quoting benchmarks.
  • Half precision is usually near lossless and raises the indexable dimension limit from 2000 to 4000, making it a safe default.
  • Binary quantisation loses real recall; use it as a cheap first pass and re-rank with the original vectors over a wider window.

D6 生成这一侧：上下文怎么排、引用怎么标、什么时候必须拒答，以及流式回答

• 知识库问答的拒答阈值怎么定？定高了和定低了各自的代价是什么？How do you set the refusal threshold for a knowledge-base assistant, and what does it cost you when the threshold is too high or too low?

  国内高频海外高频深入#refusal#thresholds#evaluation

  分析过程 · 先想清楚再作答

  1. 这题真正在考的是：你有没有意识到拒答不是一个阈值，而是好几条判据；以及你定阈值靠不靠数据。只谈一个分数阈值的回答，说明只做过最浅的一层。
  2. 先把拒答拆成三条线，它们的触发时机完全不同。检索分数太低：生成之前就能判，省一次模型调用。材料互相矛盾：也在生成之前判，代码在块之间找同一件事的不同数字，检出后要么并列两种说法与各自的更新日期，要么在有权威信号（比如一份点破了这条不一致的会议纪要）时按更新日期择一——选哪条是产品决策，但无论如何不能让模型自己悄悄挑一个。问题超出材料覆盖范围：只能在生成之后判，判据是跑完引用校验一条有效引用都没有。
  3. 强调三种话术必须不同。第一种要说「库里没有相关材料，换个说法或确认资料是否入库」，第三种要说「找到了相关文档但里面没有能直接回答的内容」——用户的下一步动作完全不同，混成一句「抱歉我不知道」等于把信息扔了。
  4. 再答代价这一半。定高了：能答的问题被挡在门外，用户看到查不到而材料其实在库里，这是最伤信任的一种错，而且它在日志里几乎不可见。定低了：低分噪声材料进上下文，模型拿着不相关的东西硬答，错误反而更隐蔽，因为回答看起来还带着引用。
  5. 怎么定：拿一批已知有答案和已知没答案的问题跑一遍，看两组的分数分布在哪里分开，按你更怕哪种错来取点。分数是没有绝对量纲的，换语料、换检索方式都要重定，所以真正要交付的是这套定阈值的流程，不是那个数字。
  6. 可预期的追问：单一分数阈值不够怎么办？答案是加判据而不是调数字——最高分与次高分的差、命中块数、以及生成后的引用校验结果，都是比原始分数更稳的信号。

  How to reason about it · think before answering

  1. What is really being tested: do you know that refusal is several rules rather than one threshold, and do you set thresholds from data. An answer that mentions only a score cutoff shows you have only touched the surface.
  2. Break refusal into three rules with different timing. Score too low: decidable before generation, saving a model call. Sources conflict: also decidable before generation, by finding differing numbers about the same thing across blocks. You then either present both with their update dates, or pick the newer one when an authoritative signal backs it, such as meeting notes that flagged the discrepancy. Which of the two is a product decision, but silently letting the model pick is never an option. Question outside coverage: only decidable after generation, when citation verification leaves you with zero verified citations.
  3. Stress that the three responses must read differently. 'Nothing relevant in the knowledge base, try rephrasing or check whether the document was ingested' is a different instruction to the user than 'we found related documents but none of them answers this'. Collapsing both into 'sorry, I don't know' throws away information.
  4. Then the cost half. Too high: answerable questions get blocked, the user is told nothing was found while the material is in fact indexed. That is the most trust-damaging failure and it is nearly invisible in logs. Too low: weak passages enter the context and the model answers from irrelevant material, which is worse because the answer still looks cited.
  5. How to set it: run a set of questions with known answers and known non-answers, look at where the two score distributions separate, and pick a point according to which error you fear more. Scores have no absolute scale, so the deliverable is the procedure, not the number.
  6. Expected follow-up: what if one score threshold is not enough? Add signals rather than tuning the number: the gap between top and second score, the number of hits above threshold, and the post-generation verification result are all steadier than the raw score.

  答题要点

  • 拒答不是一条线而是三条：分数过低、材料冲突（都在生成前判）、超出材料覆盖范围（只能生成后按引用校验结果判）。
  • 冲突检出后并列两说还是按更新日期择一，是产品决策；只有在有权威信号背书时择一才站得住，否则老实并列。
  • 三种情况的话术必须不同，因为它们给用户的下一步动作不同。
  • 定高了会把能答的问题挡住，用户看到查不到而材料其实在库里，最伤信任且日志里看不见。
  • 定低了会让噪声材料进上下文，错误更隐蔽，因为回答看起来仍然带着引用。
  • 定法是拿已知有答案与已知没答案的两组问题跑分数分布，按更怕哪种错取点；换语料或换检索方式都要重定。

  Key points

  • Refusal is three rules, not one: low score and source conflict decided before generation, out-of-coverage decided after generation from the verification result.
  • On conflict, presenting both versions versus picking the newer one is a product decision; picking only holds up when an authoritative signal backs it.
  • The three responses must be worded differently because each implies a different next action for the user.
  • Too high blocks answerable questions; the user is told nothing exists while it does, which is the most damaging and least visible failure.
  • Too low lets weak passages in, producing errors that are harder to spot because the answer still carries citations.
  • Set it by comparing score distributions over answerable and unanswerable question sets, then choose based on which error is worse; re-tune whenever the corpus or retriever changes.

D7 第一周综合：把六天的零件装成一个可一键启动的检索问答服务并复盘

• 你刚拼出来的这个检索问答系统，现在最大的风险在哪里？你打算怎么证明这个判断？What is the biggest risk in the RAG service you just assembled, and how would you prove that judgment?

  国内高频海外高频深入#evaluation#risk-assessment#retrospective

  分析过程 · 先想清楚再作答

  1. 这题有两半，后半句才是题眼。说出一个风险不难，难的是给出一个能证伪你自己判断的方法——答不出后半句，前半句就只是意见。
  2. 先排除两个常见的错误答案：说「幻觉」太笼统，没有指向任何可动的地方；说「延迟」则是把看得见的问题当成最大风险。
  3. 真正的最大风险是**没有评估**：切块大小、取几条、门槛定多少、两路怎么加权，全是拍出来的。它最重要的地方在于它让所有其他风险都无法验收——你连「改了之后变好还是变坏」都说不出口。
  4. 怎么证明：先从语料反向出一份带标准答案文档的问题集，刻意掺进无答案问题和需要跨文档的多跳问题；再实现召回率与排序指标，给当前配置跑出一个基线；然后把一个参数来回改两次，看指标动不动。如果指标对参数完全不敏感，说明是评估集有问题，不是系统没问题。
  5. 补一句成本口径：每一项优化都要同时报三笔账——指标涨了多少、延迟涨了多少、钱涨了多少。只报第一笔的结论不能用。
  6. 可预期的追问：评估集多大才够？先做二十题能覆盖主要问题类型的小集，用它挡住明显的退步；等真实用户问题攒起来，再按真实分布扩到几百题。一上来就追求规模，只会得到一堆自己出的、跟真实用法无关的题。

  How to reason about it · think before answering

  1. There are two halves here and the second is the real question. Naming a risk is easy; giving a method that could falsify your own claim is what separates answers from opinions.
  2. Rule out two common wrong answers: 'hallucination' is too vague to act on, and 'latency' mistakes a visible problem for the biggest one.
  3. The biggest risk is the absence of evaluation. Chunk size, top-k, thresholds and route weights were all guessed, and that makes every other risk unverifiable: you cannot even say whether a change helped.
  4. How to prove it: build a question set from the corpus with known answer documents, deliberately including unanswerable and multi-hop questions; implement recall and ranking metrics; produce a baseline for the current configuration; then move one parameter back and forth and watch whether the metrics move. If they do not move at all, the evaluation set is wrong, not the system.
  5. Add the accounting rule: every optimisation reports three numbers, metric gain, latency added and cost added. A claim with only the first is not usable.
  6. Expected follow-up: how large must the set be? Start with roughly twenty questions covering the main question types to catch obvious regressions, then grow toward the real distribution once you have actual user questions. Chasing size first only yields questions you invented yourself.

  答题要点

  • 最大的风险是没有评估：所有参数都是拍的，导致任何改动的好坏都无法判断。
  • 证明方式是先建标准答案集，刻意包含无答案问题与多跳问题，再跑出当前配置的基线。
  • 用参数扰动反过来验证评估集本身：指标对参数完全不敏感，说明题出得有问题。
  • 每项优化同时报三笔账：指标、延迟、成本；只报指标的结论不能用。
  • 评估集先小而全，覆盖问题类型即可，等真实问题攒起来再按真实分布扩大。

  Key points

  • The biggest risk is having no evaluation: every parameter was guessed, so no change can be judged.
  • Prove it by building a golden set with known answer documents, including unanswerable and multi-hop questions, then baseline the current configuration.
  • Validate the set itself by perturbing parameters: metrics that never move mean the questions are wrong.
  • Report three numbers per optimisation: metric gain, added latency, added cost.
  • Start small but well covered, then grow toward the real question distribution.