Interview Bank

RAG in 14 Days: From Retrieval to Trustworthy Answers

D1 Why Retrieve at All: Hallucination, Knowledge Cutoffs, and the Cost of Long Context; a Minimal Keyword-Only RAG

• A retrieval-augmented generation system gave a wrong answer. How do you determine whether retrieval or generation is at fault?一个检索增强生成系统答错了，你怎么定位是检索的锅还是生成的锅？

  Common in ChinaCommon overseasIntermediate#debugging#failure-modes#evaluation

  How to reason about it · think before answering

  1. The question asks how you localise the fault, not what the possible causes are. Listing causes loses; the interviewer wants an ordered procedure that ends in concrete actions.
  2. Give the cheapest first step: print the retrieved passages verbatim and read them. If the correct answer is not in there, retrieval is at fault. If it is in there and the model ignored it, generation is at fault. Thirty seconds, and it removes most of the guesswork.
  3. Then lay out the five stages — chunking, indexing, retrieval, context assembly, generation — with the rule: diagnose right to left, fix left to right. You see the generated answer first, but an error on the left is amplified by everything to its right.
  4. Add symptoms that pin down a stage: half-correct answers usually mean a rule was split across chunks; obviously irrelevant hits usually mean dirty parsing; the model ignoring the supplied material usually means the prompt never said it must; citation numbers that do not match their content point at generation.
  5. Land it in engineering terms: to run this procedure repeatedly you must log the retrieved hits, the passages that entered the context, and the final answer together, otherwise production issues are unreproducible. At scale this becomes a fixed question set with metrics rather than case-by-case reading.
  6. Expected follow-up: if retrieval missed the document, will prompt tuning help? No. Nothing in the prompt can conjure material that was never supplied.

  分析过程 · 先想清楚再作答

  1. 题眼在「怎么定位」，不在「有哪些原因」。答成一串可能原因的罗列就输了，面试官想听的是一个有先后顺序、能落到具体动作的排查流程。
  2. 先给最省时间的第一步：把这次检索出来的几段原文原样打印出来，自己读一遍。正确答案不在里面就是检索的锅，在里面而模型没用上才是生成的锅。这一步三十秒，能省掉大半天的瞎猜。
  3. 然后把链路展开成五个环节——切块、建索引、检索、组装上下文、生成——并给出「排查从右往左、修复从左往右」这条口径：从右往左是因为你最先看到的是生成结果，从左往右是因为左边的错会被右边放大。
  4. 补充几个能把环节钉死的症状：答案「半对」多半是切块把一条完整规则切断了；检索结果里混着一眼不相干的东西多半是解析没做干净；模型无视材料用先验知识作答，通常是提示词里少了「只能依据资料回答」；引用编号和内容对不上，那是生成侧漏读或串了行。
  5. 最后落到工程做法：这套排查要能重复做，就必须把每次请求的检索结果、进上下文的段落、最终回答一起记下来，否则线上出问题时你根本复现不了。到了要批量做的时候，就得换成一批固定问题加指标，而不是一条条人工看。
  6. 可预期的追问：如果检索确实没捞到，改提示词有没有用？答案是没用——材料里没有的东西，再好的指令也只能换一种编法。这句话最能证明你分清了两层。

  Key points

  • Always start by printing the retrieved passages and checking whether the correct answer is present at all.
  • Split the pipeline into chunking, indexing, retrieval, context assembly and generation; diagnose right to left, fix left to right.
  • Use symptoms to pin the stage: half-correct answers point at chunking, irrelevant hits at parsing, ignored material at the prompt, mismatched citations at generation.
  • If retrieval missed the document, prompt changes cannot help; the material simply is not there.
  • Log retrieved hits, the passages that entered the context, and the final answer together, or production failures are unreproducible.

  答题要点

  • 第一步永远是把检索出来的原文打印出来读一遍，判断正确答案在不在里面。
  • 把链路拆成切块、建索引、检索、组装上下文、生成五个环节，排查从右往左、修复从左往右。
  • 用症状钉环节：半对多半是切块问题，混入无关结果多半是解析问题，无视材料多半是提示词缺约束，引用与内容对不上是生成问题。
  • 检索没捞到时改提示词没有意义，材料里没有的东西模型只能编。
  • 要能重复排查就必须把检索结果、进上下文的段落和最终回答一起记录下来。

D2 Embeddings and Vector Search: Similarity, Dimensionality, and Model Choice; Storing Text in pgvector

• Why do some embedding models require different prefixes for queries and documents? What happens if you skip them, and how would you catch it before shipping?为什么有些 embedding 模型要求查询和文档加不同的前缀？不加会怎样，你怎么在上线前发现这个问题？

  Common in ChinaCommon overseasIntermediate#embeddings#model-selection#evaluation

  How to reason about it · think before answering

  1. The core of this question is silent failure. Reciting 'e5 needs query: and passage: prefixes' is the baseline; explaining why nothing errors out and how you would catch it is what shows experience.
  2. The reason: these models are trained on pairs, short questions on one side and longer passages on the other, two genuinely different distributions. The prefix is a role marker learned during training. Omit it at inference and you are off-distribution.
  3. The consequence: the model still returns vectors, distances still compute, results still have an order, quality just degrades. Nothing throws, exactly like forgetting to normalise.
  4. How to catch it: run a small labelled question set against the same corpus twice, with and without prefixes, and compare hit rate. That is the evaluation gate built on day 8, and catching silent regressions is precisely what it is for.
  5. Mention the sneakier variant: prefixing at index time but not at query time, or using the same prefix on both sides. Everything sits in one coordinate space and looks healthier, yet the query-document alignment is wrong and the loss is just as invisible. Encapsulate prefixes in the embedding call convention rather than hand-writing them everywhere.
  6. Expected follow-up: do OpenAI models need prefixes? No, they are not in that family, so this is not a universal rule but a per-model detail you re-check on the model card every time you switch.

  分析过程 · 先想清楚再作答

  1. 这题的题眼是「静默失效」。会背「e5 要加 query 和 passage 前缀」只能拿基础分，能说清它为什么不报错、以及怎么在上线前抓住它，才是做过的人。
  2. 先讲原因：这一族模型是拿成对数据训练的，一侧是短问句、一侧是长段落，两者的分布本来就不一样。前缀是训练时给模型的角色标记，告诉它这一段该按查询编码还是按文档编码。推理时不给，模型就落在了训练分布之外。
  3. 再讲后果的性质：不加前缀模型照样输出向量、照样能算距离、名次照样有先后，只是整体质量下滑。**没有任何报错**——这跟忘了归一化是同一类问题：错误不会自己浮出来。
  4. 怎么发现：唯一可靠的办法是一小份标注问题集，用同一批文档跑两遍（加前缀与不加前缀），比命中率。这就是第 8 天要做的评估闸门，它的价值恰恰在于抓这类静默错误。上线前跑一遍，比读十遍文档管用。
  5. 补一个更容易踩的变体：**建库时加了前缀、查询时忘了加**，或者两边加成同一个前缀。这种情况下所有向量都在同一个坐标系里，看起来更「正常」，但查询与文档的对齐关系是错的，掉分同样查不出来。所以前缀应该封装在 embed 的调用约定里，而不是散在各处手拼。
  6. 可预期的追问：OpenAI 的模型要不要加前缀？不需要——它不属于这一族。所以这不是一条普遍规则，而是**每换一个模型都要重新读模型卡片确认**的事。

  Key points

  • These models are trained on question-passage pairs; the prefix marks which role a text plays, and omitting it puts you off-distribution.
  • Skipping prefixes never errors, it only degrades quality, so the failure is silent.
  • The reliable detection is an A/B run over a small labelled question set, comparing hit rate.
  • A subtler bug is mismatched or identical prefixes on both sides, which looks healthier but misaligns queries and documents.
  • Keep prefixes inside the embedding call convention, and re-read the model card whenever you switch models.

  答题要点

  • 这类模型用问句与段落的成对数据训练，前缀是区分两种角色的标记，缺了就落在训练分布之外。
  • 不加前缀不会报错，只会整体掉分，属于静默失效。
  • 唯一可靠的发现方式是拿一份标注问题集跑 A/B 对比命中率。
  • 更隐蔽的错法是两边前缀不一致或用了同一个前缀，看起来更正常但对齐是错的。
  • 前缀应封装在 embed 的调用约定里；换模型必须重读模型卡片，它不是普遍规则。

D4 Chunking Strategies: Five Approaches — Fixed, Recursive, Structure-Based, Parent-Child, and Semantic — and Choosing by Evaluation, Not Intuition

• How do you decide on chunk size? Name two metrics you would look at, and one counterexample.你怎么决定切块大小？说出你会看的两个指标和一个反例。

  Common in ChinaCommon overseasIntermediate#chunking#evaluation

  How to reason about it · think before answering

  1. The question is about method, not about a number. Answering with a specific default (512 tokens, 1000 characters) already loses it — the interviewer wants to hear that you have a procedure.
  2. State the tension first: large chunks dilute the signal and cost context; small chunks lose the surrounding meaning so the model cannot use them. The two metrics you name should map onto those two failure modes.
  3. Metric one is retrieval-side hit rate: did a document that actually answers the question make it into the context. Metric two is generation-side usability, cheaply proxied by the fraction of chunks that end mid-sentence, and more seriously by faithfulness and whether citations resolve.
  4. Add the point that separates candidates: both metrics must be compared under the same token budget, never under a fixed top-k. With fixed k, bigger chunks simply buy more text and win for the wrong reason.
  5. Make the counterexample concrete: raising chunk size from 400 to 1200 characters can lift hit rate purely because whole short documents now fit in one chunk, which means retrieval stopped doing anything and you are back to stuffing full documents. The metric improved while the system got worse.
  6. Expect the follow-up: where do you start on day one. Pick the strategy from the document type first (structural splitting whenever headings exist), start around 300 to 500 characters with 10 to 20 percent overlap, then build a golden set immediately and iterate. A starting point is not a conclusion.

  分析过程 · 先想清楚再作答

  1. 这题的题眼是「怎么决定」，不是「多大合适」。答一个具体数字（512 token、1000 字符）就已经输了——面试官想看的是你有没有一套定法，而不是你记得住哪个默认值。
  2. 先把矛盾摆出来：块大则信噪比低、上下文贵，块小则单块缺语境、模型答不出所以然。切块大小就是在这两头之间找位置，所以两个指标必须分别对应这两头。
  3. 第一个指标是检索侧的命中率——答案文档有没有进上下文。第二个是生成侧的可用性，最省事的代理指标是切碎率，也就是有多少块结尾停在半句话上；再往前一步就是忠实度和引用是否可定位。
  4. 关键补一句：两个指标必须在**同一个 token 预算**下比，不能按「取前 k 块」比。k 固定时块越大塞进去的字越多，大块切法会赢在买得多而不是切得准上。这一句往往是这道题的区分点。
  5. 反例要具体。最好用的一个是：把块从 400 字调到 1200 字，命中率不降反升——但那是因为一整篇短文档被当成一块塞了进去，检索其实什么都没做，等于退化成了全文投喂。指标涨了，系统更差了。
  6. 可预期的追问是「那你第一次上手时从哪个数字起步」。答：先按文档类型选切法（有标题层级就按结构切），块长从 300 到 500 字起步、重叠取一到两成，然后立刻建一组标准问题跑评估，用两三轮迭代把它调到位。起步值是起步值，不是结论。

  Key points

  • Choose the strategy from the document type first, then tune length: split on headings whenever the structure survives parsing.
  • Watch two metrics: retrieval hit rate on one side, mid-sentence break rate (then faithfulness and citation resolvability) on the other.
  • Compare under an equal token budget, never a fixed top-k, or larger chunks win by buying more text.
  • Counterexample: hit rate rises after enlarging chunks because whole documents now fit in one chunk and retrieval has effectively stopped working.
  • Start near 300 to 500 characters with 10 to 20 percent overlap, then iterate against a fixed question set instead of guessing.

  答题要点

  • 先按文档类型选切法，再调长度：有标题层级就按结构切，没有结构才谈固定长度或语义。
  • 看两个指标：检索侧的命中率，生成侧的切碎率（进一步是忠实度与引用可定位性）。
  • 两个指标必须在同一个 token 预算下比，不能按「取前 k 块」比，否则大块只是买得更多。
  • 反例：块调大后命中率上升，但那是因为整篇被当成一块，检索退化成全文投喂。
  • 起步值 300 到 500 字、重叠一到两成，然后靠一组固定问题迭代，不靠直觉定稿。
• Semantic chunking costs considerably more than recursive splitting. How would you prove to your team that the money is well spent?语义切分比递归切分贵不少，你怎么向团队证明这笔钱值得花？

  Common in ChinaCommon overseasDeep dive#chunking#evaluation#cost

  How to reason about it · think before answering

  1. This looks like a technical question but it tests whether you can run a controlled technical argument. Launching into how semantic chunking works answers a different question.
  2. Step one is to concede that it may well not be worth it. The gain comes from documents that have no usable structure; if your knowledge base is well-formed documents, the authors' heading hierarchy already did the semantic split for free and the money is likely wasted.
  3. Step two is translating 'worth it' into three measurable numbers: how much the metric moved (hit rate on the same golden set under the same token budget), how much latency moved (chunking is offline, but the end-to-end update path changes), and how much it costs (the initial full embedding pass plus recomputation amortised over update frequency).
  4. Step three is the control. Recursive splitting is the baseline, semantic chunking the treatment, and they must share the corpus, the questions, the context budget and the retriever. Change one variable only; a two-variable experiment proves nothing.
  5. Step four is a decision threshold rather than an impression. For example: below three points of hit-rate gain, no; above five points with recomputation inside the monthly budget, yes; in between, roll it out on one document class first. Fix the threshold before you run the numbers, or you will quietly bend it to fit them.
  6. Expect the follow-up: is there a cheaper way to the same gain. Yes — try structural splitting first, since it is free and often nearly as good, and if the structure really is unusable, apply semantic chunking only to the high-value subset rather than the whole corpus.

  分析过程 · 先想清楚再作答

  1. 这题表面问技术，实际考的是你会不会做一次带对照组的技术论证。上来就讲语义切分原理的人，答的是另一道题。
  2. 第一步是先承认它可能不值。语义切分的收益来自「文档没有可用的结构」；如果知识库是结构良好的文档，作者的标题层级已经免费替你做完了语义切分，这时候花的钱大概率打水漂。**先说清适用前提，再谈证明，这一步就把大多数候选人区分开了。**
  3. 第二步是把「值不值」翻译成可测的三笔账：指标涨了多少（同一批标准问题、同一个 token 预算下的命中率）、延迟涨了多少（切块是离线的，但更新链路的端到端时间会变）、钱涨了多少（首次全量 embedding 的费用，加上按更新频率折算的重算费用）。只报第一笔的论证不成立。
  4. 第三步是设计对照。递归切分是基线，语义切分是实验组，两组必须用同一份语料、同一批问题、同一个上下文预算、同一个检索器，只改切法这一个变量。改两个变量的实验，结论一文不值。
  5. 第四步是给决策一个门槛，而不是给一个感想。比如：命中率相对基线提升低于三个百分点就不上；提升超过五个百分点且重算成本在月度预算内就上；中间地带先在一类文档上灰度。**门槛要在跑数字之前定好**，否则你会不自觉地去迁就已经跑出来的结果。
  6. 可预期的追问是「有没有更便宜的办法拿到同样的收益」。答有：先试按结构切，它零成本且效果常常接近；结构确实不可用时，再考虑只对高价值的那一部分文档做语义切分，而不是全量上。

  Key points

  • Start with the precondition: the gain comes from documents without usable structure, so on well-formed documents it usually is not worth it.
  • Translate 'worth it' into three numbers — hit rate, latency, and cost. Reporting only the first is not an argument.
  • Run a controlled comparison: same corpus, same golden set, same context budget, same retriever, with the splitting strategy as the only variable.
  • Fix the decision threshold before running the numbers so you cannot bend it to fit the result afterwards.
  • Try free structural splitting first, and if semantic chunking is genuinely needed, apply it to the high-value subset rather than the entire corpus.

  答题要点

  • 先讲适用前提：语义切分的收益来自文档没有可用结构，结构良好的文档上它大概率不值。
  • 把「值不值」翻译成三笔账：命中率涨多少、延迟涨多少、钱涨多少，只报第一笔不算论证。
  • 做对照实验：同语料、同问题集、同上下文预算、同检索器，只改切法一个变量。
  • 决策门槛必须在跑数字之前定好，避免事后迁就结果。
  • 先试零成本的按结构切；确需语义切分时也优先只覆盖高价值文档，而不是全量上。

D5 Vector Indexes and Store Selection: HNSW vs. Inverted File, Quantization to Save Memory, Filtered Queries and Multi-Tenant Isolation

• If you switch your vectors from full precision to half precision or binary quantisation, how do you verify that recall has not dropped materially?把向量从全精度换成半精度或二值量化，你会用什么方法确认召回没有明显下降？

  Common in ChinaCommon overseasIntermediate#quantization#evaluation#recall

  How to reason about it · think before answering

  1. The question looks like it is about quantisation, but it is really about whether you know how to evaluate. Answering 'try a few queries and eyeball it' fails immediately.
  2. Pin down ground truth first: it must come from an exhaustive scan with the index disabled. Using index results as ground truth is the classic self-deception, because recall then looks close to 100% no matter what you changed.
  3. Give the procedure: fix a query set of at least a few dozen covering short and long queries across topics, compute ground truth at full precision, rerun with the quantised representation, and report recall at k. Report index size, build time, and median plus p95 latency alongside it, because recall alone is not a decision.
  4. Add the judgement rule: quantisation loss depends on your vector distribution, so published numbers do not transfer. Sparse vectors suffer badly under binary quantisation because only the sign bit survives and zeros collapse together.
  5. Land on something actionable: half precision is usually near lossless and raises the indexable dimension ceiling from 2000 to 4000, so it is a safe first step. Binary quantisation loses real recall and should be used as a cheap first pass, re-ranked with the original vectors over a wider candidate window.
  6. Expected follow-up: how much loss is acceptable? It depends on what comes next. With a re-ranker downstream, a couple of points off first-stage recall is usually invisible; if retrieval feeds the prompt directly, one point means one more unanswerable question per hundred. Tie the threshold to a product metric, not to a number you made up.

  分析过程 · 先想清楚再作答

  1. 这题表面问量化，实际问的是你会不会做评估。只回答「跑几个问题看看结果对不对」的人会被直接判为没做过——面试官想听的是一套可复现的量法。
  2. 先把真值这件事说死：真值必须来自暴力全量比对，也就是把索引关掉、全表算距离取前 k。拿索引结果当真值是最常见的自欺，因为那样量出来的召回永远接近 100%，你会以为量化无损。
  3. 然后给流程：固定一批查询（几十条起步，覆盖长短查询和不同主题），先用全精度算出真值，再换量化重跑，计算召回率@k。同时记录三件事——索引大小、建索引耗时、查询延迟的中位数与 p95，只报召回是不够的。
  4. 补一条判据：量化损失有多大取决于向量分布，别人的数字不能抄。稀疏向量对二值量化尤其不友好，因为二值化只保留符号位，零和负数会被压成同一个值，信息几乎被抹平。所以换方案必须在自己的数据上重新量一次。
  5. 结论要给可操作的建议：半精度通常近乎无损，还能把建索引维度上限从 2000 提到 4000，是默认可以先上的一档；二值量化损失明显，标准用法是拿它粗筛一批候选，再用原始向量在这一小批里精排，粗筛窗口越宽召回补得越多、延迟也越高。
  6. 可预期的追问：召回掉了多少算可以接受？答这取决于下游——后面还有重排时，粗排召回掉两三个点通常无感；如果检索结果直接进提示词，掉一个点就意味着每一百次回答里多一次缺材料。要把这个判断挂到业务指标上，而不是拍一个阈值。

  Key points

  • Ground truth must come from an exhaustive scan with indexes disabled; using index output as truth pins recall near 100%.
  • Run one fixed query set before and after, report recall at k together with index size, build time and latency percentiles.
  • Quantisation loss depends on your own vector distribution, so measure it on your data instead of quoting benchmarks.
  • Half precision is usually near lossless and raises the indexable dimension limit from 2000 to 4000, making it a safe default.
  • Binary quantisation loses real recall; use it as a cheap first pass and re-rank with the original vectors over a wider window.

  答题要点

  • 真值必须来自关掉索引的暴力全量比对，拿索引结果当真值会让召回永远接近 100%。
  • 固定一批查询，量化前后跑同一批，报召回率@k，同时报索引大小、建索引耗时和延迟分位数。
  • 量化损失取决于向量分布，别人的数字不能抄，必须在自己的数据上重新量。
  • 半精度通常近乎无损，还能把索引维度上限从 2000 提到 4000，可以作为默认第一档。
  • 二值量化损失明显，正确用法是粗筛加原始向量重排，粗筛窗口越宽召回补得越多、延迟越高。

D6 The Generation Side: Ordering Context, Labeling Citations, When You Must Refuse to Answer, and Streaming Responses

• How do you set the refusal threshold for a knowledge-base assistant, and what does it cost you when the threshold is too high or too low?知识库问答的拒答阈值怎么定？定高了和定低了各自的代价是什么？

  Common in ChinaCommon overseasDeep dive#refusal#thresholds#evaluation

  How to reason about it · think before answering

  1. What is really being tested: do you know that refusal is several rules rather than one threshold, and do you set thresholds from data. An answer that mentions only a score cutoff shows you have only touched the surface.
  2. Break refusal into three rules with different timing. Score too low: decidable before generation, saving a model call. Sources conflict: also decidable before generation, by finding differing numbers about the same thing across blocks. You then either present both with their update dates, or pick the newer one when an authoritative signal backs it, such as meeting notes that flagged the discrepancy. Which of the two is a product decision, but silently letting the model pick is never an option. Question outside coverage: only decidable after generation, when citation verification leaves you with zero verified citations.
  3. Stress that the three responses must read differently. 'Nothing relevant in the knowledge base, try rephrasing or check whether the document was ingested' is a different instruction to the user than 'we found related documents but none of them answers this'. Collapsing both into 'sorry, I don't know' throws away information.
  4. Then the cost half. Too high: answerable questions get blocked, the user is told nothing was found while the material is in fact indexed. That is the most trust-damaging failure and it is nearly invisible in logs. Too low: weak passages enter the context and the model answers from irrelevant material, which is worse because the answer still looks cited.
  5. How to set it: run a set of questions with known answers and known non-answers, look at where the two score distributions separate, and pick a point according to which error you fear more. Scores have no absolute scale, so the deliverable is the procedure, not the number.
  6. Expected follow-up: what if one score threshold is not enough? Add signals rather than tuning the number: the gap between top and second score, the number of hits above threshold, and the post-generation verification result are all steadier than the raw score.

  分析过程 · 先想清楚再作答

  1. 这题真正在考的是：你有没有意识到拒答不是一个阈值，而是好几条判据；以及你定阈值靠不靠数据。只谈一个分数阈值的回答，说明只做过最浅的一层。
  2. 先把拒答拆成三条线，它们的触发时机完全不同。检索分数太低：生成之前就能判，省一次模型调用。材料互相矛盾：也在生成之前判，代码在块之间找同一件事的不同数字，检出后要么并列两种说法与各自的更新日期，要么在有权威信号（比如一份点破了这条不一致的会议纪要）时按更新日期择一——选哪条是产品决策，但无论如何不能让模型自己悄悄挑一个。问题超出材料覆盖范围：只能在生成之后判，判据是跑完引用校验一条有效引用都没有。
  3. 强调三种话术必须不同。第一种要说「库里没有相关材料，换个说法或确认资料是否入库」，第三种要说「找到了相关文档但里面没有能直接回答的内容」——用户的下一步动作完全不同，混成一句「抱歉我不知道」等于把信息扔了。
  4. 再答代价这一半。定高了：能答的问题被挡在门外，用户看到查不到而材料其实在库里，这是最伤信任的一种错，而且它在日志里几乎不可见。定低了：低分噪声材料进上下文，模型拿着不相关的东西硬答，错误反而更隐蔽，因为回答看起来还带着引用。
  5. 怎么定：拿一批已知有答案和已知没答案的问题跑一遍，看两组的分数分布在哪里分开，按你更怕哪种错来取点。分数是没有绝对量纲的，换语料、换检索方式都要重定，所以真正要交付的是这套定阈值的流程，不是那个数字。
  6. 可预期的追问：单一分数阈值不够怎么办？答案是加判据而不是调数字——最高分与次高分的差、命中块数、以及生成后的引用校验结果，都是比原始分数更稳的信号。

  Key points

  • Refusal is three rules, not one: low score and source conflict decided before generation, out-of-coverage decided after generation from the verification result.
  • On conflict, presenting both versions versus picking the newer one is a product decision; picking only holds up when an authoritative signal backs it.
  • The three responses must be worded differently because each implies a different next action for the user.
  • Too high blocks answerable questions; the user is told nothing exists while it does, which is the most damaging and least visible failure.
  • Too low lets weak passages in, producing errors that are harder to spot because the answer still carries citations.
  • Set it by comparing score distributions over answerable and unanswerable question sets, then choose based on which error is worse; re-tune whenever the corpus or retriever changes.

  答题要点

  • 拒答不是一条线而是三条：分数过低、材料冲突（都在生成前判）、超出材料覆盖范围（只能生成后按引用校验结果判）。
  • 冲突检出后并列两说还是按更新日期择一，是产品决策；只有在有权威信号背书时择一才站得住，否则老实并列。
  • 三种情况的话术必须不同，因为它们给用户的下一步动作不同。
  • 定高了会把能答的问题挡住，用户看到查不到而材料其实在库里，最伤信任且日志里看不见。
  • 定低了会让噪声材料进上下文，错误更隐蔽，因为回答看起来仍然带着引用。
  • 定法是拿已知有答案与已知没答案的两组问题跑分数分布，按更怕哪种错取点；换语料或换检索方式都要重定。

D7 Week One Capstone: Assembling Six Days of Parts Into a One-Command Question-Answering Service, and a Retrospective

• What is the biggest risk in the RAG service you just assembled, and how would you prove that judgment?你刚拼出来的这个检索问答系统，现在最大的风险在哪里？你打算怎么证明这个判断？

  Common in ChinaCommon overseasDeep dive#evaluation#risk-assessment#retrospective

  How to reason about it · think before answering

  1. There are two halves here and the second is the real question. Naming a risk is easy; giving a method that could falsify your own claim is what separates answers from opinions.
  2. Rule out two common wrong answers: 'hallucination' is too vague to act on, and 'latency' mistakes a visible problem for the biggest one.
  3. The biggest risk is the absence of evaluation. Chunk size, top-k, thresholds and route weights were all guessed, and that makes every other risk unverifiable: you cannot even say whether a change helped.
  4. How to prove it: build a question set from the corpus with known answer documents, deliberately including unanswerable and multi-hop questions; implement recall and ranking metrics; produce a baseline for the current configuration; then move one parameter back and forth and watch whether the metrics move. If they do not move at all, the evaluation set is wrong, not the system.
  5. Add the accounting rule: every optimisation reports three numbers, metric gain, latency added and cost added. A claim with only the first is not usable.
  6. Expected follow-up: how large must the set be? Start with roughly twenty questions covering the main question types to catch obvious regressions, then grow toward the real distribution once you have actual user questions. Chasing size first only yields questions you invented yourself.

  分析过程 · 先想清楚再作答

  1. 这题有两半，后半句才是题眼。说出一个风险不难，难的是给出一个能证伪你自己判断的方法——答不出后半句，前半句就只是意见。
  2. 先排除两个常见的错误答案：说「幻觉」太笼统，没有指向任何可动的地方；说「延迟」则是把看得见的问题当成最大风险。
  3. 真正的最大风险是**没有评估**：切块大小、取几条、门槛定多少、两路怎么加权，全是拍出来的。它最重要的地方在于它让所有其他风险都无法验收——你连「改了之后变好还是变坏」都说不出口。
  4. 怎么证明：先从语料反向出一份带标准答案文档的问题集，刻意掺进无答案问题和需要跨文档的多跳问题；再实现召回率与排序指标，给当前配置跑出一个基线；然后把一个参数来回改两次，看指标动不动。如果指标对参数完全不敏感，说明是评估集有问题，不是系统没问题。
  5. 补一句成本口径：每一项优化都要同时报三笔账——指标涨了多少、延迟涨了多少、钱涨了多少。只报第一笔的结论不能用。
  6. 可预期的追问：评估集多大才够？先做二十题能覆盖主要问题类型的小集，用它挡住明显的退步；等真实用户问题攒起来，再按真实分布扩到几百题。一上来就追求规模，只会得到一堆自己出的、跟真实用法无关的题。

  Key points

  • The biggest risk is having no evaluation: every parameter was guessed, so no change can be judged.
  • Prove it by building a golden set with known answer documents, including unanswerable and multi-hop questions, then baseline the current configuration.
  • Validate the set itself by perturbing parameters: metrics that never move mean the questions are wrong.
  • Report three numbers per optimisation: metric gain, added latency, added cost.
  • Start small but well covered, then grow toward the real question distribution.

  答题要点

  • 最大的风险是没有评估：所有参数都是拍的，导致任何改动的好坏都无法判断。
  • 证明方式是先建标准答案集，刻意包含无答案问题与多跳问题，再跑出当前配置的基线。
  • 用参数扰动反过来验证评估集本身：指标对参数完全不敏感，说明题出得有问题。
  • 每项优化同时报三笔账：指标、延迟、成本；只报指标的结论不能用。
  • 评估集先小而全，覆盖问题类型即可，等真实问题攒起来再按真实分布扩大。

D8 Evaluation First: Building a Golden Set, Computing Recall and Ranking Metrics, Using a Model as Judge for Faithfulness

• You need to build an evaluation set from scratch for a RAG system over a company knowledge base. How would you do it, and how many questions are enough?让你从零给一个公司知识库的 RAG 系统建评估集，你会怎么做？多少题才算够用？

  Common in ChinaCommon overseasIntermediate#evaluation#golden-set#rag

  How to reason about it · think before answering

  1. The discriminator here is the direction you generate questions in, and whether you can justify a size rather than name one.
  2. Go corpus-first: read each document and write the questions it can answer. The answer document is fixed at authoring time, so labeling is nearly free. Question-first gives you items whose answers nobody can locate.
  3. Give the schema: question, answer document ids, and a type. At minimum three types - single-document, multi-hop, and unanswerable. Multi-hop counts as a hit only when every answer document makes it into the context; unanswerable items are scored on abstention, not recall.
  4. Justify the size: 20 items separate 'broken' from 'usable' and are enough for a smoke gate; 100 to 200 are needed before a two-point delta means anything. Then grow the set - every production failure becomes a new item.
  5. Mention cost and decay: roughly two hours for 20 items, and answer labels must be rechecked whenever the corpus changes, or the set rots and you misread the drop as a system regression.
  6. Expected follow-up: how do you avoid overfitting to the eval set? Keep a held-out slice that never informs tuning, and refresh it from real production questions.

  分析过程 · 先想清楚再作答

  1. 这题的区分度在「出题方向」和「规模的理由」两处。开口就说「找几百个用户真实问题」的，多半没真做过——真实问题的答案在哪篇文档里，没人标得出来。
  2. 先给方向：从语料反向出题，打开每一篇读它能回答什么，出题的那一刻答案文档就已经确定了，标注成本几乎为零。反方向（先想问题再找答案）会得到一堆自己都不知道答案的题。
  3. 再给结构：每题记问题、答案文档列表、类型三个字段；类型至少分单文档、多跳、无答案三类，并说明多跳必须全部答案文档命中才算命中，无答案不参与召回率而是考拒答。
  4. 规模的理由要给出来，不能只报一个数字：20 题能把「完全不能用」和「基本能用」分开，够做冒烟；100 到 200 题才有资格判断「涨了两个点」是真的还是噪声。上线之后每次线上出问题就把那个问题补进集合——评估集是长出来的。
  5. 补一句成本与保鲜：出题是人力活，20 题两小时是正常量级；语料更新后要复核答案文档还在不在，否则集合会悄悄腐烂，指标下跌你会误以为是系统坏了。
  6. 可预期的追问是「怎么防止评估集被过拟合」。答案是留一份不参与调优的保留集，并且定期从线上真实问题里补充新题，只用来验收不用来调参。

  Key points

  • Author corpus-first so the answer document is known at authoring time.
  • Label every item with a type: single-document, multi-hop, unanswerable.
  • Multi-hop requires all answer documents; unanswerable items score abstention, not recall.
  • 20 items for a smoke gate, 100 to 200 to trust small deltas, and keep growing it from production failures.
  • Hold out a slice that never informs tuning to avoid overfitting the set.

  答题要点

  • 从语料反向出题，出题时答案文档就已确定，标注成本最低。
  • 每题标类型：单文档、多跳、无答案，三类缺一不可。
  • 多跳要求全部答案文档命中；无答案不算召回率，考的是拒答。
  • 20 题够冒烟，100 到 200 题才能判断小幅变化；线上故障持续补题。
  • 留一份不参与调优的保留集，防止对评估集过拟合。
• Recall, mean reciprocal rank, and normalized discounted cumulative gain - which failure mode does each one catch first, and what do you miss by watching only one?召回率、平均倒数排名、归一化折损累计增益，这三个检索指标分别在什么故障下会先掉下来？只盯一个会漏掉什么？

  Common in ChinaCommon overseasIntermediate#retrieval-metrics#evaluation#ranking

  How to reason about it · think before answering

  1. This tests whether you know each metric's blind spot, not whether you can recite definitions. Layer them as 'did it show up / how high / how good overall' and you are halfway there.
  2. Recall is boolean: is the answer document in the final context. It catches 'never retrieved', but it does not move when the answer slips from rank 1 to rank 8, as long as it still fits the budget.
  3. MRR looks only at the rank of the first relevant hit, so ranking degradation shows up immediately. Its blind spot: one relevant item in the top ten scores exactly the same as five.
  4. nDCG discounts every relevant hit in the top k by its position, so it tracks overall ranking quality and is the direct optimization target for reranking. Its blind spot is existence - it is zero both when nothing was retrieved and when ranking is terrible.
  5. Conclusion: together they localize the failure. Recall drops means retrieval or chunking; recall flat but MRR down means ranking degraded, reach for a reranker; both stable but nDCG down means more noise crept into the top results.
  6. Expected follow-up: what if a metric saturates? Make the questions harder - a saturated metric means the eval set lost its discriminative power, and further tuning is blind.

  分析过程 · 先想清楚再作答

  1. 这题考的是「知不知道指标之间的盲区」，不是背定义。能把三者按「有没有 / 靠不靠前 / 整体好不好」分层的，基本就答对了一半。
  2. 推导链是这样的：召回率是布尔的——答案文档在不在最终上下文里。它对「压根没捞到」最敏感，但答案从第 1 名掉到第 8 名它一动不动，只要还在预算内。
  3. 倒数排名只看第一条相关结果的名次，所以「答案还在但被挤到后面」它立刻掉。反过来它有个盲区：前十条里有一条命中还是五条命中，它给的分完全一样。
  4. 归一化折损累计增益把前 k 名里每一条相关结果都按名次折算再累加，所以它对「整体排序质量」敏感，是重排最直接的优化目标。它的盲区是不告诉你「有没有」——召回率为零时它也是零，看不出是没捞到还是排得差。
  5. 结论：三个一起看才能定位故障层。召回率掉说明检索或切块出了问题，要动召回策略；召回率不动而倒数排名掉，说明排序退化，该上重排；两者都稳而 nDCG 掉，说明前几名里混进了更多噪声。
  6. 可预期的追问是「指标顶格了怎么办」。真实答案是把题目做难：指标撞天花板说明评估集失去区分度，这时候继续优化系统是在瞎调。

  Key points

  • Recall answers 'did it make it into the context', sensitive to total misses, blind to rank shifts.
  • MRR answers 'how high is the first hit', sensitive to ranking degradation, blind to how many hits there are.
  • nDCG answers 'how good is the top k overall', the direct target for reranking, blind to existence.
  • Only the combination localizes the failure to retrieval, ranking, or noise.
  • State the hit criterion: context is packed against a token budget, not a fixed top-k.

  答题要点

  • 召回率管「有没有进上下文」，对完全没捞到最敏感，对名次变化不敏感。
  • 平均倒数排名管「第一条排第几」，对排序退化最敏感，但分不清命中一条还是五条。
  • 归一化折损累计增益管「前 k 名整体质量」，是重排的直接优化目标，但看不出有没有。
  • 三者组合才能定位故障在召回层、排序层还是噪声层。
  • 命中口径要说清：按 token 预算装上下文，不是按固定条数取前 k。
• What systematic biases does an LLM judge have when scoring RAG faithfulness, and how do you detect them and prove your judge is trustworthy?用模型当裁判来评 RAG 的忠实度，有哪些系统性偏差？你怎么发现它们、又怎么证明你的裁判可信？

  Common in ChinaCommon overseasDeep dive#llm-as-judge#evaluation#faithfulness

  How to reason about it · think before answering

  1. The second half of the question is the discriminator. Plenty of people can name position, length, and self-preference bias; few can say how they prove the judge is trustworthy.
  2. Pair each bias with its mitigation: position bias - score pointwise instead of pairwise, and if you must compare, swap the order and call disagreement a tie; length bias - decompose into claims and score a ratio, so a longer answer grows its own denominator; self-preference - judge with a different vendor or tier than the generator.
  3. Add two prompt-level requirements: fixed rubric anchors (spell out what 1.0, 0.6 and 0.3 mean, or the same input scores differently on different days) and forced structured output that quotes the unsupported sentences verbatim, which is what makes human review possible.
  4. Proving trust has exactly one route: human spot-checks and an agreement rate. Stratify ten to thirty items across types, hits and misses, high and low judge scores; answer one binary question only - is anything here not in the material - and compare. Below 0.8 the judge's scores cannot gate a merge.
  5. A detail that scores points: a very high agreement rate may mean your spot-check was too easy. If all ten sampled answers copy the material verbatim, agreeing is trivial and 100% says nothing about the judge.
  6. Expected follow-up: can the judge itself break? Add probes - fixed inputs with known verdicts, one faithful and one obviously fabricated, checked on every run. An evaluation system fails silently: the numbers keep coming, they just stop meaning anything.

  分析过程 · 先想清楚再作答

  1. 这题的题眼在后半句。能背出「位置偏好、长度偏好、自我偏好」三个名词的人很多，能说出「怎么证明可信」的很少——面试官要的是后者。
  2. 先把三个偏差和各自的缓解手段一一对应：位置偏好用逐条独立打分代替两两比较，非要比较就交换顺序跑两遍、结论不一致判平局；长度偏好用逐句判定加比例计分，写得越长分母越大，长度红利自动消失；自我偏好用跨供应商或跨档位的模型评判，生成和评判不同源。
  3. 再补两条提示词层面的：给死评分锚点，1.0 / 0.6 / 0.3 各自是什么必须写明，否则同一份输入不同天给的分都不一样；强制结构化输出并要求把没支撑的句子原样列出，这是人工复核的抓手。
  4. 证明可信只有一条路：人工抽检算一致率。分层抽十到三十条——各类型都要有、命中和没命中都要有、裁判给高分和低分都要有，只判一个二元问题（有没有材料外的内容），跟裁判的结论比对。低于 0.8 就不能拿它的分数做拦合并这类决策。
  5. 一个能加分的细节：一致率很高不一定是好消息。如果抽的十条都是「答案原样抄自材料」的简单题，判对是理所当然的，这时候 100% 说明的是抽检没难度，不是裁判可靠。
  6. 可预期的追问是「裁判本身会不会坏」。答案是给裁判写探针：喂几组已知正确答案的输入（照抄材料的、明显编造的），每次跑评估都验一遍——评估系统坏掉的方式最阴险，分数照常输出，只是不再有意义。

  Key points

  • Three biases: position, verbosity, and self-preference, each with a matching mitigation.
  • Score pointwise rather than pairwise; decompose into claims and score a ratio to kill the length premium; never let the generator judge itself.
  • Pin rubric anchors in the prompt and force structured output that quotes unsupported sentences.
  • Establish trust through stratified human spot-checks and an agreement rate; below 0.8 the judge cannot gate merges.
  • Add probes with known verdicts so a broken judge is caught on every run.

  答题要点

  • 三个偏差：位置偏好、偏爱长答案、自己评自己，各自有对应的缓解手段。
  • 逐条独立打分代替两两比较；逐句判定按比例计分抵消长度红利；生成与评判不同源。
  • 提示词要给死评分锚点，并强制结构化输出、列出没支撑的句子。
  • 可信度靠人工分层抽检算一致率，低于 0.8 不能用它做拦合并的决策。
  • 给裁判本身写探针，每次跑评估都验一遍它有没有坏。
• Why must a RAG evaluation set include questions the corpus cannot answer, and what does leaving them out hide?RAG 的评估集里为什么一定要放语料里没有答案的问题？不放会掩盖什么？

  Common in ChinaCommon overseasBasic#evaluation#abstention#golden-set

  How to reason about it · think before answering

  1. It looks easy but really asks whether you have considered that the eval set itself can lie. 'To test the refusal path' is a pass; 'without them the worst failure is invisible in the report' is a full mark.
  2. The derivation is one step: a system that always answers scores well on a set of answerable questions only. It stuffs context in, the model writes something, and the set has no column for 'should have refused'. The most dangerous failure simply does not appear.
  3. Conclusion: unanswerable questions are the only thing that makes fabrication visible. They are excluded from recall and scored on abstention instead - did retrieval gate out every weak candidate, and did generation actually say the material does not cover this.
  4. One authoring detail worth stating: unanswerable questions need strong distractor terms. Ask which browsers the web client supports when the corpus only says 'attach your browser and version when filing a ticket'. Without distractors retrieval returns nothing and you are testing your tokenizer, not your system.
  5. Expected follow-up: what if the abstention rate is low? Check two layers - whether the retrieval score gate is effectively a no-op, and whether the generation prompt carries an explicit refusal instruction. You need both; a prompt alone is not a reliable gate.

  分析过程 · 先想清楚再作答

  1. 这题看着简单，实际是在问「你有没有想过评估集本身也会说谎」。答成「为了测试拒答功能」只算及格，答出「不放会让某个故障在报表上完全不可见」才是满分。
  2. 推导只有一步：一个只会硬答的系统，在只有可答问题的评估集上能拿到很高的分——它每次都塞材料给模型，模型每次都编一段话，而评估集根本没有「应该拒答」这一栏。于是最危险的故障在报表上是不存在的。
  3. 结论：无答案问题是唯一能让「乱编」显形的东西。它不参与召回率，它的指标是拒答率——检索侧有没有把不够格的候选全挡下来，生成侧有没有真的说出「资料里没有」。
  4. 出题上有个必须说的细节：无答案问题必须留强干扰词，比如问「网页端支持哪些浏览器」而语料里恰好有一句「提交工单请附上浏览器与版本」。没有干扰词的无答案题检索器一条都捞不到，你测出来的是分词器不是系统。
  5. 可预期的追问是「拒答率低怎么办」。分两层查：先看检索侧的门槛是不是形同虚设（分数阈值定得太低，不相干的块也过关），再看生成侧的提示词有没有明确的拒答指令，两层都要有，只靠提示词兜是不牢的。

  Key points

  • An all-answerable eval set makes 'answers confidently when it should not' completely invisible.
  • Unanswerable items are scored on abstention, not recall, and you check both the retrieval gate and the generation refusal.
  • Author them with strong distractor terms, or retrieval returns nothing and you are testing the tokenizer.
  • Keep them at roughly 15% or more of the set, alongside multi-hop items, as the coverage floor.
  • A low abstention rate splits into two causes: a no-op retrieval score gate, or a missing refusal instruction in the prompt.

  答题要点

  • 只有可答问题的评估集，会让「不知道也硬答」这个故障完全不可见。
  • 无答案问题不算召回率，它的指标是拒答率，检索侧和生成侧各看一层。
  • 出题必须留强干扰词，否则检索器一条都捞不到，测的是分词器。
  • 建议无答案题占比不低于评估集的一成五，跟多跳题一起构成覆盖度底线。
  • 拒答率低要分两层查：检索门槛是否形同虚设，生成提示词有没有拒答指令。

D9 Hybrid Search and Reranking: Two-Path Retrieval, Reciprocal Rank Fusion, Then Re-Ranking the Top Results With a Cross-Encoder

• You replaced pure vector retrieval with hybrid search plus reranking, and after shipping it your eval metrics went down. How do you investigate?你把纯向量检索换成了混合检索加重排，上线之后评估指标反而掉了。你会怎么排查？

  Common in ChinaCommon overseasDeep dive#hybrid-search#evaluation

  How to reason about it · think before answering

  1. This question tests whether you have actually done stage-by-stage attribution. Answering `I would tune the weights and see` loses — that is guessing, not investigating.
  2. Step one is to run the stages apart, not to change code: pure keyword, pure vector, hybrid, and hybrid plus rerank, all on the **same eval set with the same context budget**. Whichever stage the drop appears in is where you look, and this alone separates `fusion is broken` from `reranking is broken`.
  3. Step two asks a specific question: did recall drop, or did the ranking metrics drop? A recall drop means the answer never entered the context at all — a candidate-pool or budget problem. Ranking metrics dropping while recall holds means the answer is still there but pushed down — a fusion-weight or rerank-model problem. The two failures have completely different fixes.
  4. A third common root cause is recall depth. This knob runs against intuition: going deeper is not safer, it lets noise vote too. On a 134-chunk corpus I measured that narrowing each route from 50 to 5 took hybrid recall from 87.5% back to 93.8% and multi-hop from 50% to 75%, while nDCG fell by almost 0.1. The metrics fight each other, so decide which one the product needs first.
  5. A fourth root cause is that the eval protocol quietly changed. Touch the context budget, the hit rule, or the candidate depth, and the old and new numbers stop being comparable — in which case the `drop` may not be a drop at all.
  6. Expected follow-up: how do you avoid this next time? Make the four-way comparison a single command, store the previous report as a baseline, and fail the build with a non-zero exit code on regression. That is precisely why evaluation comes before optimization.

  分析过程 · 先想清楚再作答

  1. 这题考的是你有没有真的做过分阶段归因。答「调一下权重再看看」就输了——那是在猜，不是在查。
  2. 第一步是拆档跑，不是改代码：纯关键词、纯向量、混合、混合加重排四档在**同一份评估集、同一个上下文预算**下各跑一遍。指标掉在哪一档就在哪一档找原因，这一步能立刻区分「融合坏了」和「重排坏了」。
  3. 第二步问一个具体问题：掉的是召回率还是排序指标？召回率掉说明答案根本没进上下文，是候选池或者预算的问题；排序指标掉而召回率没动，说明答案还在、只是被挤到了后面，那是融合权重或重排模型的问题。这两类故障的解法完全不同。
  4. 第三个常见根因是召回深度。每路取多少条这个旋钮方向反直觉：取深了不是更保险，是把噪声也一起投了票。我在一份 134 块的语料上实测过，每路从取 50 收到取 5，混合那一档的召回率从 87.5% 回到 93.8%、多跳档从 50% 回到 75%，而 nDCG 反而掉了近 0.1——两个指标会打架，先想清楚业务要哪个。
  5. 第四个根因是评估口径被悄悄改了。上下文预算、命中判定、候选池深度只要动过一个，新旧数字就不可比，这时候「掉了」可能根本不是真的掉了。
  6. 可预期的追问：怎么防止下次再踩？答把四档对照做成一条命令、把上一版报告存成基线、指标退步就以非 0 退出码拦住合并——这就是评估要先于优化的原因。

  Key points

  • Run all four configurations separately for attribution, on one eval set with one context budget, before touching any parameter.
  • Separate a recall drop from a ranking drop: the first is a candidate-pool or budget issue, the second is a fusion or rerank issue.
  • Check recall depth: taking too many per route lets noise vote, and narrowing it can bring recall back.
  • Confirm the eval protocol did not change; touching budget, hit rule, or candidate depth makes old and new numbers incomparable.
  • Freeze the four-way comparison into one command plus a baseline report, and block merges on regression.

  答题要点

  • 先拆档跑四种配置，在同一份评估集和同一个上下文预算下归因，不要一上来就调参。
  • 区分召回率掉与排序指标掉：前者是候选池或预算问题，后者是融合或重排问题。
  • 查召回深度：每路取太深会把噪声也投进融合，收窄反而可能救回召回率。
  • 确认评估口径没被改：预算、命中判定、候选池深度动过一个，新旧数字就不可比。
  • 把四档对照固化成一条命令加一份基线报告，指标退步直接拦住合并。

D11 Advanced Indexing: Parent-Child Documents, Summary Indexes, Contextual Retrieval, and the Trade-Offs of Tree Aggregation vs. Graph Retrieval

• You have built three different indexes over the same corpus. How do you decide which one a query goes to?同一份语料建了三套索引，检索时你怎么决定走哪一套？

  Common in ChinaCommon overseasIntermediate#index-routing#evaluation#architecture

  How to reason about it · think before answering

  1. Whether this is an easy point or a lost one depends on whether you first ask 'do we actually need three?'. Jumping straight to routing accepts an unverified premise.
  2. Step one is admitting the answer is usually 'none of them — use the default'. Across 30 documents we measured five index structures and every one landed at 93.8% recall, none beating the baseline. The only metric that moved was nDCG@10, which headers lifted from 0.6438 to 0.7218, while the two-stage summary index fell to 87.5%. Each structure patches one specific weakness; without that weakness it is pure overhead.
  3. Step two is routing, and the criterion is not 'which index is more accurate' — that is an offline evaluation question, not something you know at request time. What you do have at request time is the shape of the question: detail-seeking, summarizing, or entity-chaining. Those map onto the chunk index, the tree-summary index and the graph index.
  4. Implementation is a lightweight intent classifier — the same one from the previous day's intent routing, no need to invent another. Carry the decision as request metadata so you can replay it later.
  5. Spell out the fallback: on a misclassification, fall back to the default index rather than fanning out across all three and fusing. Fan-out looks safe but multiplies latency and cost by the number of indexes, and the extra routes usually never make it into the context budget anyway.
  6. Expect 'how do you know the classifier is right'. Log every routing decision and replay the golden set periodically: run each question through all three indexes and check whether the classifier picked the best-scoring one. It is a standing offline job that needs no human labelling.

  分析过程 · 先想清楚再作答

  1. 这题是送分还是丢分，取决于你有没有先反问一句『真的需要三套吗』。上来就答路由策略的人，默认了一个没被验证的前提。
  2. 第一步是承认多数情况下答案是『都不走，走默认那套』。我们在 30 篇语料上把五种索引结构各测一遍，**召回率全部停在 93.8%，没有一种跑赢基线**；唯一动了的是 nDCG@10（块头把它从 0.6438 抬到 0.7218），而两段式的摘要索引还掉到了 87.5%。每种结构补的都是一个特定短板，你没有那个短板时它只带来成本。
  3. 第二步才是路由，而判据不是『哪套准』——那是离线评估该回答的问题，不是运行时能知道的。运行时能拿到的只有**问题的形状**：细节型（答案落在某一段）、概括型（要全库的一个概括）、多跳型（要跨实体串联）。按形状分流，正好对应块级索引、树状聚合索引、图索引。
  4. 实现上就是一个轻量意图分类器，跟前一天的意图路由是同一套东西，不必再造一个。分类结果作为元数据带进请求，方便事后拿评估集回看分错了多少。
  5. 兜底策略要说清楚：分类错了**回落到默认那一套**，不要并行全查一遍再融合。并行看着稳，实际上把延迟和成本按索引套数翻倍，而多出来的那两路大概率一条都进不了上下文预算。
  6. 可预期的追问是『怎么知道分类器分对了』。答案是把路由决策记进日志，定期拿标准答案集回放：对每个问题分别走三套索引，看分类器选的那套是不是指标最好的那套。这是一个能持续跑的离线作业，不需要人工标注。

  Key points

  • First challenge the premise: all five index structures landed at the same 93.8% recall in our measurement, so an index without a matching weakness is pure cost.
  • At request time the usable signal is question shape — detail, summary, or entity-chaining — mapping to chunk, tree-summary and graph indexes.
  • Reuse the previous day's intent router for classification and record the routing decision as request metadata.
  • Fall back to the default index on misclassification instead of fanning out and fusing, which multiplies latency and cost.
  • Replay the golden set periodically to check whether the classifier picks the best-scoring index.

  答题要点

  • 先反问是否真需要三套：实测五种索引结构召回率全部持平在 93.8%，没有对应短板就是纯成本。
  • 运行时的判据是问题的形状——细节型、概括型、多跳型，分别对应块级、树状摘要、图索引。
  • 复用前一天的意图路由做分类，把路由决策记进请求元数据。
  • 分类错了回落到默认索引，不要并行全查再融合——延迟和成本按套数翻倍。
  • 用标准答案集定期回放，检验分类器选的那套是不是指标最好的那套。

D14 Capstone Project and Retrospective: A Multi-Tenant Enterprise Knowledge-Base Q&A, a RAG Decision Map, and an Interview Deep Dive

• How do you convince a non-technical stakeholder that your retrieval system actually got better?怎么向不懂技术的业务方证明你的检索系统真的变好了？

  Common in ChinaCommon overseasBasic#evaluation#stakeholder-communication#abstention

  How to reason about it · think before answering

  1. This is a communication question whose scoring hinges on technical judgement: which numbers you choose to show reveals whether you understand the metrics yourself. Dumping recall, nDCG and MRR on a business stakeholder reads as tone-deaf; saying 'user feedback improved' reads as unmeasured.
  2. Start from a principle: show them something they can adjudicate themselves. They cannot judge normalized discounted cumulative gain, but they can absolutely judge 'out of these hundred real questions, how many did it answer correctly, how many wrongly, and how many did it honestly decline'. So the external framing is three numbers — correct, wrong, declined — and they sum to one hundred.
  3. The crucial move is separating wrong from declined, and it is the fastest way to earn trust: saying 'not found' is a correct output, not a failure; the failure is inventing an answer when nothing was found. Teams that report a single 'accuracy' number can be gamed by a system that learns to decline everything, which is why all three must appear side by side.
  4. Then supply checkable evidence rather than only numbers: take ten real questions and show before-and-after answers with clickable citations on every claim. A stakeholder who opens the source and verifies one claim is more convinced than by any percentage, and the exercise doubles as the human spot-check you need anyway to calibrate whether your model judge is trustworthy.
  5. There is a lesson from this course worth volunteering: a column of perfect scores means the ruler is broken. Our questions were written backwards from the corpus, lexical overlap is unusually high, and mean reciprocal rank sits at exactly 1.0000. Showing that to a stakeholder only invites the misreading that you are already perfect, when in fact the metric has saturated. When a metric hits the ceiling, the response is to make the questions harder.
  6. Expected follow-up: how do you get the business side involved? One very practical answer: let them supply questions. Every production miss gets appended to the golden set, so the evaluation set grows rather than being built once. Then each release can point at 'the question you raised last month now answers correctly', which lands better than any status report.

  分析过程 · 先想清楚再作答

  1. 这题在考沟通，但拿分点在技术判断上：你选哪几个数字给业务方看，暴露了你自己有没有看懂这些指标。把召回率、nDCG、MRR 一股脑摊出去的答法会被判成不懂受众；只说「用户反馈变好了」又会被判成没有度量。
  2. 先立一条原则：**给业务方看的必须是他们能自己判断对错的东西**。归一化折损累计增益他们没法判断，而「这一百个真实问题里，系统答对了多少、答错了多少、老老实实说查不到了多少」他们一眼就能判断。所以对外的口径应该是三个数：答对率、答错率、拒答率，而且三个加起来是一百。
  3. 关键是把**答错和拒答分开**。这一条最能建立信任：查不到就说查不到不是故障，是正确输出；真正的故障是查不到还编一段。很多团队只报「准确率」，结果一个学会了一直拒答的系统能刷出满分——所以这三个数必须并排出现，缺一个都能被骗。
  4. 然后给可核对的证据，而不是只给数字：**挑十条真实问题做前后对照**，各贴出改动前和改动后的回答，每句结论后面挂着可点开的引用。业务方点开原文核对一遍，比看任何百分比都有说服力，而且这个动作顺带完成了一次人工抽检——你自己也需要它来校准模型裁判靠不靠谱。
  5. 本课里有一条要主动说的教训：**一列全是满分说明尺子坏了**。我们的题目是从语料反向出的，字面重合度过高，平均倒数排名恒为 1.0000。这个数字拿给业务方看，只会换来一次「那你们已经完美了」的误会，而它其实是指标饱和。指标撞天花板时该做的是把题目出难一点。
  6. 可预期的追问：那怎么让业务方参与进来？答一条很实用的：让他们提供题目。把线上答错的问题一条条补进标准答案集，评估集是长出来的，而不是一次性造好的；这样每一次改进都能指着「你上次提的那个问题现在答对了」，比任何汇报都直接。

  Key points

  • Externally report three numbers they can adjudicate: correct, wrong, declined — summing to one hundred.
  • Keep wrong and declined separate; a single accuracy number is gamed by a system that learns to decline everything.
  • Pair it with ten before-and-after real questions, every claim carrying a citation they can open and verify.
  • Volunteer the saturation caveat: a column of perfect scores means a broken ruler, and the fix is harder questions.
  • Let stakeholders contribute questions; append every production miss to the golden set so it grows over time.

  答题要点

  • 对外只用三个他们能自己判断的数：答对率、答错率、拒答率，三者相加为一百。
  • 答错和拒答必须分开——查不到就说查不到是正确输出，只报一个准确率会被「一直拒答」刷满分。
  • 配十条真实问题的前后对照，每句结论挂可点开的引用，让他们自己核对原文。
  • 主动说明指标饱和：某一列恒为满分是尺子坏了，不是系统完美，该做的是把题目出难一点。
  • 让业务方提供题目，把线上答错的问题补进标准答案集——评估集是长出来的。
• If you could only fund three changes to improve an existing RAG system, which three would you pick and why those three?如果预算只够做三件事来提升一个已有 RAG 系统的效果，你选哪三件？为什么是这三件？

  Common in ChinaCommon overseasIntermediate#prioritization#evaluation#abstention

  How to reason about it · think before answering

  1. This tests prioritisation, not breadth. Answering with a list of techniques — add reranking, add hybrid retrieval, add query rewriting — almost always loses points, because it skips a prerequisite: how do you know those three help your system? That is precisely the sentence the interviewer is waiting for.
  2. So the first item has to be building evaluation, with a reason specific enough to be unarguable: without a scale, you cannot tell whether the other two helped or hurt; with one, every subsequent spend has a measurable return. It is also cheap — the three retrieval metrics are pure local computation, run in seconds, cost nothing, and can gate every commit; the only real effort is labelling answer documents once. Include the composition rule: multi-hop and unanswerable each above ten percent, because without the unanswerable class a system that only ever guesses scores perfectly on your report.
  3. Second, move abstention out of the prompt and into code — usually the best return per unit of effort, and the item most often skipped. Writing 'say you don't know' ten times in a prompt buys almost nothing. Citation numbers are a closed set, so checking existence is one line, and adding a substantive-overlap check catches the harder forgery where the number is real but the content is not. Our baseline abstention rate was 0.0 percent: four questions with no answer in the corpus, zero of them declined — a defect that is completely invisible on a report that only shows recall.
  4. Third, look at the failure cases before deciding, which is the actual answer to this question. After reading the panel you land on one of a few branches: a high share of multi-hop means bridging retrieval or a different index structure; queries that miss when phrased differently mean you need the vector route or hybrid retrieval; answers retrieved but never packed into context means reranking or budget. Failure cases first, technique second — we tried five advanced index structures and not one beat the baseline, because our system simply did not have the weakness they address.
  5. Why not the flashier options: agentic retrieval concentrates its gains on multi-hop while spreading cost across every question, and in our measurements turning on every query-side technique produced exactly the same recall as the default configuration while using 2.5 times the model calls and 4.3 times the retrievals. Stacking techniques is easy; explaining why you switched several off is the skill.
  6. Expected follow-up: once the three are done, how do you prove the money was well spent? Toggle each one individually and report three ledgers — how much the metric moved, how much latency moved, how much cost moved. A proposal that reports only the first should not be approved, including your own.

  分析过程 · 先想清楚再作答

  1. 这题在考优先级判断，而不是知识面。答成「上重排、上混合检索、上查询改写」这类手法清单几乎必然掉分——因为它跳过了一个前提：**你凭什么知道这三件对你的系统有用？** 面试官等的就是这句话。
  2. 所以第一件必须是**建评估**，而且理由要具体到不可反驳：没有秤，剩下两件做完你也说不清是变好还是变坏；有了秤，后面每一笔钱都能算回报。而且它便宜——检索侧三个指标是纯本地计算、几秒钟、零成本，能挂进每次提交；花时间的只是给题目标答案文档那一次。顺带说清评估集的配比：多跳与无答案各占一成以上，缺了无答案那一类，一个只会硬答的系统在报表上就是满分。
  3. 第二件是**把拒答从提示词搬进代码**，这一件的性价比通常最高而最容易被跳过。提示词里写十遍「找不到就说找不到」增益接近于零；而引用编号是一个闭集，判它存不存在只要一行代码，再加一道「这句话与被引块的实质重合度」就能拦住「编号是真的、内容是假的」那一类。我们实验里的基线拒答率是 0.0%——四道语料里根本没有答案的题一道都没闭嘴，这类缺陷在只报召回率的报表上完全不可见。
  4. 第三件要**先看失败案例再决定**，这才是这道题真正的答案。看完面板你会落到其中之一：多跳题占比高就补桥接检索或改索引结构；换个说法就捞不到，说明该上向量那一路或混合检索；答案捞到了却排不进上下文，那是重排或者预算的活。**先有失败案例，再有手法**——我们试过五种高级索引结构，没有一种跑赢基线，因为我们的系统压根没有那些结构要补的短板。
  5. 为什么不选那些看起来更亮的：Agentic 检索的收益集中在多跳题上而代价摊给全部问题；「全开」所有查询侧手法在我们的实测里召回率和默认配置一模一样，模型调用却是 2.5 倍、检索次数 4.3 倍。**堆手法很容易，说清楚为什么关掉某几项才是本事。**
  6. 可预期的追问：三件做完怎么证明钱花对了？答：每一项单独开关各跑一遍，报三笔账——指标涨了多少、延迟涨了多少、钱涨了多少。只报第一笔的提案不该被批准，包括你自己的。

  Key points

  • First, build evaluation: without a scale the other two changes are unverifiable, and the retrieval metrics are cheap enough to gate every commit.
  • The golden set must include unanswerable questions, or a system that only ever guesses scores perfectly on your report.
  • Second, move abstention from the prompt into code: citation numbers are a closed set, and a substantive-overlap check catches real-number-fake-content forgeries.
  • Third is chosen by the failure cases, not by a list of techniques — failure cases first, index structure or retrieval trick second.
  • Toggle each change individually and report three ledgers: metric, latency, cost. A proposal reporting only the first should not be approved.

  答题要点

  • 第一件是建评估：没有秤，另外两件做完也说不清变好还是变坏；检索侧指标零成本可挂进每次提交。
  • 评估集必须含无答案那一类，否则一个只会硬答的系统在报表上就是满分。
  • 第二件是把拒答从提示词搬进代码：编号是闭集，再加实质重合度就能拦住「编号真、内容假」。
  • 第三件由失败案例决定，不由手法清单决定——先有失败案例，再有索引结构或检索手法。
  • 每一项单独开关跑一遍并报三笔账：指标、延迟、钱。只报第一笔的提案不该被批准。