面试题库

14 天 RAG：从检索到可信回答

D1 为什么要检索：幻觉、知识截止与长上下文的代价，以及一个纯关键词的最小 RAG

• 上下文窗口已经做到上百万 token 了，检索这一步会被淘汰吗？Context windows are now in the millions of tokens. Does that make the retrieval step obsolete?

  国内高频海外高频深入#long-context#cost#system-design

  分析过程 · 先想清楚再作答

  1. 这是一道立场题，容易答成非黑即白。判断你有没有做过的地方在于：会不会区分「技术上能不能塞进去」和「工程上该不该每次都塞」，只谈前者的答案一听就是纸上谈兵。
  2. 先承认对方有道理的部分：窗口变大确实吃掉了检索的一部分场景。几十篇文档、更新不频繁、调用量不大的内部工具，直接全塞是最省事的选择，为它建一套检索系统是过度设计。
  3. 再给三条它吃不掉的理由。第一是成本：材料是按次计费的，同一份材料被问一万次就要付一万次，而检索只付取回的那几段；预填充缓存能缓解但不能消除，缓存也有有效期和命中率。
  4. 第二是规模：企业知识库动辄几十万篇，再大的窗口也塞不下，检索是唯一的入口。第三是归因与权限：答案要指回具体某一段，以及不同的人只能看到自己有权访问的材料——这两件事必须在把材料喂给模型之前完成，窗口再大也不解决。
  5. 还要补一条经验事实：材料变多之后，模型在长上下文里定位关键信息的稳定性会下降，出现「读了但没读到」。所以「全塞」并不总是等于「效果更好」，很多时候少而准反而更好。
  6. 可预期的追问：那检索的形态会不会变？会——窗口变大之后，取回的块可以更大、条数可以更多，重排与压缩的压力变小，检索从「精挑几句」变成「粗筛一批」。趋势是检索的粒度变粗，不是检索消失。

  How to reason about it · think before answering

  1. This is a position question and it is easy to answer as a binary. The signal is whether you separate what fits technically from what is worth paying for on every request.
  2. Concede the valid half first: bigger windows genuinely absorb part of the use case. For an internal tool over a few dozen stable documents with low traffic, stuffing everything in is the right call and building a retrieval stack would be over-engineering.
  3. Then give three reasons it does not absorb the rest. Cost is the first: context is billed per request, so the same corpus is paid for on every one of ten thousand queries, whereas retrieval only pays for the passages it returns. Prompt caching softens this but does not remove it.
  4. Scale is the second: enterprise corpora run to hundreds of thousands of documents and no window holds them. Attribution and access control are the third: pointing an answer at a specific passage, and showing each user only what they are permitted to see, both have to happen before the material reaches the model.
  5. Add the empirical point: as the supplied material grows, models become less reliable at locating the one relevant fact inside it. More context is not automatically better; fewer and more precise passages often win.
  6. Expected follow-up: does retrieval change shape? Yes. Larger windows allow bigger chunks and more of them, which relieves pressure on reranking and compression. Retrieval gets coarser, it does not disappear.

  答题要点

  • 先区分「能不能塞进去」和「该不该每次都塞」，前者是技术问题，后者是成本问题。
  • 小规模、低频、少变的语料确实可以直接全塞，为它建检索系统是过度设计。
  • 检索不会被淘汰的三个理由：按次计费的成本、几十万篇塞不下的规模、必须在喂给模型之前完成的归因与权限过滤。
  • 材料越多，模型定位关键信息的稳定性越差，全塞不等于效果更好。
  • 趋势是检索粒度变粗——块更大、条数更多、重排压力变小，而不是检索消失。

  Key points

  • Separate whether it fits from whether it is worth paying for on every request.
  • Small, stable, low-traffic corpora can legitimately be stuffed whole; building retrieval for them is over-engineering.
  • Three reasons retrieval survives: per-request cost, corpora too large for any window, and attribution plus access control that must happen before the model sees the material.
  • More supplied context reduces the reliability of locating a single fact, so stuffing everything is not automatically better.
  • The trend is coarser retrieval — bigger chunks, more of them, less reranking pressure — not the removal of retrieval.

D3 文档进来这一关：PDF 与 HTML 解析、表格与扫描件、清洗规则和必须留下的元数据

• 扫描件走光学字符识别之后错字率不低，这些噪声会怎样影响检索和生成？怎么缓解？OCR output from scanned documents carries a non-trivial error rate. How does that noise propagate into retrieval and generation, and how do you mitigate it?

  国内高频海外高频深入#ocr#data-quality#hybrid-search

  分析过程 · 先想清楚再作答

  1. 这题考的是你会不会顺着链条推传导，而不是背「OCR 会有错字」这句废话。判据是有没有分别说清「检索侧怎么错」和「生成侧怎么错」——它们的失效方式完全不同。
  2. 先说检索侧。中文 OCR 的错主要是形近字，「已」认成「己」、「板」认成「版」。关键词检索是字面匹配，一个字错了这个词就查不到；更隐蔽的是二元组分词会连带毁掉相邻两个词元，一个错字影响的其实是两处。这一路的表现是召回悄悄掉下去，而且不报错。
  3. 再说生成侧。错字进了上下文，模型往往能读懂大意，但一旦是关键实体（人名、型号、金额、日期）出错，它会照着错的答，而且答得很自信。更麻烦的是引用校验也会跟着失效——原文本身就是错的，校验通过了也没意义。
  4. 缓解按三层说。入口层：先用空文本比例这类断言判断这份 PDF 有没有文本层，有就别走 OCR；真要走，保留原图链接以便人工复核。
  5. 检索层：靠混合检索兜底，向量一路对个别错字不敏感，能补上关键词一路的失手，这是 D9 那套东西在这里的具体价值。生成层：把低置信度的页面标出来，让模型在引用它们时明确提示「该材料来自扫描件，可能有识别误差」。
  6. 可预期的追问：能不能自动纠错？可以但要克制——用词典或模型做后处理会修好一批，也会「修」坏一批原本正确的专有名词。稳妥的做法是只对置信度低的片段做纠错，并且保留原文以便回退。

  How to reason about it · think before answering

  1. This tests whether you can trace propagation rather than recite that OCR makes mistakes. The differentiator is separating how retrieval fails from how generation fails, because the two failure modes are entirely different.
  2. Retrieval first. Chinese OCR errors are mostly visually similar characters. Keyword search is literal, so one wrong character makes the term unmatchable, and bigram tokenisation makes it worse because a single wrong character corrupts two adjacent tokens. Recall drops quietly and nothing raises an error.
  3. Generation second. The model usually reads through minor noise, but when the corrupted token is a key entity such as a name, a model number, an amount or a date, it answers confidently with the wrong value. Citation checking degrades too: verifying against a source that is itself wrong proves nothing.
  4. Mitigate in three layers. At ingest, use an empty-text assertion to decide whether the PDF even needs OCR, and keep a link to the original image so a human can verify.
  5. At retrieval, hybrid search absorbs some of the damage because dense retrieval is less sensitive to a single wrong character than literal matching. At generation, mark low-confidence pages so the answer can state that the source came from a scan and may contain recognition errors.
  6. Expected follow-up: can you auto-correct? Yes, but carefully. Dictionary or model based post-processing fixes some errors and breaks correct proper nouns. Restrict correction to low-confidence spans and keep the raw text so you can fall back.

  答题要点

  • 检索侧：形近字让字面匹配直接查不到，二元组分词还会让一个错字毁掉相邻两个词元，表现是召回悄悄下降且不报错。
  • 生成侧：模型能读懂大意，但关键实体出错时会自信地答错，引用校验也失去意义。
  • 入口层缓解：先判断有没有文本层再决定要不要 OCR，并保留原图链接供人工复核。
  • 检索层缓解：混合检索里的向量一路对个别错字不敏感，能兜住关键词一路的失手。
  • 生成层缓解：标出低置信度来源，让回答显式提示可能存在识别误差；自动纠错只对低置信片段做并保留原文。

  Key points

  • Retrieval: visually similar characters break literal matching, and bigram tokenisation lets one bad character corrupt two tokens, so recall drops silently.
  • Generation: the model reads through general noise but confidently repeats corrupted entities, and citation verification against a corrupted source proves nothing.
  • At ingest: check for a text layer before running OCR at all, and keep the source image for human verification.
  • At retrieval: hybrid search helps because dense retrieval tolerates a single wrong character better than literal matching.
  • At generation: flag low-confidence sources in the answer, and restrict auto-correction to low-confidence spans while keeping the raw text.

D4 切块策略：固定、递归、按结构、父子与语义五种切法，以及用评估而不是直觉来选

• 语义切分比递归切分贵不少，你怎么向团队证明这笔钱值得花？Semantic chunking costs considerably more than recursive splitting. How would you prove to your team that the money is well spent?

  国内高频海外高频深入#chunking#evaluation#cost

  分析过程 · 先想清楚再作答

  1. 这题表面问技术，实际考的是你会不会做一次带对照组的技术论证。上来就讲语义切分原理的人，答的是另一道题。
  2. 第一步是先承认它可能不值。语义切分的收益来自「文档没有可用的结构」；如果知识库是结构良好的文档，作者的标题层级已经免费替你做完了语义切分，这时候花的钱大概率打水漂。**先说清适用前提，再谈证明，这一步就把大多数候选人区分开了。**
  3. 第二步是把「值不值」翻译成可测的三笔账：指标涨了多少（同一批标准问题、同一个 token 预算下的命中率）、延迟涨了多少（切块是离线的，但更新链路的端到端时间会变）、钱涨了多少（首次全量 embedding 的费用，加上按更新频率折算的重算费用）。只报第一笔的论证不成立。
  4. 第三步是设计对照。递归切分是基线，语义切分是实验组，两组必须用同一份语料、同一批问题、同一个上下文预算、同一个检索器，只改切法这一个变量。改两个变量的实验，结论一文不值。
  5. 第四步是给决策一个门槛，而不是给一个感想。比如：命中率相对基线提升低于三个百分点就不上；提升超过五个百分点且重算成本在月度预算内就上；中间地带先在一类文档上灰度。**门槛要在跑数字之前定好**，否则你会不自觉地去迁就已经跑出来的结果。
  6. 可预期的追问是「有没有更便宜的办法拿到同样的收益」。答有：先试按结构切，它零成本且效果常常接近；结构确实不可用时，再考虑只对高价值的那一部分文档做语义切分，而不是全量上。

  How to reason about it · think before answering

  1. This looks like a technical question but it tests whether you can run a controlled technical argument. Launching into how semantic chunking works answers a different question.
  2. Step one is to concede that it may well not be worth it. The gain comes from documents that have no usable structure; if your knowledge base is well-formed documents, the authors' heading hierarchy already did the semantic split for free and the money is likely wasted.
  3. Step two is translating 'worth it' into three measurable numbers: how much the metric moved (hit rate on the same golden set under the same token budget), how much latency moved (chunking is offline, but the end-to-end update path changes), and how much it costs (the initial full embedding pass plus recomputation amortised over update frequency).
  4. Step three is the control. Recursive splitting is the baseline, semantic chunking the treatment, and they must share the corpus, the questions, the context budget and the retriever. Change one variable only; a two-variable experiment proves nothing.
  5. Step four is a decision threshold rather than an impression. For example: below three points of hit-rate gain, no; above five points with recomputation inside the monthly budget, yes; in between, roll it out on one document class first. Fix the threshold before you run the numbers, or you will quietly bend it to fit them.
  6. Expect the follow-up: is there a cheaper way to the same gain. Yes — try structural splitting first, since it is free and often nearly as good, and if the structure really is unusable, apply semantic chunking only to the high-value subset rather than the whole corpus.

  答题要点

  • 先讲适用前提：语义切分的收益来自文档没有可用结构，结构良好的文档上它大概率不值。
  • 把「值不值」翻译成三笔账：命中率涨多少、延迟涨多少、钱涨多少，只报第一笔不算论证。
  • 做对照实验：同语料、同问题集、同上下文预算、同检索器，只改切法一个变量。
  • 决策门槛必须在跑数字之前定好，避免事后迁就结果。
  • 先试零成本的按结构切；确需语义切分时也优先只覆盖高价值文档，而不是全量上。

  Key points

  • Start with the precondition: the gain comes from documents without usable structure, so on well-formed documents it usually is not worth it.
  • Translate 'worth it' into three numbers — hit rate, latency, and cost. Reporting only the first is not an argument.
  • Run a controlled comparison: same corpus, same golden set, same context budget, same retriever, with the splitting strategy as the only variable.
  • Fix the decision threshold before running the numbers so you cannot bend it to fit the result afterwards.
  • Try free structural splitting first, and if semantic chunking is genuinely needed, apply it to the high-value subset rather than the entire corpus.

D5 向量索引与库选型：HNSW 与倒排文件、量化省内存、带过滤的查询与多租户隔离

• 为什么加了 WHERE 条件的向量检索会漏结果？有哪几种修法，代价分别是什么？Why does a vector search with a WHERE clause return fewer results than expected, and what are the fixes and their costs?

  国内高频海外高频深入#filtering#iterative-scan#recall

  分析过程 · 先想清楚再作答

  1. 这题是本天的核心，也是最能筛掉「只跑过 demo」的人的一题。题眼在「漏」这个字：能不能说清楚漏的是条数还是排序，直接决定你被归到哪一档。
  2. 先讲机制，一句话就够：近似索引的过滤发生在索引扫描之后。索引先按距离取回 ef_search 个候选，然后才拿 WHERE 去筛这一批。条件命中率越低，活下来的越少——命中 1% 的条件配默认的 40 个候选，平均只剩零点几条。
  3. 然后把漏召回拆成两类，这是拿分点：一类是**结果条数不够**，十条只给了一两条；另一类是**条数够但排序不对**，十条都在只是排错了。两类的修法完全不同，混为一谈说明没真跑过。
  4. 修法一是迭代扫描（pgvector 0.8.0 起）：候选被过滤掉太多时自动回索引里继续扫，直到凑够。它只解决第一类。两种模式的取舍要说清楚——严格顺序保证结果按距离排好，宽松顺序允许略微乱序换更高召回，代价都是延迟明显上升。
  5. 修法二是预过滤，即让过滤条件先生效：条件很挑剔时给过滤列建普通索引走精确检索，取值只有少数几个时建部分索引，取值很多时按值做列表分区。代价分别是失去近似索引的加速、索引数量随取值爆炸、以及 DDL 与运维复杂度上升。
  6. 可预期的追问：怎么判断该用哪一种？给一条可执行的判据——先看返回条数够不够。不够是第一类，先试迭代扫描；够了但召回低是第二类，只能加大 probes 或 ef_search，或者干脆改成预过滤。

  How to reason about it · think before answering

  1. This is the question that separates people who ran a demo from people who ran this in production. The tell is whether you distinguish missing rows from mis-ordered rows.
  2. State the mechanism in one sentence: with approximate indexes, filtering is applied after the index scan. The index first collects ef_search candidates by distance, and only then applies the WHERE clause to that batch.
  3. Do the arithmetic out loud: a condition matching 1% of rows against a default candidate list of 40 leaves well under one row on average. That is why the query looks broken even though the rows exist.
  4. Split the failure into two kinds. Too few rows returned is one; enough rows but the wrong ones ranked first is the other. They have different fixes, and conflating them signals inexperience.
  5. Fix one is iterative scanning, available since pgvector 0.8.0: when too many candidates are filtered out, keep scanning more of the index until enough results are found. Strict ordering keeps exact distance order, relaxed ordering trades slight reordering for better recall, and both cost latency.
  6. Fix two is making the filter apply first: a plain index on the filter column for highly selective conditions, a partial index when there are only a few distinct values, list partitioning when there are many. The costs are losing the approximate speedup, index count exploding per value, and DDL plus operational complexity.
  7. Expected follow-up: how do you pick? Check the returned row count first. Too few means iterative scanning; enough rows with low recall means raising probes or ef_search, or switching to pre-filtering.

  答题要点

  • 近似索引的过滤发生在索引扫描之后，条件命中率低时候选几乎被筛光，所以返回条数不够。
  • 漏召回分两类：条数不够，和条数够但排序不对。判断顺序永远是先看返回条数。
  • 迭代扫描只修第一类，严格顺序保序、宽松顺序召回更高，代价是延迟明显上升。
  • 预过滤是另一条路：过滤列建索引走精确检索、取值少建部分索引、取值多按值分区，代价依次是失去索引加速、索引数量爆炸、运维复杂度上升。
  • 第二类只能靠加大 probes 或 ef_search，迭代扫描对它完全无效。

  Key points

  • With approximate indexes the filter runs after the index scan, so a selective condition wipes out most candidates and the query returns too few rows.
  • There are two failure modes: too few rows, and enough rows in the wrong order. Always check the returned count first.
  • Iterative scanning fixes only the first. Strict ordering preserves distance order, relaxed ordering gives better recall, and both raise latency noticeably.
  • Pre-filtering is the alternative: index the filter column for exact search, use a partial index for a few distinct values, partition by value for many. Costs are losing the approximate speedup, index sprawl, and operational complexity.
  • The second failure mode is only fixed by raising probes or ef_search; iterative scanning does nothing for it.

D6 生成这一侧：上下文怎么排、引用怎么标、什么时候必须拒答，以及流式回答

• 知识库问答的拒答阈值怎么定？定高了和定低了各自的代价是什么？How do you set the refusal threshold for a knowledge-base assistant, and what does it cost you when the threshold is too high or too low?

  国内高频海外高频深入#refusal#thresholds#evaluation

  分析过程 · 先想清楚再作答

  1. 这题真正在考的是：你有没有意识到拒答不是一个阈值，而是好几条判据；以及你定阈值靠不靠数据。只谈一个分数阈值的回答，说明只做过最浅的一层。
  2. 先把拒答拆成三条线，它们的触发时机完全不同。检索分数太低：生成之前就能判，省一次模型调用。材料互相矛盾：也在生成之前判，代码在块之间找同一件事的不同数字，检出后要么并列两种说法与各自的更新日期，要么在有权威信号（比如一份点破了这条不一致的会议纪要）时按更新日期择一——选哪条是产品决策，但无论如何不能让模型自己悄悄挑一个。问题超出材料覆盖范围：只能在生成之后判，判据是跑完引用校验一条有效引用都没有。
  3. 强调三种话术必须不同。第一种要说「库里没有相关材料，换个说法或确认资料是否入库」，第三种要说「找到了相关文档但里面没有能直接回答的内容」——用户的下一步动作完全不同，混成一句「抱歉我不知道」等于把信息扔了。
  4. 再答代价这一半。定高了：能答的问题被挡在门外，用户看到查不到而材料其实在库里，这是最伤信任的一种错，而且它在日志里几乎不可见。定低了：低分噪声材料进上下文，模型拿着不相关的东西硬答，错误反而更隐蔽，因为回答看起来还带着引用。
  5. 怎么定：拿一批已知有答案和已知没答案的问题跑一遍，看两组的分数分布在哪里分开，按你更怕哪种错来取点。分数是没有绝对量纲的，换语料、换检索方式都要重定，所以真正要交付的是这套定阈值的流程，不是那个数字。
  6. 可预期的追问：单一分数阈值不够怎么办？答案是加判据而不是调数字——最高分与次高分的差、命中块数、以及生成后的引用校验结果，都是比原始分数更稳的信号。

  How to reason about it · think before answering

  1. What is really being tested: do you know that refusal is several rules rather than one threshold, and do you set thresholds from data. An answer that mentions only a score cutoff shows you have only touched the surface.
  2. Break refusal into three rules with different timing. Score too low: decidable before generation, saving a model call. Sources conflict: also decidable before generation, by finding differing numbers about the same thing across blocks. You then either present both with their update dates, or pick the newer one when an authoritative signal backs it, such as meeting notes that flagged the discrepancy. Which of the two is a product decision, but silently letting the model pick is never an option. Question outside coverage: only decidable after generation, when citation verification leaves you with zero verified citations.
  3. Stress that the three responses must read differently. 'Nothing relevant in the knowledge base, try rephrasing or check whether the document was ingested' is a different instruction to the user than 'we found related documents but none of them answers this'. Collapsing both into 'sorry, I don't know' throws away information.
  4. Then the cost half. Too high: answerable questions get blocked, the user is told nothing was found while the material is in fact indexed. That is the most trust-damaging failure and it is nearly invisible in logs. Too low: weak passages enter the context and the model answers from irrelevant material, which is worse because the answer still looks cited.
  5. How to set it: run a set of questions with known answers and known non-answers, look at where the two score distributions separate, and pick a point according to which error you fear more. Scores have no absolute scale, so the deliverable is the procedure, not the number.
  6. Expected follow-up: what if one score threshold is not enough? Add signals rather than tuning the number: the gap between top and second score, the number of hits above threshold, and the post-generation verification result are all steadier than the raw score.

  答题要点

  • 拒答不是一条线而是三条：分数过低、材料冲突（都在生成前判）、超出材料覆盖范围（只能生成后按引用校验结果判）。
  • 冲突检出后并列两说还是按更新日期择一，是产品决策；只有在有权威信号背书时择一才站得住，否则老实并列。
  • 三种情况的话术必须不同，因为它们给用户的下一步动作不同。
  • 定高了会把能答的问题挡住，用户看到查不到而材料其实在库里，最伤信任且日志里看不见。
  • 定低了会让噪声材料进上下文，错误更隐蔽，因为回答看起来仍然带着引用。
  • 定法是拿已知有答案与已知没答案的两组问题跑分数分布，按更怕哪种错取点；换语料或换检索方式都要重定。

  Key points

  • Refusal is three rules, not one: low score and source conflict decided before generation, out-of-coverage decided after generation from the verification result.
  • On conflict, presenting both versions versus picking the newer one is a product decision; picking only holds up when an authoritative signal backs it.
  • The three responses must be worded differently because each implies a different next action for the user.
  • Too high blocks answerable questions; the user is told nothing exists while it does, which is the most damaging and least visible failure.
  • Too low lets weak passages in, producing errors that are harder to spot because the answer still carries citations.
  • Set it by comparing score distributions over answerable and unanswerable question sets, then choose based on which error is worse; re-tune whenever the corpus or retriever changes.
• 流式输出的场景下，你怎么保证吐出去的内容不会因为引用校验失败而需要撤回？In a streaming setup, how do you make sure nothing you have already sent needs to be retracted because its citation failed verification?

  国内高频海外高频深入#streaming#citation-verification#api-design

  分析过程 · 先想清楚再作答

  1. 这题在考一个真实的架构矛盾：流式要尽早出字，引用校验要等话说完才能核对。看回答里有没有出现「取舍」两个字，以及有没有把代价说清楚。
  2. 先说清矛盾在哪：一旦一个 token 发到了浏览器就撤不回来，你在末尾才发现第三句引用是编的，那句话已经在用户屏幕上了，只能补一句「刚才那句请忽略」，体验比不流式还糟。
  3. 给方案：按句缓冲。攒够一个完整句子就立刻校验一次，通过了才把这句连同已核实的引用发出去，没通过就整句丢掉。代价是首字延迟从一个 token 变成一句话，通常两三百毫秒，用户几乎察觉不到，而错误引用一旦上屏赔的是信任。
  4. 补两个实现细节，它们能证明你写过：流式模式没法用 JSON 输出（要等右花括号闭合才能解析），所以改成纯文本加行内标记，但校验必须和非流式共用同一套；标记要从正文里剥掉，正文保持干净，编号单独走校验再作为结构化数据发出去。
  5. 再补一条顺序上的讲究：生成前就能判的两条拒答线（分数过低、材料冲突）要在流开始之前发出去，用户不会先看到半句回答再被收回；生成后才能判的那条，在按句缓冲之下表现为一句都没发出来，收尾补一个拒答事件即可。
  6. 可预期的追问：那用户体验上的流式感是不是就没了？没有，句级流式在中文长回答里仍然是明显的渐进呈现；真要更细，可以在句子发出前先流一个「正在核对」的占位态，但不要流未校验的正文。

  How to reason about it · think before answering

  1. This tests a real architectural conflict: streaming wants the first token out early, citation verification cannot run until a statement is complete. Listen for whether the candidate names the trade-off and prices it.
  2. Name the conflict: once a token reaches the browser you cannot take it back. Discovering at the end that the third sentence cited a fabricated block leaves you posting 'please ignore that last sentence', which is worse than not streaming at all.
  3. Give the solution: buffer by sentence. As soon as a complete sentence lands, verify it, and only then emit it together with its verified citations; drop the whole sentence otherwise. The cost is that time-to-first-token becomes time-to-first-sentence, typically a few hundred milliseconds, which users barely notice, whereas a bad citation on screen costs trust.
  4. Add two implementation details that prove you have built it. Streaming cannot use JSON output because JSON is only parseable once closed, so switch to plain text with inline markers, while keeping exactly the same verifier as the non-streaming path. Strip the markers out of the prose and send the numbers as structured data after verification.
  5. Add the ordering point: the two rules decidable before generation, low score and source conflict, should be emitted before the stream starts, so the user never sees half an answer being withdrawn. The rule that needs generation shows up as 'no sentence was ever emitted', so close the stream with a refusal event.
  6. Expected follow-up: does this kill the streaming feel? No. Sentence-level streaming is still visibly progressive on long answers. If you need finer granularity, stream a 'checking sources' placeholder, but never stream unverified prose.

  答题要点

  • 矛盾在于发出去的内容撤不回来，而引用只有一句说完才能核对。
  • 解法是按句缓冲：攒够一句校验一次，通过才发，没通过整句丢掉。
  • 代价是首字延迟从一个 token 变成一句话，这个代价必须付也付得起。
  • 流式用不了 JSON，改纯文本加行内标记，但校验逻辑与非流式共用同一套；标记从正文剥出，编号作为结构化数据单独发。
  • 生成前能判的拒答要在流开始之前发出去，生成后才能判的那条以「一句都没发」的形式收尾补事件。

  Key points

  • The conflict: emitted text cannot be recalled, while a citation can only be checked once its sentence is complete.
  • The fix is sentence-level buffering: verify each completed sentence, emit only if it passes, drop the whole sentence if it does not.
  • The cost is time-to-first-sentence instead of time-to-first-token, which is affordable and worth paying.
  • Streaming cannot use JSON, so use inline markers in plain text while sharing one verifier with the non-streaming path; strip markers from the prose and send numbers as structured data.
  • Emit pre-generation refusals before the stream opens; the post-generation one manifests as an empty stream and is closed with a refusal event.

D7 第一周综合：把六天的零件装成一个可一键启动的检索问答服务并复盘

• 你刚拼出来的这个检索问答系统，现在最大的风险在哪里？你打算怎么证明这个判断？What is the biggest risk in the RAG service you just assembled, and how would you prove that judgment?

  国内高频海外高频深入#evaluation#risk-assessment#retrospective

  分析过程 · 先想清楚再作答

  1. 这题有两半，后半句才是题眼。说出一个风险不难，难的是给出一个能证伪你自己判断的方法——答不出后半句，前半句就只是意见。
  2. 先排除两个常见的错误答案：说「幻觉」太笼统，没有指向任何可动的地方；说「延迟」则是把看得见的问题当成最大风险。
  3. 真正的最大风险是**没有评估**：切块大小、取几条、门槛定多少、两路怎么加权，全是拍出来的。它最重要的地方在于它让所有其他风险都无法验收——你连「改了之后变好还是变坏」都说不出口。
  4. 怎么证明：先从语料反向出一份带标准答案文档的问题集，刻意掺进无答案问题和需要跨文档的多跳问题；再实现召回率与排序指标，给当前配置跑出一个基线；然后把一个参数来回改两次，看指标动不动。如果指标对参数完全不敏感，说明是评估集有问题，不是系统没问题。
  5. 补一句成本口径：每一项优化都要同时报三笔账——指标涨了多少、延迟涨了多少、钱涨了多少。只报第一笔的结论不能用。
  6. 可预期的追问：评估集多大才够？先做二十题能覆盖主要问题类型的小集，用它挡住明显的退步；等真实用户问题攒起来，再按真实分布扩到几百题。一上来就追求规模，只会得到一堆自己出的、跟真实用法无关的题。

  How to reason about it · think before answering

  1. There are two halves here and the second is the real question. Naming a risk is easy; giving a method that could falsify your own claim is what separates answers from opinions.
  2. Rule out two common wrong answers: 'hallucination' is too vague to act on, and 'latency' mistakes a visible problem for the biggest one.
  3. The biggest risk is the absence of evaluation. Chunk size, top-k, thresholds and route weights were all guessed, and that makes every other risk unverifiable: you cannot even say whether a change helped.
  4. How to prove it: build a question set from the corpus with known answer documents, deliberately including unanswerable and multi-hop questions; implement recall and ranking metrics; produce a baseline for the current configuration; then move one parameter back and forth and watch whether the metrics move. If they do not move at all, the evaluation set is wrong, not the system.
  5. Add the accounting rule: every optimisation reports three numbers, metric gain, latency added and cost added. A claim with only the first is not usable.
  6. Expected follow-up: how large must the set be? Start with roughly twenty questions covering the main question types to catch obvious regressions, then grow toward the real distribution once you have actual user questions. Chasing size first only yields questions you invented yourself.

  答题要点

  • 最大的风险是没有评估：所有参数都是拍的，导致任何改动的好坏都无法判断。
  • 证明方式是先建标准答案集，刻意包含无答案问题与多跳问题，再跑出当前配置的基线。
  • 用参数扰动反过来验证评估集本身：指标对参数完全不敏感，说明题出得有问题。
  • 每项优化同时报三笔账：指标、延迟、成本；只报指标的结论不能用。
  • 评估集先小而全，覆盖问题类型即可，等真实问题攒起来再按真实分布扩大。

  Key points

  • The biggest risk is having no evaluation: every parameter was guessed, so no change can be judged.
  • Prove it by building a golden set with known answer documents, including unanswerable and multi-hop questions, then baseline the current configuration.
  • Validate the set itself by perturbing parameters: metrics that never move mean the questions are wrong.
  • Report three numbers per optimisation: metric gain, added latency, added cost.
  • Start small but well covered, then grow toward the real question distribution.

D8 评估先行：搭 golden set、算召回与排序指标、用模型当裁判判忠实度

• 用模型当裁判来评 RAG 的忠实度，有哪些系统性偏差？你怎么发现它们、又怎么证明你的裁判可信？What systematic biases does an LLM judge have when scoring RAG faithfulness, and how do you detect them and prove your judge is trustworthy?

  国内高频海外高频深入#llm-as-judge#evaluation#faithfulness

  分析过程 · 先想清楚再作答

  1. 这题的题眼在后半句。能背出「位置偏好、长度偏好、自我偏好」三个名词的人很多，能说出「怎么证明可信」的很少——面试官要的是后者。
  2. 先把三个偏差和各自的缓解手段一一对应：位置偏好用逐条独立打分代替两两比较，非要比较就交换顺序跑两遍、结论不一致判平局；长度偏好用逐句判定加比例计分，写得越长分母越大，长度红利自动消失；自我偏好用跨供应商或跨档位的模型评判，生成和评判不同源。
  3. 再补两条提示词层面的：给死评分锚点，1.0 / 0.6 / 0.3 各自是什么必须写明，否则同一份输入不同天给的分都不一样；强制结构化输出并要求把没支撑的句子原样列出，这是人工复核的抓手。
  4. 证明可信只有一条路：人工抽检算一致率。分层抽十到三十条——各类型都要有、命中和没命中都要有、裁判给高分和低分都要有，只判一个二元问题（有没有材料外的内容），跟裁判的结论比对。低于 0.8 就不能拿它的分数做拦合并这类决策。
  5. 一个能加分的细节：一致率很高不一定是好消息。如果抽的十条都是「答案原样抄自材料」的简单题，判对是理所当然的，这时候 100% 说明的是抽检没难度，不是裁判可靠。
  6. 可预期的追问是「裁判本身会不会坏」。答案是给裁判写探针：喂几组已知正确答案的输入（照抄材料的、明显编造的），每次跑评估都验一遍——评估系统坏掉的方式最阴险，分数照常输出，只是不再有意义。

  How to reason about it · think before answering

  1. The second half of the question is the discriminator. Plenty of people can name position, length, and self-preference bias; few can say how they prove the judge is trustworthy.
  2. Pair each bias with its mitigation: position bias - score pointwise instead of pairwise, and if you must compare, swap the order and call disagreement a tie; length bias - decompose into claims and score a ratio, so a longer answer grows its own denominator; self-preference - judge with a different vendor or tier than the generator.
  3. Add two prompt-level requirements: fixed rubric anchors (spell out what 1.0, 0.6 and 0.3 mean, or the same input scores differently on different days) and forced structured output that quotes the unsupported sentences verbatim, which is what makes human review possible.
  4. Proving trust has exactly one route: human spot-checks and an agreement rate. Stratify ten to thirty items across types, hits and misses, high and low judge scores; answer one binary question only - is anything here not in the material - and compare. Below 0.8 the judge's scores cannot gate a merge.
  5. A detail that scores points: a very high agreement rate may mean your spot-check was too easy. If all ten sampled answers copy the material verbatim, agreeing is trivial and 100% says nothing about the judge.
  6. Expected follow-up: can the judge itself break? Add probes - fixed inputs with known verdicts, one faithful and one obviously fabricated, checked on every run. An evaluation system fails silently: the numbers keep coming, they just stop meaning anything.

  答题要点

  • 三个偏差：位置偏好、偏爱长答案、自己评自己，各自有对应的缓解手段。
  • 逐条独立打分代替两两比较；逐句判定按比例计分抵消长度红利；生成与评判不同源。
  • 提示词要给死评分锚点，并强制结构化输出、列出没支撑的句子。
  • 可信度靠人工分层抽检算一致率，低于 0.8 不能用它做拦合并的决策。
  • 给裁判本身写探针，每次跑评估都验一遍它有没有坏。

  Key points

  • Three biases: position, verbosity, and self-preference, each with a matching mitigation.
  • Score pointwise rather than pairwise; decompose into claims and score a ratio to kill the length premium; never let the generator judge itself.
  • Pin rubric anchors in the prompt and force structured output that quotes unsupported sentences.
  • Establish trust through stratified human spot-checks and an agreement rate; below 0.8 the judge cannot gate merges.
  • Add probes with known verdicts so a broken judge is caught on every run.

D9 混合检索与重排：两路召回、倒数排名融合，再用交叉编码器把前几名重新排一遍

• 你把纯向量检索换成了混合检索加重排，上线之后评估指标反而掉了。你会怎么排查？You replaced pure vector retrieval with hybrid search plus reranking, and after shipping it your eval metrics went down. How do you investigate?

  国内高频海外高频深入#hybrid-search#evaluation

  分析过程 · 先想清楚再作答

  1. 这题考的是你有没有真的做过分阶段归因。答「调一下权重再看看」就输了——那是在猜，不是在查。
  2. 第一步是拆档跑，不是改代码：纯关键词、纯向量、混合、混合加重排四档在**同一份评估集、同一个上下文预算**下各跑一遍。指标掉在哪一档就在哪一档找原因，这一步能立刻区分「融合坏了」和「重排坏了」。
  3. 第二步问一个具体问题：掉的是召回率还是排序指标？召回率掉说明答案根本没进上下文，是候选池或者预算的问题；排序指标掉而召回率没动，说明答案还在、只是被挤到了后面，那是融合权重或重排模型的问题。这两类故障的解法完全不同。
  4. 第三个常见根因是召回深度。每路取多少条这个旋钮方向反直觉：取深了不是更保险，是把噪声也一起投了票。我在一份 134 块的语料上实测过，每路从取 50 收到取 5，混合那一档的召回率从 87.5% 回到 93.8%、多跳档从 50% 回到 75%，而 nDCG 反而掉了近 0.1——两个指标会打架，先想清楚业务要哪个。
  5. 第四个根因是评估口径被悄悄改了。上下文预算、命中判定、候选池深度只要动过一个，新旧数字就不可比，这时候「掉了」可能根本不是真的掉了。
  6. 可预期的追问：怎么防止下次再踩？答把四档对照做成一条命令、把上一版报告存成基线、指标退步就以非 0 退出码拦住合并——这就是评估要先于优化的原因。

  How to reason about it · think before answering

  1. This question tests whether you have actually done stage-by-stage attribution. Answering `I would tune the weights and see` loses — that is guessing, not investigating.
  2. Step one is to run the stages apart, not to change code: pure keyword, pure vector, hybrid, and hybrid plus rerank, all on the **same eval set with the same context budget**. Whichever stage the drop appears in is where you look, and this alone separates `fusion is broken` from `reranking is broken`.
  3. Step two asks a specific question: did recall drop, or did the ranking metrics drop? A recall drop means the answer never entered the context at all — a candidate-pool or budget problem. Ranking metrics dropping while recall holds means the answer is still there but pushed down — a fusion-weight or rerank-model problem. The two failures have completely different fixes.
  4. A third common root cause is recall depth. This knob runs against intuition: going deeper is not safer, it lets noise vote too. On a 134-chunk corpus I measured that narrowing each route from 50 to 5 took hybrid recall from 87.5% back to 93.8% and multi-hop from 50% to 75%, while nDCG fell by almost 0.1. The metrics fight each other, so decide which one the product needs first.
  5. A fourth root cause is that the eval protocol quietly changed. Touch the context budget, the hit rule, or the candidate depth, and the old and new numbers stop being comparable — in which case the `drop` may not be a drop at all.
  6. Expected follow-up: how do you avoid this next time? Make the four-way comparison a single command, store the previous report as a baseline, and fail the build with a non-zero exit code on regression. That is precisely why evaluation comes before optimization.

  答题要点

  • 先拆档跑四种配置，在同一份评估集和同一个上下文预算下归因，不要一上来就调参。
  • 区分召回率掉与排序指标掉：前者是候选池或预算问题，后者是融合或重排问题。
  • 查召回深度：每路取太深会把噪声也投进融合，收窄反而可能救回召回率。
  • 确认评估口径没被改：预算、命中判定、候选池深度动过一个，新旧数字就不可比。
  • 把四档对照固化成一条命令加一份基线报告，指标退步直接拦住合并。

  Key points

  • Run all four configurations separately for attribution, on one eval set with one context budget, before touching any parameter.
  • Separate a recall drop from a ranking drop: the first is a candidate-pool or budget issue, the second is a fusion or rerank issue.
  • Check recall depth: taking too many per route lets noise vote, and narrowing it can bring recall back.
  • Confirm the eval protocol did not change; touching budget, hit rule, or candidate depth makes old and new numbers incomparable.
  • Freeze the four-way comparison into one command plus a baseline report, and block merges on regression.
• 加上重排之后每次提问多了两百毫秒延迟，还多了一笔按次计费的开销。你怎么判断这笔钱该不该付？Adding a reranker costs you 200 ms of extra latency per question plus a per-search fee. How do you decide whether that spend is worth it?

  国内高频海外高频深入#rerank#cost-tradeoff

  分析过程 · 先想清楚再作答

  1. 这题考的是你会不会把技术选择翻译成业务判断。只答「看指标涨没涨」只答了三分之一。
  2. 先把账拆成三笔：指标涨了多少、延迟涨了多少、钱涨了多少。三笔必须一起报，只报第一笔的方案在评审会上过不去。
  3. 第一笔要问清楚重排改善的是**哪个**指标。重排改的是顺序，不是候选集合——它救不了「答案压根没被召回」这种故障。如果你的召回率本来就不够，先去加召回路数或者调召回深度，重排这两百毫秒是白花的。
  4. 第二笔要看这两百毫秒落在哪。它是同步卡在检索之后、生成之前的，用户全程在等；但如果后面接的是一个流式生成、首字节本来就要一两秒，这两百毫秒的相对占比就小得多。反过来，如果这是一个自动补全式的即时搜索框，两百毫秒就是致命的。
  5. 第三笔要注意计价单位：重排普遍按检索次数计价而不是按 token，所以「多送几条给它排」几乎不涨钱，真正贵的是提问次数本身。这直接决定了优化方向是压提问量（缓存、意图路由）而不是压候选数。
  6. 可预期的追问：如果就是付不起怎么办？答三条路——只对判定为复杂的查询走重排（意图路由）、把结果缓存起来、或者换成自部署的开源交叉编码器把按次付费变成固定的算力成本。

  How to reason about it · think before answering

  1. This question tests whether you can translate a technical choice into a business judgment. Answering `check whether the metrics went up` covers only a third of it.
  2. Split it into three ledgers: how much the metrics moved, how much latency grew, and how much money it costs. All three must be reported together; a proposal with only the first will not survive review.
  3. For the first ledger, be specific about **which** metric reranking improves. Reranking changes the order, not the candidate set — it cannot fix `the answer was never retrieved`. If your recall is the bottleneck, add a retrieval route or adjust recall depth first; the 200 ms buys nothing.
  4. For the second, ask where those 200 ms land. They sit synchronously between retrieval and generation, with the user waiting; but if a streaming generation follows and time-to-first-token is already a second or two, the relative cost is small. In an as-you-type search box, 200 ms is fatal.
  5. For the third, note the billing unit: rerankers usually charge per search rather than per token, so sending a few more candidates barely changes the bill — what is expensive is the number of questions. That points optimization at reducing query volume (caching, intent routing) rather than at trimming the candidate list.
  6. Expected follow-up: what if you simply cannot afford it? Three paths — rerank only queries classified as hard (intent routing), cache results, or self-host an open-weights cross-encoder to convert per-call fees into fixed compute cost.

  答题要点

  • 三笔账一起报：指标增量、延迟增量、成本增量，缺一笔方案就不完整。
  • 先确认瓶颈是排序还是召回：重排只改顺序，救不了没被召回的答案。
  • 延迟要看落在哪：流式生成场景下相对占比小，即时搜索框里两百毫秒就是致命的。
  • 重排按检索次数计价而不是按 token，涨钱的是提问量而不是候选条数。
  • 付不起时的三条路：意图路由只对难查询重排、结果缓存、换自部署的开源交叉编码器。

  Key points

  • Report all three ledgers together: metric gain, latency growth, cost growth; a proposal missing one is incomplete.
  • Confirm whether the bottleneck is ordering or recall first; reranking only reorders and cannot rescue an answer that was never retrieved.
  • Judge the latency by where it lands: it is small relative to a streaming generation, but fatal in an as-you-type search box.
  • Rerankers bill per search rather than per token, so cost scales with question volume, not candidate count.
  • If it is unaffordable: route only hard queries to the reranker, cache results, or self-host an open-weights cross-encoder.

D10 查询侧优化：改写、假设文档嵌入、多路查询、后退提问与意图路由

• 上线查询改写之后每问多了一次模型调用，端到端延迟涨了一倍，你怎么判断这笔开销值不值？Query rewriting adds a model call per question and doubles end-to-end latency. How do you decide whether it is worth paying?

  国内高频海外高频深入#cost-tradeoff#latency#query-rewriting

  分析过程 · 先想清楚再作答

  1. 这题考的是「能不能把工程判断落到数字上」。凡是回答「改写当然要做，能提升效果」的，一律判为没做过——他连收益是多少都没量。
  2. 先问自己一句：**收益出现在哪一类流量上**。这一条几乎决定了答案。查询改写在单轮问答上的收益接近零（我们在 20 道单轮题上实测开关它指标一模一样），收益全在多轮追问。所以第一步是去线上日志里查多轮会话占比，占比很低的话这笔钱不该花在全量流量上。
  3. 第二步是把三笔账摆齐，缺一笔就不能下判断：指标涨了多少（用固定的标准答案集跑，不要用感觉）、延迟涨了多少（改写是短输出任务，可以换便宜快的那一档模型，往往只多两三百毫秒而不是翻倍）、多了几次调用（改写是一次，多路查询是一次调用加几次检索，成本结构完全不同，别混着算）。
  4. 第三步是找**便宜的替代路径**再比一次：只在命中触发条件时才改写（短问题、含指代词、含省略），纯新话题直接跳过；改写结果按会话缓存；用小模型跑改写而不是主模型。这三招通常能把这笔开销压掉一大半，而收益几乎不掉。
  5. 结论要落成一条可执行的判据：**收益乘以受影响流量占比，除以增加的延迟与成本**，跟你手上其他候选优化排个序。改写通常能排到很前面，因为它的失败方式是「用户明明追问同一件事却被告知查不到」，那是会直接导致弃用的体验故障，不只是指标掉几个点。
  6. 可预期的追问是「延迟真的不能接受怎么办」。答：把改写和第一次检索**并行发**，用原查询先检索一路，改写回来后再补一路，两路用倒数排名融合合起来——延迟只多一个 max 而不是一个加法，代价是多一次检索。

  How to reason about it · think before answering

  1. This question is about turning an engineering judgement into numbers. "Rewriting obviously helps quality" is a fail — the candidate never measured the gain.
  2. Start with one question that nearly settles it: which slice of traffic does the gain land on? Query rewriting buys almost nothing on single-turn questions (we measured identical metrics with it on and off across 20 single-turn items); the entire payoff is in follow-up turns. So step one is to pull the share of multi-turn sessions from production logs.
  3. Step two is to lay out all three ledgers, because one alone cannot support a decision: how much the metrics moved on a fixed golden set, how much latency grew (a rewrite is a short-output task, so a cheap fast model often costs a few hundred milliseconds rather than doubling anything), and how many extra calls were added — one model call for rewriting versus one call plus several retrievals for multi-query is a completely different cost shape.
  4. Step three is to price the cheaper variants before deciding: rewrite only when a trigger fires (short query, pronoun, ellipsis), cache rewrites per session, and run a small model instead of the main one. These usually remove most of the cost while keeping the gain.
  5. Land on a usable rule: gain times affected traffic share, divided by added latency and cost, ranked against your other candidate optimisations. Rewriting usually ranks high because its failure mode is a false "not found" immediately after the user's own question — an abandonment-grade experience bug, not a few metric points.
  6. Expect "what if the latency genuinely is unacceptable?" Fire the rewrite and the first retrieval in parallel: search with the raw query immediately, search again when the rewrite returns, and fuse both rankings. You pay a max instead of a sum, at the cost of one extra retrieval.

  答题要点

  • 先定位收益落在哪一类流量：改写在单轮上接近零收益，价值全在多轮追问，先查多轮会话占比。
  • 三笔账缺一不可：标准答案集上的指标变化、增加的延迟、增加的调用次数与 token 成本。
  • 先试便宜的替代路径：条件触发、按会话缓存、用小模型跑改写，通常能压掉大半开销。
  • 判据是「收益 × 受影响流量占比 ÷ 增加的延迟与成本」，再和其他候选优化排序。
  • 延迟真的卡死时，把改写与首次检索并行发，两路名次用倒数排名融合，延迟从加法变成取最大值。

  Key points

  • Locate the gain first: rewriting is near-zero on single-turn traffic and pays off on follow-ups, so start from the share of multi-turn sessions.
  • All three ledgers are mandatory: metric delta on a golden set, added latency, added calls and token cost.
  • Try the cheap variants before deciding: conditional triggering, per-session caching, and a small model for the rewrite.
  • Decide on gain times affected traffic share over added latency and cost, then rank it against your other optimisations.
  • If latency is a hard constraint, fire the rewrite in parallel with the first retrieval and fuse both rankings, turning a sum into a max.

D11 高级索引：父子文档、摘要索引、上下文检索，以及树状聚合与图检索的取舍

• 上下文检索要给每个块调一次模型，这笔一次性成本怎么估？有哪些办法能压下来？Contextual retrieval needs one model call per chunk. How do you estimate that one-off cost, and what levers bring it down?

  国内高频海外高频深入#contextual-retrieval#prompt-caching#cost

  分析过程 · 先想清楚再作答

  1. 这题考的是你有没有真的算过账。只会说『用提示词缓存就便宜了』属于听过没做过——面试官会追问缓存到底省在哪一项上。
  2. 先把成本拆开：一次性 = 每块的输入 + 输出 + 全量 embedding；每次查询 = 块头在重排和上下文里各被读一遍。**这两笔要分开记**，因为它们随业务量的增长方式完全不同。
  3. 一次性那笔的主项是『同一篇文档被重复读了多少遍』。一篇切成 n 块就要读 n 遍，这是成本的大头。提示词缓存省的正是这一项：把整篇放在提示词最前面并标记为可缓存，第一块付一次缓存写入，后面 n-1 块只付缓存读取，而读取价通常比输入价低一个数量级。
  4. 顺序不能反：缓存按前缀匹配，整篇必须在前、块内容在后。把变化的块放前面，前缀次次都变，缓存一次都不会命中——这是最常见的翻车点。
  5. 我们的实测：30 篇、134 块，不开缓存输入 103017 token，开缓存后拆成写入 17340 加读取 60137，一次性成本降约 29%。**块切得越碎这个比例越高**，因为重复读的次数更多。
  6. 结论反直觉但很实用：一次性那笔是小钱，摊到 217 次查询就降到每次查询成本的一成以下；真正的长期账是每次查询多出来的那几十个 token（我们量到 +12.3%）。所以压成本的第一优先级不是压建索引，而是让块头别进上下文、别过长、别对全库无差别地生成。
  7. 最后一条是加分项：花这笔钱之前先确认你的评估环境**测得出**收益。我们的离线环境里向量路对召回的独立贡献实测为 0，所以它根本没法回答『块头对向量侧有没有用』——在这种环境里做的 A/B 会给你一个看起来有数字支撑的错误结论。

  How to reason about it · think before answering

  1. This checks whether you have actually done the arithmetic. Saying 'prompt caching makes it cheap' without knowing which line item it touches is a tell.
  2. Split the bill first: one-off = per-chunk input + output + full re-embedding; per-query = the header read twice, once by the reranker and once in the context. Keep them separate, because they scale with completely different things.
  3. The dominant term on the one-off side is how many times the same document is re-read. A doc split into n chunks is read n times. Prompt caching attacks exactly that: put the whole document first and mark it cacheable, pay a cache write once, then cache reads for the remaining n-1, typically an order of magnitude cheaper than input.
  4. Order matters. Caching is prefix-matched, so the document must come first and the chunk after. Put the varying part first and the prefix changes every call — zero cache hits. This is the most common way people get it wrong.
  5. Our measurement: 30 docs, 134 chunks. Without caching, 103017 input tokens; with caching, 17340 written plus 60137 read, cutting the one-off cost by roughly 29%. The finer the chunks, the bigger the saving, because re-reads multiply.
  6. The counter-intuitive part is the useful part: the one-off cost amortizes below 10% of per-query cost after about 217 queries. The lasting bill is the extra tokens every query carries (we measured +12.3%). So the first lever is not cheaper index building — it is keeping the header out of the context, keeping it short, and not generating it for the whole corpus indiscriminately.
  7. A bonus point: before spending any of it, confirm your evaluation setup can actually detect the benefit. In our offline harness the vector route contributed exactly zero unique answer documents, so it cannot answer whether headers help embeddings at all — an A/B run there hands you a wrong conclusion that looks numerically supported.

  答题要点

  • 把账拆成一次性（每块的输入输出 + 全量 embedding）和每次查询（块头在重排与上下文里各读一遍）两笔。
  • 一次性的大头是同一篇被重复读 n 遍；提示词缓存把它压成一次写入加 n-1 次读取。
  • 缓存按前缀匹配，整篇必须放在提示词最前面，块内容在后，顺序反了一次都不会命中。
  • 实测 30 篇 134 块，一次性成本降约 29%，块越碎省得越多。
  • 长期账在每次查询：块头别进上下文、控制长度、只对真正需要的文档生成。

  Key points

  • Split into one-off (per-chunk input/output plus re-embedding) and per-query (header read by both reranker and generator).
  • The one-off is dominated by re-reading each document n times; caching turns that into one write plus n-1 reads.
  • Caching is prefix-matched: the full document must come first, the chunk after, or you get zero hits.
  • Measured on 30 docs / 134 chunks, caching cut the one-off cost by about 29%, and finer chunks save more.
  • The lasting cost is per query: keep headers out of the context window, keep them short, and generate them selectively.
• 什么样的问题必须上图检索？给一个该上的具体例子和一个不该上的例子。What kind of question actually requires graph retrieval? Give one concrete case where it is justified and one where it is not.

  国内高频海外高频深入#graph-rag#multi-hop#cost

  分析过程 · 先想清楚再作答

  1. 这题在考你会不会为了用而用。只要答案里出现『多跳问题就要上图检索』，面试官基本就知道你没落地过——多跳只是必要条件，远不是充分条件。
  2. 判据要落在一个可观察的现象上：**答案的第二篇文档和查询之间，有没有字面或语义上的重合**。有重合，普通的混合检索就能捞到它，多跳是假的；完全没有重合，只能靠一条关系边走过去，这才是图检索的领地。
  3. 该上的例子：问『生产库主备切换必须谁书面审批、这个人叫什么』。一篇写着须平台组组长审批，另一篇写着平台组组长是某人。第二篇跟查询一个词都不重合，我们在五种索引结构下测了一遍，它在 20 条候选池里一次都没出现过——换切法、加块头、父子回填全都无效。
  4. 不该上的例子：问『扩容要走哪个流程、最晚提前几个工作日提单』。同样跨两篇文档，但两篇都跟查询有明显字面重合，混合检索把它们分别排在第 2 名，一次检索就凑齐了。为它建图是拿几倍成本买一个已经解决的问题。
  5. 然后说代价，这一段决定了你像不像做过：建图不止一次抽取调用，实体要消歧、关系要去重、文档更新时受影响的子图要重算，还要多维护一套图存储和一套更新链路。
  6. 可预期的追问是『不上图检索还有什么办法』。答案是把多跳交给 Agentic 检索：让模型先查出中间实体，再拿这个实体发起第二次检索。它的一次性成本几乎为零，代价换成了每次查询的延迟与调用次数——先试这条，试不通再考虑建图。

  How to reason about it · think before answering

  1. This one tests whether you reach for tools you don't need. If the answer is 'multi-hop questions need a graph', the interviewer knows you haven't shipped one — multi-hop is necessary, nowhere near sufficient.
  2. Anchor the criterion on something observable: does the second required document share any lexical or semantic overlap with the query? If it does, ordinary hybrid retrieval will surface it and the hop is illusory. If it shares nothing, only a relation edge gets you there — that is graph territory.
  3. Justified case: 'who must sign off on a production failover, and what is that person's name?' One doc says the platform lead must approve; another says who the platform lead is. The second shares not one term with the query. Across all five index structures we tested, it never once appeared in a 20-item candidate pool — rechunking, headers and parent backfill all failed.
  4. Unjustified case: 'which process covers a capacity change, and how many working days ahead must the ticket be filed?' Also two documents, but both overlap the query lexically; hybrid retrieval ranked them second each, and one pass collected both. Building a graph for this buys a solved problem at several times the cost.
  5. Then state the cost, which is what makes the answer sound operational: graph building is not one extraction call. Entities need disambiguation, relations need dedup, updates force recomputing affected subgraphs, and you now run a graph store and its update pipeline.
  6. Expect 'what else could you do instead'. Hand multi-hop to agentic retrieval: let the model retrieve the intermediate entity first, then issue a second query with it. Near-zero build cost, paid back in latency and call count per query. Try that before you build a graph.

  答题要点

  • 判据不是『是不是多跳』，而是『第二篇文档跟查询有没有字面或语义重合』——没有重合才轮得到图检索。
  • 该上：审批人那类问题，中间实体是唯一的桥，第二篇文档在候选池里一次都不出现。
  • 不该上：两篇都跟查询有重合的多跳题，混合检索一次就能凑齐。
  • 建图的真实成本是实体消歧、关系去重、增量重算和一套额外的图存储，不是一次抽取调用。
  • 先试 Agentic 检索的两次查询，走不通再考虑建图。

  Key points

  • The test is not 'is it multi-hop' but 'does the second document overlap the query at all' — only zero overlap earns a graph.
  • Justified: the approver question, where an intermediate entity is the only bridge and the second doc never enters the candidate pool.
  • Not justified: a multi-hop question whose documents both overlap the query — hybrid retrieval collects them in one pass.
  • Real graph cost is entity disambiguation, relation dedup, incremental subgraph recomputation and a whole extra store — not a single extraction call.
  • Try two-pass agentic retrieval first; build the graph only when that fails.

D12 Agentic RAG：把检索做成工具，让模型自己决定查不查、查几次、要不要推翻重来

• 多跳检索里第一跳查错了，后面全跟着错。你会怎么设计容错？In multi-hop retrieval a wrong first hop poisons every hop after it. How would you design for that?

  国内高频海外高频深入#multi-hop#error-propagation#agentic-rag

  分析过程 · 先想清楚再作答

  1. 这题考的是错误传播意识。只答「加个重试」是不够的——重试只在「同一条路走不通」时有用，而多跳的问题是走上了错误的路还越走越远。
  2. 先把两类失败分开，这是整题的骨架：一类是根本没捞到（答案文档在候选池里一次都没出现，放宽门槛毫无用处，只能靠新的查询词重查，也就是多跳），一类是捞到了却被自己的过滤器扔了（排在第二名但没过准入门槛，这一类跳多少跳都没用，只能降级放宽门槛重判）。判据是要素覆盖率加答案槽位：覆盖率低是方向错了走放宽，覆盖率高但槽位空是只查到半路走多跳。判错类型，容错就完全用反了。
  3. 然后给具体机制。判断下一跳查什么，不能凭模型自由发挥，要有可解释的判据：从最贴题的那一句里挑出问题没提过、且在别的文档里也出现过的具体名词当作桥接短语。「问题没提过」保证它是新信息，「别的文档里也有」保证真的有下一跳可跳——只在这一篇里出现的短语，查了只会把同一篇再捞回来。
  4. 结论层面，容错有三层：不要丢掉上一跳的材料（第二跳查错了，第一跳的证据还在）；每一跳独立记录轨迹，事后能定位是哪一跳歪的；追不动时主动认输，把「材料不足」交给生成侧的拒答，而不是硬凑一个答案。
  5. 有一个非常容易被忽略的坑，说出来会加分：跳成功了，命中却没变。第二跳确实把目标文档检索回来了，但如果把两跳的候选混在一起按名次装上下文，第一跳的材料字面重合度更高，会把预算占满，目标文档根本挤不进去。上下文预算必须按跳轮转分配——多跳的收益是在这一步兑现的，不是在检索那一步。
  6. 可预期的追问是「怎么知道第一跳错了」。答案是靠自评的结构化输出，而不是靠最终答案对不对；等到答案错了再回头找，链路已经断了三跳，定位成本高得多。

  How to reason about it · think before answering

  1. This is about error propagation. 'Add a retry' is not enough - retries help when one path fails, but the multi-hop problem is walking confidently down the wrong path.
  2. Separate two failure kinds first, because the fixes are opposite. Either the answer document never entered the candidate pool - no amount of loosening helps, only a new query term does, which is the multi-hop path - or it was retrieved and then dropped by your own admission threshold, where extra hops are useless and only relaxing the gate recovers it. Coverage plus the answer slot tells them apart: low coverage means wrong direction (broaden), high coverage with an empty slot means halfway there (hop).
  3. Then give the mechanism. Do not let the model freestyle the next query: pick a bridge phrase from the sentence that best matches the question - a concrete noun the question never mentioned that also appears in another document. 'Not in the question' makes it new information; 'appears elsewhere' guarantees there is somewhere to hop to.
  4. Fault tolerance has three layers: keep the earlier hop's material, so a bad second hop does not destroy the evidence you already had; trace each hop separately so you can locate where it went wrong; and surrender explicitly when there is no lead left, handing 'insufficient evidence' to the generation-side refusal.
  5. The overlooked trap is worth points: the hop succeeds but the metric does not move. The second hop really did retrieve the target document, yet if you merge both hops' candidates and pack by score, the first hop's higher lexical overlap fills the budget and the target never enters the context. Allocate the context budget round-robin across hops - that is where multi-hop gains are actually realized.
  6. Expected follow-up: how do you know the first hop was wrong? From the structured self-assessment, not from the final answer. By the time the answer is wrong the chain is three hops deep and much more expensive to debug.

  答题要点

  • 先分类：根本没捞到只能靠多跳换查询词，捞到了被门槛扔了只能靠降级放宽，两者修法相反。
  • 下一跳的查询用桥接短语：问题没提过、且别的文档里也出现过的具体名词。
  • 保留上一跳的材料，第二跳失败时第一跳的证据仍在。
  • 每一跳独立记轨迹，能定位是哪一跳歪的。
  • 上下文预算按跳轮转分配，否则跳成功了命中也不会变。
  • 追不动时主动认输，把材料不足交给生成侧拒答，不硬凑答案。

  Key points

  • Classify first: never retrieved needs a new query term (a hop); retrieved-then-filtered needs a relaxed gate. The fixes are opposite.
  • Derive the next query from a bridge phrase - a concrete noun absent from the question that also appears in another document.
  • Keep the previous hop's material so a failed hop does not discard existing evidence.
  • Trace every hop separately so you can pinpoint which one drifted.
  • Allocate context budget round-robin across hops, or a successful hop still fails to change the metric.
  • Surrender explicitly when no lead remains and hand it to the generation-side refusal.

D13 上生产：增量同步与去重、按权限过滤、缓存分层、链路追踪与成本延迟账

• 为什么权限过滤不能放在生成阶段做？放在那里会泄露什么？Why can't access control be applied at the generation step? What exactly leaks if you put it there?

  国内高频海外高频深入#access-control#filter-pushdown#multi-tenancy

  分析过程 · 先想清楚再作答

  1. 这题在考你有没有真的把 RAG 当系统看。答成「因为不安全」拿不到分，面试官要的是「具体泄露了什么」和「除了泄露还有什么后果」两件事。
  2. 先用一个画面把位置说清楚：档案管理员先把全部档案摊在桌上让你挑，你挑完他再抽走三份说这些不能看——你已经看见标题了。在生成阶段过滤就是这个动作。
  3. 然后拆后果，两条，第二条更能显出做过工程：一是**泄露面**，越权文档已经进过检索、参与过排序、被进程读进过内存、大概率写进了检索日志和链路追踪，哪怕最终答案里没有它的内容；二是**结果被稀释**，取前 8 条里有 3 条不该看，筛掉只剩 5 条，而本该补位的第 9、10 名合法结果永远没机会上来——用户体感是「查不到」，你的日志里却是一次正常检索。
  4. 给正确做法：把权限谓词和排序、LIMIT 写进同一条查询，数据库先裁行再排序取前 k，越权的行一次都没被比较过。两种落法要都讲：行级过滤是一份索引加一个谓词，索引隔离是按边界各建各的索引。
  5. 选型判据要给出来：看隔离边界的数量和稳定性。部门这种个位数且几乎不变的边界，隔离划算；几万个用户各自的私有文档就只能行级过滤，否则运维扛不住。补一句共用索引的副作用——数据量大的租户会拖慢别人的检索质量，因为候选名额是共享的。
  6. 可预期的追问：缓存怎么办？这是同一个问题的第二现场——答案缓存的 key 里必须带上权限范围，否则一个用户的答案会被另一个用户命中，而且这条泄露路径连检索日志都不会留下痕迹。

  How to reason about it · think before answering

  1. This checks whether you think about RAG as a system. 'Because it's insecure' scores nothing; the interviewer wants what specifically leaks, and what else goes wrong besides the leak.
  2. Anchor the position with an image: the archivist spreads every file on the table, you pick nine, and only then does he pull three back saying you may not read those. You have already seen the titles. Filtering at generation time is that gesture.
  3. Then split the consequences, and note the second one is what shows engineering experience. First, exposure: the unauthorized documents were retrieved, ranked, read into process memory, and almost certainly written to retrieval logs and traces, even if none of their text reaches the answer. Second, dilution: you take the top 8, three are off-limits, the user gets five, and the legitimate results ranked ninth and tenth never get promoted. The user experiences 'it can't find anything' while your logs show a perfectly normal retrieval.
  4. State the fix: put the permission predicate in the same query as the ordering and the LIMIT, so the database prunes rows before ranking and unauthorized vectors are never compared. Cover both shapes: row-level filtering is one index plus a predicate; index isolation is a separate index per boundary.
  5. Give the selection criterion: the number and stability of the isolation boundaries. A handful of departments that rarely change makes isolation worthwhile; tens of thousands of per-user private document sets leave you with row-level filtering, because that many indexes is unmanageable. Add the shared-index side effect: a large tenant degrades everyone else's retrieval quality because candidate slots are shared.
  6. Expected follow-up: what about caching? It is the same bug's second crime scene. The answer cache key must include the permission scope, or one user's answer will be served to another, and that leak leaves no trace in the retrieval log at all.

  答题要点

  • 在生成阶段过滤时，越权文档已经被检索、排序、读进内存并写进日志与追踪，泄露面比「答案里有没有」大得多。
  • 第二个后果是结果被稀释：筛掉之后名额空着不补，用户体感是查不到，日志里却是一次正常检索。
  • 正确做法是把权限谓词和排序、LIMIT 写进同一条查询，让数据库先裁行再排序取前 k。
  • 行级过滤与索引隔离的选型判据是隔离边界的数量与稳定性；共用索引时大租户会挤占小租户的候选名额。
  • 缓存是同一个漏洞的第二现场：答案缓存的 key 必须包含权限范围，否则会跨用户串答案且不留痕迹。

  Key points

  • Filtering at generation time means unauthorized documents were already retrieved, ranked, held in memory and written to logs and traces; the exposure is far wider than 'did the text reach the answer'.
  • The second consequence is dilution: filtered-out slots are not backfilled, so users see 'nothing found' while the log shows a normal retrieval.
  • The fix is to put the permission predicate in the same statement as ordering and LIMIT so the database prunes before ranking.
  • Choose between row-level filtering and index isolation by the count and stability of the boundaries; a shared index lets a large tenant crowd out a small one's candidate slots.
  • Caching is the same bug's second crime scene: the answer cache key must carry the permission scope or answers leak across users without a trace.
• 要换一个 embedding 模型，线上系统怎么迁移才能不停机也不掉召回？You need to switch embedding models. How do you migrate a live system without downtime and without losing recall?

  国内高频海外高频深入#embedding-migration#zero-downtime#rollout

  分析过程 · 先想清楚再作答

  1. 这题的题眼是「为什么不能就地换」。没有先说清这一点就直接讲步骤，会显得是在背流程。
  2. 先给前提：不同模型的向量之间**没有可比性**。维度可能不同，即使维度相同坐标系也完全不是一回事，用新模型编码问题去和旧模型编码的文档比距离，算出来的相似度是纯噪声。所以「换模型」实质上等于「把整个知识库重新向量化一遍」。
  3. 然后给四步：加一列新向量、允许为空；后台任务慢慢回填新列，旧列一个字节不动，线上仍走旧列；小流量灰度到新列，同时用标准答案集在两列上各跑一遍比召回率与忠实度；数字站得住再全量切换，旧列观察一两周后才删。
  4. 把这套流程的价值点破，这是给分点：**它的价值全在回滚成本上**。切换只是改一个配置项「走哪一列」，出问题时切回去是一秒钟的事，而不是重跑一遍八小时的重建任务。凡是拿不出回滚路径的迁移方案都不算方案。
  5. 补两个工程细节：新列的近似最近邻索引要在回填完之后再建，边写边建又慢又容易锁表；回填要能断点续传并限速，否则会把 embedding 接口的配额打满，把线上查询一起拖垮。
  6. 可预期的追问：怎么证明新模型确实更好？答：不能只看离线指标涨没涨，要在同一份标准答案集、同一套检索参数下跑 A/B，报召回率、忠实度、延迟、花费四笔账；只报第一笔的结论不成立。另外注意换模型会让缓存里的向量全部作废，那是这次迁移唯一该作废向量缓存的时刻。

  How to reason about it · think before answering

  1. The crux is why you cannot swap in place. Jumping straight to the steps without establishing that reads like reciting a runbook.
  2. Set up the premise: vectors from different models are not comparable. Dimensions may differ, and even at equal dimensions the coordinate spaces are unrelated, so encoding the query with the new model and comparing against documents encoded with the old one yields noise. Switching models therefore means re-embedding the entire corpus.
  3. Then the four steps: add a nullable second vector column; backfill it with a background job while the old column is untouched and still serves live traffic; canary a slice of traffic onto the new column while running the golden set against both columns to compare recall and faithfulness; cut over fully once the numbers hold, and drop the old column only after a week or two of observation.
  4. Name the payoff explicitly, because this is where the points are: the value of the whole procedure is the rollback cost. Cutover is a config change naming which column to read, so reverting takes a second rather than re-running an eight-hour rebuild. A migration plan with no rollback path is not a plan.
  5. Add two engineering details: build the approximate-nearest-neighbour index on the new column after the backfill, not during it, since concurrent building is slow and prone to locking; and make the backfill resumable and rate-limited, or it will exhaust the embedding API quota and drag live queries down with it.
  6. Expected follow-up: how do you prove the new model is actually better? Not from an offline metric alone — run an A/B on the same golden set with identical retrieval parameters and report four numbers: recall, faithfulness, latency and cost. A conclusion resting on the first number only does not hold. Note also that switching models is the one moment when the embedding cache genuinely must be invalidated.

  答题要点

  • 不同模型的向量之间没有可比性，所以换模型等价于把整个知识库重新向量化一遍。
  • 四步：加一列可空的新向量、后台回填、小流量灰度并用标准答案集在两列上对比、数字站得住再全量切换。
  • 这套流程的价值全在回滚成本上：切换是改一个配置项，回滚是一秒钟的事而不是重跑一次重建。
  • 新列的近似最近邻索引在回填完成后再建；回填要可断点续传并限速，别把接口配额打满拖垮线上查询。
  • 验证要在同一份标准答案集上跑 A/B，同时报召回率、忠实度、延迟与花费四笔账；换模型也是唯一该作废向量缓存的时刻。

  Key points

  • Vectors from different models are not comparable, so a model switch is equivalent to re-embedding the entire corpus.
  • Four steps: add a nullable second vector column, backfill in the background, canary with the golden set scored on both columns, then cut over once the numbers hold.
  • The whole value lies in rollback cost: cutover is a config change, so reverting takes a second instead of another full rebuild.
  • Build the ANN index on the new column after the backfill; make the backfill resumable and rate-limited so it does not exhaust the embedding quota and stall live queries.
  • Validate with an A/B on one golden set reporting recall, faithfulness, latency and cost; a model switch is also the only time the embedding cache truly must be invalidated.

D14 综合项目与复盘：多租户企业知识库问答，一张 RAG 决策地图与面试专题

• 给你一个五百万文档、要求秒级响应、准确率优先的知识库场景，你会怎么设计这套系统？You are handed a knowledge base of five million documents that must answer in about a second, with accuracy as the top priority. How would you design it?

  国内高频海外高频深入#system-design#scaling#latency-budget

  分析过程 · 先想清楚再作答

  1. 这题的题眼不在「你会用什么技术」，而在「你有没有一套从约束推配置的方法」。开口就报架构图和技术栈的答案会被判成背方案；拿到分的答法是先把约束翻译成数字，再让每个选择被某个数字逼出来。
  2. 先把三个约束量化：五百万文档按一篇四五块估，是两千多万块，单精度 1536 维就是上百 GB，**索引塞不进单机内存**，这一条直接决定了存储选型；秒级响应意味着从收到问题到第一个字的预算大约一秒，而生成本身通常就吃掉七八百毫秒，检索侧只剩两三百毫秒；准确率优先意味着可以拿延迟和钱换指标，但只能换到那两三百毫秒为止。
  3. 然后逐项落地，每一项都挂在上面某个数字上：存储上专用向量库或分区加半精度量化（半精度的召回损失通常落在重跑噪声里，索引却小四成，这是白捡的）；检索保留关键词与向量两路加倒数排名融合，因为精确匹配的文档号、错误码、人名是向量的固定盲区；重排只作用于融合后的前二十条——它买的是排序质量，一次同步往返，秒级预算里放得下一次，放不下两次。
  4. 接着讲两个「不上」的决定，这一段比上面更能显出做过工程：**Agentic 检索不作为默认路径**，它的收益集中在多跳题上而代价摊给全部问题，秒级预算下更是直接超支——正确做法是先用一次便宜的分类把多跳分流出来，只让那一小部分进循环；**上下文块头之类的手法先不上**，因为它对关键词一路是稀释、对向量一路才是补位，方向相反，得在自己的真实 embedding 上测过再说。
  5. 准确率优先必须落成可验收的东西，否则是空话：一份不少于一百题的标准答案集（其中多跳与无答案各占一成以上）、召回率与排序质量分开看、无答案题的拒答率单独一栏、引用由代码回查而不是靠提示词自觉。**报数字时把最难看的那一栏也报出来**，比只报总分可信得多。
  6. 可预期的追问：五百万文档怎么建第一版索引？答案是这笔钱是一次性大额支出，要按批做、可断点续跑，并且从第一天就上基于内容指纹的增量同步——否则每次改配置都等于把整个知识库重买一遍。再追问就谈灰度：新旧两套向量双写在两列上，用同一份标准答案集在两列上各跑一遍再切流量，回滚只是改一个配置项。

  How to reason about it · think before answering

  1. The real subject here is not which technologies you know, it is whether you have a repeatable way to derive a configuration from constraints. Opening with an architecture diagram reads as a memorized answer; the way to score is to turn each constraint into a number first, then let every choice be forced by one of those numbers.
  2. Quantify the three constraints. Five million documents at roughly four or five chunks each is over twenty million chunks; at 1536 float dimensions that is hundreds of gigabytes, so the index does not fit in one machine's memory — that alone settles storage. A one-second budget to first token, with generation typically eating seven or eight hundred milliseconds, leaves only two or three hundred for retrieval. Accuracy first means you may trade latency and money for metrics, but only within that remaining budget.
  3. Now derive each knob from one of those numbers: a dedicated vector store or partitioning, plus half precision (its recall loss usually sits inside run-to-run noise while the index shrinks by about forty percent — essentially free); keep both keyword and vector routes with reciprocal rank fusion, because exact matches on document ids, error codes and names are a permanent blind spot for embeddings; rerank only the top twenty after fusion, since it buys ranking quality at the cost of one synchronous round trip, and a one-second budget affords exactly one.
  4. Then state two things you deliberately do not build, which is the part that reads as field experience. Agentic retrieval is not the default path: its gains concentrate on multi-hop questions while its cost is spread over every question, and it blows a one-second budget outright — the right move is a cheap classifier that routes only the multi-hop minority into the loop. Contextual chunk headers and similar tricks also wait, because they dilute the keyword route while helping the vector route; the directions are opposite, so measure on your own embeddings before committing.
  5. Accuracy first has to become something you can sign off on. That means a golden set of at least a hundred questions with multi-hop and unanswerable each above ten percent, recall and ranking quality read separately, abstention rate on unanswerable questions as its own column, and citations verified by code rather than trusted from the model. Reporting the ugliest column alongside the headline number is far more credible than reporting a single score.
  6. Expected follow-up: how do you build the first index over five million documents? It is a one-off large expense, so batch it, make it resumable, and put content-hash incremental sync in from day one, or every config change means buying the whole corpus again. Push further and you get to rollout: dual-write the new embeddings into a second column, evaluate both columns on the same golden set, then shift traffic, so rollback is a config flip rather than an eight-hour rebuild.

  答题要点

  • 先把约束翻译成数字：两千多万块决定索引塞不进单机内存，一秒预算里检索侧只剩两三百毫秒。
  • 存储用专用库或分区加半精度；检索保留关键词与向量两路加倒数排名融合，重排只作用于前二十条。
  • 明确说出「不上」的两项：Agentic 只对分流出来的多跳开，块头这类方向相反的手法先测再说。
  • 准确率优先要落成一百题以上的标准答案集、拒答率单独一栏、引用由代码回查。
  • 第一版建索引是一次性大额支出：分批可续跑，并从第一天就上增量同步；换模型走双写切列。

  Key points

  • Translate constraints into numbers first: twenty million chunks means the index will not fit one machine, and a one-second budget leaves retrieval two to three hundred milliseconds.
  • Dedicated store or partitions plus half precision; keep keyword and vector routes with RRF, and rerank only the top twenty after fusion.
  • Name the two things you will not ship: agentic only for a routed multi-hop minority, and chunk headers only after measuring on your own embeddings.
  • Turn accuracy-first into a hundred-plus question golden set, abstention rate as its own column, and code-verified citations.
  • First index build is a one-off large expense: batch it, make it resumable, add incremental sync on day one, and dual-write columns for model swaps.
• 你做过的这个 RAG 项目里，哪个决定你现在会改？为什么？Looking back at the RAG project you built, which decision would you change now, and why?

  国内高频海外高频深入#retrospective#evidence#chunking

  分析过程 · 先想清楚再作答

  1. 这题看着是软性问题，其实区分度极高。答「暂时没有」等于承认没做过复盘；答成一长串自我批评又会显得没有判断力。面试官真正在听的是：你能不能把一个决定、它当时的依据、后来的证据、以及新的判断，四样东西串成一条链子说清楚。
  2. 怎么拆：挑一个**当时有理由、后来被数据推翻**的决定，而不是一个「当时就知道是凑合」的决定。前者证明你有量化的习惯，后者只证明你赶过工期。所以答案的骨架固定是四段——当时选了什么、依据是什么、后来量到了什么、现在的判断是什么。
  3. 本课里有一个现成的样本：某一天先量到「给每个块拼上标题块头之后，命中率不变、索引 token 涨一成、答案文档平均名次从 2.88 退到 3.25」，据此写下「块头对关键词检索是负收益」。后来换成按文档结构切块再复核，这条名次退化**没有复现**。原因是切法变了：固定长度硬切时块边界跟小节边界不对齐，块头会把不属于这一块的标题词塞进来；按结构切时块本身就落在一个小节里，块头补的信息跟块里已有的高度重合。所以正确的表述不是「块头有害」，而是「**块头在块边界与结构不对齐时才有害**」。
  4. 这条链子的价值在于它演示了一个可复用的动作：**给每个结论标出它绑定的前提**。前提变了就要重跑，不能拿新配置去配旧结论。顺着这个思路还能给出第二个例子：曾经写过「向量侧确实是正收益」，后来把融合前的两路拆开数了一遍才发现，向量路独有的候选只占很小一部分，其中含答案文档的次数是零——那个「涨」根本不是语义检索带来的，于是这条结论被自己推翻。
  5. 可预期的追问：那你以后怎么避免这类错误？答两条具体的：一是每个数字旁边写清它绑定了哪几个前提（语料、题集、预算、切法），二是报结论前先问自己一句「这是这次实验测出来的，还是这个结构必然导致的」——前者要标边界，后者才能直接讲。

  How to reason about it · think before answering

  1. This looks like a soft question but it separates people sharply. Saying 'nothing yet' admits you never ran a retrospective; a long list of self-criticism reads as poor judgement. What the interviewer is listening for is whether you can chain four things together: the decision, the evidence you had then, the evidence you got later, and your current call.
  2. How to pick: choose a decision that was justified at the time and later overturned by data, not one you always knew was a shortcut. The first proves you measure; the second only proves you were behind schedule. So the answer has a fixed four-part shape — what you chose, on what basis, what you measured later, and what you now believe.
  3. This course supplies a ready example. One day measured that prepending a heading-path header to every chunk left hit rate unchanged, grew index tokens by about ten percent, and pushed the answer document's mean rank from 2.88 to 3.25 — hence 'headers hurt keyword retrieval'. A later day re-ran the same comparison under structure-aware chunking and the rank regression did not reproduce. The reason was the chunker: with fixed-length cuts, chunk boundaries do not line up with section boundaries, so the header injects heading terms that do not belong to that chunk; with structure-aware cuts, each chunk already sits inside one section and the header largely restates what is already there. The correct statement is therefore not 'headers hurt' but 'headers hurt when chunk boundaries are misaligned with document structure'.
  4. The value of the chain is that it demonstrates a reusable habit: attach the premises to every conclusion. Change a premise and you owe a re-run; you may not pair new settings with an old conclusion. The same reasoning yields a second example: an earlier claim that 'the vector route is clearly a net gain' collapsed once the two routes were counted separately before fusion — the vector-only candidates were a small share and contained the answer document zero times, so the improvement was never semantic at all.
  5. Expected follow-up: how will you avoid this class of error in future? Give two concrete practices. Write the bound premises next to every number — corpus, question set, budget, chunker. And before publishing any conclusion, ask whether it was measured by this experiment or forced by the structure of the implementation; the first needs its boundaries stated, only the second can be asserted flatly.

  答题要点

  • 答案要串成四段：当时选了什么、依据是什么、后来量到了什么、现在的判断是什么。
  • 挑一个当时有理由、后来被数据推翻的决定，而不是一个当时就知道在凑合的决定。
  • 样本：块头从「对关键词检索有害」修正成「块边界与结构不对齐时才有害」，因为切法这个前提变了。
  • 每个数字旁边写清它绑定的前提；前提变了就必须重跑，不能新配置配旧结论。
  • 报结论前先分类：这是实验测出来的，还是实现结构必然导致的——前者要标边界。

  Key points

  • Structure the answer in four beats: the choice, the evidence then, the evidence later, the call now.
  • Pick a decision that was defensible at the time and later overturned by data, not one you knew was a shortcut.
  • Worked example: 'headers hurt keyword retrieval' was corrected to 'headers hurt when chunk boundaries misalign with structure', because the chunker premise changed.
  • Record the premises bound to every number; when a premise changes you owe a re-run rather than a reinterpretation.
  • Classify before asserting: measured by this experiment, or forced by the implementation's structure — the former needs its boundaries stated.