面试题库

14 天 RAG：从检索到可信回答

D2 embedding 与向量检索：相似度、维度与模型选型，把文本存进 pgvector

• 向量检索能完全取代关键词检索吗？举一个向量必然失手的查询，并说说你会怎么补。Can vector search fully replace keyword search? Give a query where vectors are bound to fail, and say how you would fix it.

  国内高频海外高频进阶#hybrid-search#embeddings#retrieval-failure

  分析过程 · 先想清楚再作答

  1. 这题是典型的「立场题」，答「能」或「不能」都不重要，重要的是你能不能举出一个具体到能复现的反例。举不出例子，前面说得再漂亮也会被判成没做过。
  2. 先给失手的类型，一次给全：错误码与状态码（429、E1032）、版本号与型号（v2.3.1、X20 Pro）、人名与工号、订单号与文档编号、以及否定表达。前四类的共同点是**这些词的价值在于字面唯一，而向量只保留语义邻近**，模型会把 429 和「限流」「超时」这些话题相近的东西编到一起，反而把真正写着 429 的那篇挤下去。
  3. 拿一个能复现的例子说：问「限流超了返回 429 吗」，BM25 稳稳命中写着 429 的接口文档，向量却可能把话题相近但没提 429 的产品手册排在前面。这个现象在本课第 2 天的实验里就能亲眼看到。
  4. 否定表达要单独强调：「支持导出 PDF」和「不支持导出 PDF」在向量空间里几乎重合，因为它们谈的是同一件事。指望向量区分肯定与否定一定翻车，这一层要靠生成侧读原文来判断。
  5. 怎么补：两路并行跑再融合，关键词一路用 BM25、向量一路用最近邻，用倒数排名融合把两个名次合成一个。这就是混合检索，本课第 9 天展开。要点是**两套的错法不一样**，所以合起来才有增益——如果两套错在同一批查询上，融合是白做的。
  6. 可预期的追问：那关键词一路能不能扔掉、改成让模型改写查询？可以缓解一部分（第 10 天的查询改写），但改写救不了字面唯一的标识符——你没法把 429 改写成别的说法。

  How to reason about it · think before answering

  1. This is a stance question where the stance matters less than the counter-example. Without a concrete, reproducible failing query, the rest of the answer reads as theory.
  2. Enumerate the failure classes up front: error and status codes, version numbers and SKUs, names and employee IDs, order or document identifiers, and negation. The first four share one property: their value lies in exact literal identity, which embeddings deliberately blur into semantic neighbourhoods.
  3. Give a reproducible example: ask whether rate limiting returns 429. BM25 lands on the API document that literally contains 429, while vector search may rank a topically similar product manual that never mentions the code.
  4. Call out negation separately: 'supports PDF export' and 'does not support PDF export' sit almost on top of each other because they discuss the same thing. Vectors cannot carry that distinction; the generation step reading the source has to.
  5. The fix: run both retrievers and fuse the rankings, BM25 on the lexical side and nearest neighbour on the vector side, combined with reciprocal rank fusion. That is hybrid search, covered on day 9. Fusion helps precisely because the two systems fail on different queries.
  6. Expected follow-up: could you drop the keyword path and rewrite queries instead? Rewriting helps with vocabulary mismatch, but it cannot rescue exact identifiers, since there is no paraphrase of 429.

  答题要点

  • 不能取代：错误码、版本号、人名、单号这类词的价值在于字面唯一，向量只保留语义邻近。
  • 具体反例：问「限流超了返回 429 吗」，BM25 命中写着 429 的文档，向量把话题相近却没提 429 的文档排前面。
  • 否定表达是另一类失手：肯定句与否定句在向量空间里几乎重合。
  • 补法是混合检索：两路并行再用倒数排名融合合并名次。
  • 融合有增益的前提是两套的错法不同；查询改写能缓解词汇不匹配，但救不了字面唯一的标识符。

  Key points

  • No: codes, version numbers, names and IDs matter as exact literals, which embeddings blur into neighbourhoods.
  • Concrete example: asking whether rate limiting returns 429, where BM25 hits the document containing 429 and vectors surface a topically similar one that never mentions it.
  • Negation is a second failure class, since affirmative and negative statements sit almost on top of each other.
  • The remedy is hybrid retrieval: run both paths and merge with reciprocal rank fusion.
  • Fusion pays off because the two paths fail differently; query rewriting helps vocabulary mismatch but not exact identifiers.

D3 文档进来这一关：PDF 与 HTML 解析、表格与扫描件、清洗规则和必须留下的元数据

• 扫描件走光学字符识别之后错字率不低，这些噪声会怎样影响检索和生成？怎么缓解？OCR output from scanned documents carries a non-trivial error rate. How does that noise propagate into retrieval and generation, and how do you mitigate it?

  国内高频海外高频深入#ocr#data-quality#hybrid-search

  分析过程 · 先想清楚再作答

  1. 这题考的是你会不会顺着链条推传导，而不是背「OCR 会有错字」这句废话。判据是有没有分别说清「检索侧怎么错」和「生成侧怎么错」——它们的失效方式完全不同。
  2. 先说检索侧。中文 OCR 的错主要是形近字，「已」认成「己」、「板」认成「版」。关键词检索是字面匹配，一个字错了这个词就查不到；更隐蔽的是二元组分词会连带毁掉相邻两个词元，一个错字影响的其实是两处。这一路的表现是召回悄悄掉下去，而且不报错。
  3. 再说生成侧。错字进了上下文，模型往往能读懂大意，但一旦是关键实体（人名、型号、金额、日期）出错，它会照着错的答，而且答得很自信。更麻烦的是引用校验也会跟着失效——原文本身就是错的，校验通过了也没意义。
  4. 缓解按三层说。入口层：先用空文本比例这类断言判断这份 PDF 有没有文本层，有就别走 OCR；真要走，保留原图链接以便人工复核。
  5. 检索层：靠混合检索兜底，向量一路对个别错字不敏感，能补上关键词一路的失手，这是 D9 那套东西在这里的具体价值。生成层：把低置信度的页面标出来，让模型在引用它们时明确提示「该材料来自扫描件，可能有识别误差」。
  6. 可预期的追问：能不能自动纠错？可以但要克制——用词典或模型做后处理会修好一批，也会「修」坏一批原本正确的专有名词。稳妥的做法是只对置信度低的片段做纠错，并且保留原文以便回退。

  How to reason about it · think before answering

  1. This tests whether you can trace propagation rather than recite that OCR makes mistakes. The differentiator is separating how retrieval fails from how generation fails, because the two failure modes are entirely different.
  2. Retrieval first. Chinese OCR errors are mostly visually similar characters. Keyword search is literal, so one wrong character makes the term unmatchable, and bigram tokenisation makes it worse because a single wrong character corrupts two adjacent tokens. Recall drops quietly and nothing raises an error.
  3. Generation second. The model usually reads through minor noise, but when the corrupted token is a key entity such as a name, a model number, an amount or a date, it answers confidently with the wrong value. Citation checking degrades too: verifying against a source that is itself wrong proves nothing.
  4. Mitigate in three layers. At ingest, use an empty-text assertion to decide whether the PDF even needs OCR, and keep a link to the original image so a human can verify.
  5. At retrieval, hybrid search absorbs some of the damage because dense retrieval is less sensitive to a single wrong character than literal matching. At generation, mark low-confidence pages so the answer can state that the source came from a scan and may contain recognition errors.
  6. Expected follow-up: can you auto-correct? Yes, but carefully. Dictionary or model based post-processing fixes some errors and breaks correct proper nouns. Restrict correction to low-confidence spans and keep the raw text so you can fall back.

  答题要点

  • 检索侧：形近字让字面匹配直接查不到，二元组分词还会让一个错字毁掉相邻两个词元，表现是召回悄悄下降且不报错。
  • 生成侧：模型能读懂大意，但关键实体出错时会自信地答错，引用校验也失去意义。
  • 入口层缓解：先判断有没有文本层再决定要不要 OCR，并保留原图链接供人工复核。
  • 检索层缓解：混合检索里的向量一路对个别错字不敏感，能兜住关键词一路的失手。
  • 生成层缓解：标出低置信度来源，让回答显式提示可能存在识别误差；自动纠错只对低置信片段做并保留原文。

  Key points

  • Retrieval: visually similar characters break literal matching, and bigram tokenisation lets one bad character corrupt two tokens, so recall drops silently.
  • Generation: the model reads through general noise but confidently repeats corrupted entities, and citation verification against a corrupted source proves nothing.
  • At ingest: check for a text layer before running OCR at all, and keep the source image for human verification.
  • At retrieval: hybrid search helps because dense retrieval tolerates a single wrong character better than literal matching.
  • At generation: flag low-confidence sources in the answer, and restrict auto-correction to low-confidence spans while keeping the raw text.

D9 混合检索与重排：两路召回、倒数排名融合，再用交叉编码器把前几名重新排一遍

• 混合检索为什么普遍用倒数排名融合，而不是把两路分数归一化之后加权相加？加权那条路在什么情况下会失控？Why do hybrid retrieval systems usually use reciprocal rank fusion instead of normalizing both scores and adding them with weights? When does the weighted approach break down?

  国内高频海外高频进阶#hybrid-search#rank-fusion

  分析过程 · 先想清楚再作答

  1. 这题的题眼在「分数」两个字。只答「RRF 更简单」是背概念，面试官想听的是你知道分数为什么不可比。
  2. 先给量纲差异：BM25 是一堆对数项累加，没有上界，同一套索引里不同查询的第一名可以从 5 分到 50 分；余弦被钉死在负一到正一。两个读数相加没有意义。
  3. 再点出归一化的静默失败：除以本路最高分之后，分母随查询浮动。一个语料里根本没有答案的问题，向量那一路最高分只有 0.09，归一化之后照样是满分 1.0 带权重进融合——你以为在比相关性，其实在比「本路矮子里有多高」。
  4. 然后是权重的维护成本：1 比 0.6 这个配比要靠跑评估调出来，两路是二维搜索，加上多路查询就是四维五维，而且换一个 embedding 模型全部作废。RRF 只有一个 k，而且 60 这个默认值几乎不用动。
  5. 结论：名次是两路唯一可比的东西。RRF 主动扔掉分数，是为了不被不可比的量误导。
  6. 可预期的追问：那 k 是干什么的？答 k 是压平器——k 越大，头几名之间的差距越小，于是「两路都排进前列」比「一路排第一」更有分量，这正是混合检索想要的交叉验证效果。再追问同分怎么办，答必须按文档 id 兜底排序，否则跨次运行名次会飘、评估数字跟着抖。

  How to reason about it · think before answering

  1. The hinge word is `scores`. Answering `RRF is simpler` is reciting a concept; the interviewer wants to hear that you know why the two scores are not comparable in the first place.
  2. Start with scale: BM25 is an unbounded sum of log terms, and on one index the top hit can range from 5 to 50 depending on the query; cosine is pinned between -1 and 1. Adding those two readings is meaningless.
  3. Then name the silent failure of normalization: dividing by the per-route maximum makes the denominator float with the query. For a question with no answer in the corpus, the vector route's best hit may score 0.09 and still normalize to a perfect 1.0, entering the fusion at full weight. You think you are comparing relevance; you are comparing `tallest among the short`.
  4. Then the maintenance cost of weights: a 1-to-0.6 ratio has to be tuned against an eval set, tuning two routes is a 2-D search, adding multi-query retrieval makes it 4-D or 5-D, and swapping the embedding model invalidates all of it. RRF has a single k, and the default of 60 rarely needs touching.
  5. Conclusion: rank is the only thing the two routes share. RRF throws the scores away on purpose so that an incomparable quantity cannot mislead it.
  6. Expected follow-up: what does k do? It flattens — the larger k is, the smaller the gap between the top few ranks, so `ranked well by both routes` outweighs `ranked first by one route`, which is exactly the cross-validation effect hybrid retrieval is after. A second follow-up on ties: you must fall back to sorting by document id, or ranks drift between runs and every eval number wobbles with them.

  答题要点

  • BM25 无上界、余弦有界，两个量纲不可比，直接相加没有意义。
  • 按本路最高分归一化的分母随查询浮动，无答案的查询里最不相干的结果也能拿到满分。
  • 权重要跑评估调，路数一多就是高维搜索，换模型还得重来；RRF 只有一个常数 k。
  • RRF 只吃每一路的有序 id 列表，名次是两路唯一可比的东西。
  • k 越大越奖励「两路都排进前列」；同分必须按 id 兜底排序才可复现。

  Key points

  • BM25 is unbounded, cosine is bounded; the two scales are not comparable, so adding them is meaningless.
  • Per-route max normalization has a denominator that floats with the query, so the least relevant hit of an unanswerable query still normalizes to 1.0.
  • Weights must be tuned against an eval set, the search is high-dimensional once you add routes, and swapping models invalidates it; RRF has a single constant k.
  • RRF consumes only the ordered id list from each route, because rank is the one thing the routes share.
  • Larger k rewards `ranked well by both routes`; ties must fall back to document id so results are reproducible.
• 你把纯向量检索换成了混合检索加重排，上线之后评估指标反而掉了。你会怎么排查？You replaced pure vector retrieval with hybrid search plus reranking, and after shipping it your eval metrics went down. How do you investigate?

  国内高频海外高频深入#hybrid-search#evaluation

  分析过程 · 先想清楚再作答

  1. 这题考的是你有没有真的做过分阶段归因。答「调一下权重再看看」就输了——那是在猜，不是在查。
  2. 第一步是拆档跑，不是改代码：纯关键词、纯向量、混合、混合加重排四档在**同一份评估集、同一个上下文预算**下各跑一遍。指标掉在哪一档就在哪一档找原因，这一步能立刻区分「融合坏了」和「重排坏了」。
  3. 第二步问一个具体问题：掉的是召回率还是排序指标？召回率掉说明答案根本没进上下文，是候选池或者预算的问题；排序指标掉而召回率没动，说明答案还在、只是被挤到了后面，那是融合权重或重排模型的问题。这两类故障的解法完全不同。
  4. 第三个常见根因是召回深度。每路取多少条这个旋钮方向反直觉：取深了不是更保险，是把噪声也一起投了票。我在一份 134 块的语料上实测过，每路从取 50 收到取 5，混合那一档的召回率从 87.5% 回到 93.8%、多跳档从 50% 回到 75%，而 nDCG 反而掉了近 0.1——两个指标会打架，先想清楚业务要哪个。
  5. 第四个根因是评估口径被悄悄改了。上下文预算、命中判定、候选池深度只要动过一个，新旧数字就不可比，这时候「掉了」可能根本不是真的掉了。
  6. 可预期的追问：怎么防止下次再踩？答把四档对照做成一条命令、把上一版报告存成基线、指标退步就以非 0 退出码拦住合并——这就是评估要先于优化的原因。

  How to reason about it · think before answering

  1. This question tests whether you have actually done stage-by-stage attribution. Answering `I would tune the weights and see` loses — that is guessing, not investigating.
  2. Step one is to run the stages apart, not to change code: pure keyword, pure vector, hybrid, and hybrid plus rerank, all on the **same eval set with the same context budget**. Whichever stage the drop appears in is where you look, and this alone separates `fusion is broken` from `reranking is broken`.
  3. Step two asks a specific question: did recall drop, or did the ranking metrics drop? A recall drop means the answer never entered the context at all — a candidate-pool or budget problem. Ranking metrics dropping while recall holds means the answer is still there but pushed down — a fusion-weight or rerank-model problem. The two failures have completely different fixes.
  4. A third common root cause is recall depth. This knob runs against intuition: going deeper is not safer, it lets noise vote too. On a 134-chunk corpus I measured that narrowing each route from 50 to 5 took hybrid recall from 87.5% back to 93.8% and multi-hop from 50% to 75%, while nDCG fell by almost 0.1. The metrics fight each other, so decide which one the product needs first.
  5. A fourth root cause is that the eval protocol quietly changed. Touch the context budget, the hit rule, or the candidate depth, and the old and new numbers stop being comparable — in which case the `drop` may not be a drop at all.
  6. Expected follow-up: how do you avoid this next time? Make the four-way comparison a single command, store the previous report as a baseline, and fail the build with a non-zero exit code on regression. That is precisely why evaluation comes before optimization.

  答题要点

  • 先拆档跑四种配置，在同一份评估集和同一个上下文预算下归因，不要一上来就调参。
  • 区分召回率掉与排序指标掉：前者是候选池或预算问题，后者是融合或重排问题。
  • 查召回深度：每路取太深会把噪声也投进融合，收窄反而可能救回召回率。
  • 确认评估口径没被改：预算、命中判定、候选池深度动过一个，新旧数字就不可比。
  • 把四档对照固化成一条命令加一份基线报告，指标退步直接拦住合并。

  Key points

  • Run all four configurations separately for attribution, on one eval set with one context budget, before touching any parameter.
  • Separate a recall drop from a ranking drop: the first is a candidate-pool or budget issue, the second is a fusion or rerank issue.
  • Check recall depth: taking too many per route lets noise vote, and narrowing it can bring recall back.
  • Confirm the eval protocol did not change; touching budget, hit rule, or candidate depth makes old and new numbers incomparable.
  • Freeze the four-way comparison into one command plus a baseline report, and block merges on regression.