面试题库

14 天 RAG：从检索到可信回答

D8 评估先行：搭 golden set、算召回与排序指标、用模型当裁判判忠实度

• RAG 的评估集里为什么一定要放语料里没有答案的问题？不放会掩盖什么？Why must a RAG evaluation set include questions the corpus cannot answer, and what does leaving them out hide?

  国内高频海外高频基础#evaluation#abstention#golden-set

  分析过程 · 先想清楚再作答

  1. 这题看着简单，实际是在问「你有没有想过评估集本身也会说谎」。答成「为了测试拒答功能」只算及格，答出「不放会让某个故障在报表上完全不可见」才是满分。
  2. 推导只有一步：一个只会硬答的系统，在只有可答问题的评估集上能拿到很高的分——它每次都塞材料给模型，模型每次都编一段话，而评估集根本没有「应该拒答」这一栏。于是最危险的故障在报表上是不存在的。
  3. 结论：无答案问题是唯一能让「乱编」显形的东西。它不参与召回率，它的指标是拒答率——检索侧有没有把不够格的候选全挡下来，生成侧有没有真的说出「资料里没有」。
  4. 出题上有个必须说的细节：无答案问题必须留强干扰词，比如问「网页端支持哪些浏览器」而语料里恰好有一句「提交工单请附上浏览器与版本」。没有干扰词的无答案题检索器一条都捞不到，你测出来的是分词器不是系统。
  5. 可预期的追问是「拒答率低怎么办」。分两层查：先看检索侧的门槛是不是形同虚设（分数阈值定得太低，不相干的块也过关），再看生成侧的提示词有没有明确的拒答指令，两层都要有，只靠提示词兜是不牢的。

  How to reason about it · think before answering

  1. It looks easy but really asks whether you have considered that the eval set itself can lie. 'To test the refusal path' is a pass; 'without them the worst failure is invisible in the report' is a full mark.
  2. The derivation is one step: a system that always answers scores well on a set of answerable questions only. It stuffs context in, the model writes something, and the set has no column for 'should have refused'. The most dangerous failure simply does not appear.
  3. Conclusion: unanswerable questions are the only thing that makes fabrication visible. They are excluded from recall and scored on abstention instead - did retrieval gate out every weak candidate, and did generation actually say the material does not cover this.
  4. One authoring detail worth stating: unanswerable questions need strong distractor terms. Ask which browsers the web client supports when the corpus only says 'attach your browser and version when filing a ticket'. Without distractors retrieval returns nothing and you are testing your tokenizer, not your system.
  5. Expected follow-up: what if the abstention rate is low? Check two layers - whether the retrieval score gate is effectively a no-op, and whether the generation prompt carries an explicit refusal instruction. You need both; a prompt alone is not a reliable gate.

  答题要点

  • 只有可答问题的评估集，会让「不知道也硬答」这个故障完全不可见。
  • 无答案问题不算召回率，它的指标是拒答率，检索侧和生成侧各看一层。
  • 出题必须留强干扰词，否则检索器一条都捞不到，测的是分词器。
  • 建议无答案题占比不低于评估集的一成五，跟多跳题一起构成覆盖度底线。
  • 拒答率低要分两层查：检索门槛是否形同虚设，生成提示词有没有拒答指令。

  Key points

  • An all-answerable eval set makes 'answers confidently when it should not' completely invisible.
  • Unanswerable items are scored on abstention, not recall, and you check both the retrieval gate and the generation refusal.
  • Author them with strong distractor terms, or retrieval returns nothing and you are testing the tokenizer.
  • Keep them at roughly 15% or more of the set, alongside multi-hop items, as the coverage floor.
  • A low abstention rate splits into two causes: a no-op retrieval score gate, or a missing refusal instruction in the prompt.

D9 混合检索与重排：两路召回、倒数排名融合，再用交叉编码器把前几名重新排一遍

• 交叉编码器为什么比双编码器准？既然更准，为什么不干脆拿它直接检索全库？Why is a cross-encoder more accurate than a bi-encoder? And if it is more accurate, why not just use it to search the whole corpus directly?

  国内高频海外高频基础#cross-encoder#bi-encoder

  分析过程 · 先想清楚再作答

  1. 这是一道送分题，但送分题的区分度在第二问。只答「交叉编码器慢」是不够的，要说清慢在结构上的哪一处。
  2. 先给结构差异：双编码器把查询和文档**各自**编码成向量，两者从头到尾没有见过面，最后只靠一次内积凑到一起；交叉编码器把查询和文档拼成一段文本一起过模型，每一层注意力都能让查询的词去看文档的词。
  3. 由此推出准确率差异的来源：双编码器要把一篇文档压成一个固定长度的向量，压缩必然丢信息，「周敏是平台组组长」里两个词的绑定关系未必留得下来；交叉编码器不压缩，它当场对齐。
  4. 第二问的答案就藏在同一个结构里：双编码器的文档向量**可以离线算好**，查询时只做向量检索；交叉编码器没有任何东西能预先算好，N 篇文档就要跑 N 次前向。十万块的语料重排一遍，等于每次提问都把整个库过一遍模型。
  5. 所以工程上的定位是分工：召回负责在全库里捞出一小批（便宜、可索引），重排负责把这一小批的顺序改对（贵、准）。默认只重排融合后的前 20 条。
  6. 可预期的追问：有没有中间路线？答有——后期交互（late interaction）那一类，文档侧提前算好词级表示、查询侧当场做交互，精度和成本都在两者之间，代价是索引体积大得多。

  How to reason about it · think before answering

  1. This is a giveaway question, but the discriminating half is the second part. Saying `cross-encoders are slow` is not enough; you have to point at the structural reason.
  2. Start with the structure: a bi-encoder encodes query and document **separately** into vectors that never meet until a single dot product at the end; a cross-encoder concatenates query and document into one sequence, so every attention layer lets query tokens attend to document tokens.
  3. That yields the accuracy gap: a bi-encoder must compress a document into one fixed-length vector, and compression loses information — the binding between `Zhou Min` and `platform team lead` may not survive. A cross-encoder does not compress; it aligns them on the spot.
  4. The answer to the second half hides in the same structure: bi-encoder document vectors can be computed **offline** and indexed, so query time is just a vector search. A cross-encoder has nothing to precompute — N documents means N forward passes. Reranking a 100k-chunk corpus means pushing the entire corpus through a model on every question.
  5. So the engineering split is a division of labor: recall pulls a small batch out of the whole corpus (cheap, indexable), reranking fixes the order of that batch (expensive, accurate). The default is to rerank only the top 20 after fusion.
  6. Expected follow-up: is there a middle path? Yes — late interaction, where token-level document representations are precomputed and the interaction happens at query time. Accuracy and cost land between the two, at the price of a much larger index.

  答题要点

  • 双编码器各自编码、最后一次内积；交叉编码器把查询和文档拼在一起过模型，注意力可以跨两者对齐。
  • 准确率差异来自压缩：双编码器把整篇文档压成一个向量，绑定关系会丢；交叉编码器不压缩。
  • 双编码器的文档向量能离线算好并建索引，交叉编码器没有任何东西可以预先算好。
  • 全库重排等于每次提问把整个语料过一遍模型，成本随语料规模线性增长。
  • 标准分工是召回加重排，重排只作用于融合后的前几十条。

  Key points

  • A bi-encoder encodes both sides separately and joins them with one dot product; a cross-encoder concatenates them so attention can align across the pair.
  • The accuracy gap comes from compression: a bi-encoder squeezes a whole document into one vector and loses bindings; a cross-encoder does not compress.
  • Bi-encoder document vectors can be computed offline and indexed; a cross-encoder has nothing to precompute.
  • Reranking the full corpus means running every chunk through a model on every question, so cost scales linearly with corpus size.
  • The standard split is recall plus rerank, with reranking applied only to the top few dozen after fusion.

D10 查询侧优化：改写、假设文档嵌入、多路查询、后退提问与意图路由

• 多轮对话里怎么处理指代？不做指代消解最典型的翻车场景是什么？How do you handle coreference in multi-turn RAG, and what is the classic failure when you skip it?

  国内高频海外高频基础#coreference#multi-turn#query-rewriting

  分析过程 · 先想清楚再作答

  1. 这是一道送分题，但送分题也有高下之分：只说「把历史拼进查询里」的答案，会被追问一句「历史越拼越长怎么办」就卡住。
  2. 先把做法说清：在检索之前加一次很短的改写调用，输入是最近几轮对话加本轮问题，输出是一行可以直接检索的检索式；温度设 0 保证同一句话每次改成同一个结果，并在提示词里明确禁止模型顺手回答问题。
  3. 为什么不是「把历史整个拼进查询」：历史越拼越长，噪声词把逆文档频率摊薄，检索反而更差；而且历史里包含上一轮的答案，等于拿答案去检索答案。改写的产出是一句话，不是一段历史。
  4. 最典型的翻车场景要举实例：上一轮问「这个操作必须由谁审批」，答「必须由某某组组长审批」；这一轮问「这个人叫什么名字」。不消解直接检索这七个字，实测是**一条候选都过不了门槛，系统只能拒答**。注意失败方式不是答错，是「明明语料里有答案却说找不到」，用户体验是崩塌式的。
  5. 补一条顺序上的坑：改写必须在意图路由**之前**。「这个人」是典型的多跳信号词，路由看到它会判成多跳、白跑一轮；改写之后它只是个普通单跳问题。同理，多路查询、后退提问也都要建立在改写后的那句话上，否则错误被放大好几倍。
  6. 可预期的追问是「怎么知道要不要改写」。答：短问题、含指代词、含省略（「那审计日志呢」）时才触发，纯新话题跳过——这一步很便宜，但能省掉一大半调用。

  How to reason about it · think before answering

  1. This is a warm-up question, but there is still a gap between answers. Saying "just concatenate the history into the query" invites a follow-up about growing histories that most candidates cannot handle.
  2. State the mechanism: insert a short rewrite call before retrieval that takes the last few turns plus the current question and returns one retrieval-ready line. Set temperature to 0 so the same input always yields the same query, and forbid the model from answering the question in the prompt.
  3. Explain why concatenation is worse: history grows without bound, filler words dilute inverse document frequency, and the previous answer leaks in — you end up retrieving an answer with an answer. The rewriter emits one sentence, not a transcript.
  4. Make the failure concrete. Turn one: "who must sign off on this operation?" Answer: "the platform team lead." Turn two: "what is that person's name?" Retrieved unresolved, not a single candidate clears the admission gate and the system refuses — even though the corpus contains the answer. The failure is not a wrong answer, it is a false "not found" right after the user's own question.
  5. Add the ordering trap: rewrite before intent routing. A pronoun is a classic multi-hop signal, so an unresolved query gets routed to the expensive path for nothing; after rewriting it is an ordinary single-hop question. Multi-query and step-back must also sit downstream of the rewrite, or one unresolved pronoun becomes three.
  6. Expect "how do you decide when to rewrite?" Trigger on short queries, pronouns and elliptical follow-ups; skip on a clearly new topic. The check is nearly free and removes most of the calls.

  答题要点

  • 在检索前加一次短改写调用，输入最近几轮加本轮问题，输出一行检索式，温度 0，禁止模型回答问题。
  • 不要把历史整段拼进查询：越拼越长、噪声稀释逆文档频率，还会拿上一轮的答案去检索。
  • 典型翻车：上一轮的「这个人 / 他 / 那个」不消解，检索一条都过不了门槛，系统在有答案的情况下拒答。
  • 失败方式是「假的查不到」，比答错更伤体验，因为用户刚刚才问过同一件事。
  • 顺序：先改写、再路由，多路查询与后退提问都建立在改写后的查询上。

  Key points

  • Add a short rewrite call before retrieval: last few turns plus current question in, one retrieval line out, temperature 0, answering explicitly forbidden.
  • Do not splice the whole history into the query — it grows unbounded, dilutes IDF, and leaks the previous answer into the search.
  • Classic failure: an unresolved pronoun means no candidate clears the gate, so the system refuses a question the corpus can answer.
  • That false "not found" hurts more than a wrong answer, since the user just asked about the same thing.
  • Order matters: rewrite first, then route; multi-query and step-back both build on the rewritten query.

D12 Agentic RAG：把检索做成工具，让模型自己决定查不查、查几次、要不要推翻重来

• 把检索包成一个工具交给模型，这个工具的描述该怎么写？写不好会导致哪些具体的错误行为？You are exposing retrieval to a model as a tool. How do you write the tool description, and what concrete failure modes appear when you write it badly?

  国内高频海外高频基础#tool-design#agentic-rag#prompting

  分析过程 · 先想清楚再作答

  1. 这题的题眼是「具体的错误行为」。只会背「描述要写清楚工具的用途」的，一句话就暴露了没上过线——面试官想听的是描述里少一句话，线上就多一类工单。
  2. 先给结构：一段合格的工具描述要回答四件事——库里有什么和没有什么、什么时候必须用、什么时候不要用、查询串写成什么形状。四条各对应一类事故，逐条挂钩着说最有说服力。
  3. 逐条挂钩：不写范围，模型拿它当搜索引擎，问天气也去查；不写「必须用」，涉及公司制度的问题被模型凭记忆编答案，而且编得非常像真的；不写「不要用」，闲聊和翻译都触发一次无谓检索，成本和延迟白涨；不写查询形状，模型把用户整句问话塞进 query，「叫什么名字」这种疑问词进了检索，纯噪声。
  4. 最后一条最值钱也最容易漏：在描述里加一句「写成关键词短语，不要带疑问词」，比在检索侧做十种查询清洗都管用——问题在源头，就在源头修。
  5. 补一个生产视角：参数里的过滤字段（比如部门）要写明「只在确定时才填」。模型倾向于把可选参数填满，填错一个部门就把正确答案挡在库外，而这种错误在日志里看不出来，表现是「检索没结果」。
  6. 可预期的追问是「怎么验证描述写对了」。答案是拿一批负样本跑：闲聊、翻译、算术、以及答案已在对话里的追问，看模型有没有多调一次工具；这类回归是能自动化的。

  How to reason about it · think before answering

  1. The discriminator is whether you can name concrete failure modes. Reciting 'the description should be clear' signals you have never shipped one.
  2. Give the structure first: a usable description answers four things - what is and is not in the corpus, when the tool must be called, when it must not be called, and what shape the query string should take.
  3. Attach a failure to each: no scope and the model treats it as a web search; no 'must call' and it answers policy questions from memory, convincingly; no 'must not call' and greetings or translations each burn a retrieval; no query shape and the model pastes the raw user sentence in, dragging interrogative words into the index.
  4. The query-shape line is the cheapest win: one sentence saying 'keyword phrase, no question words' beats ten heuristics for query cleaning on the retrieval side.
  5. Production angle: optional filter parameters such as department need an explicit 'only set this when you are certain'. Models like to fill optional fields, and a wrong filter hides the correct answer while the logs only show 'no results'.
  6. Expected follow-up: how do you verify the description works? Run a negative suite - small talk, translation, arithmetic, follow-ups already answered in the conversation - and assert the tool was not called. That regression is automatable.

  答题要点

  • 描述是写给模型看的提示词，不是注释；四段式：范围、什么时候用、什么时候不用、查询写成什么形状。
  • 不写范围会被当成搜索引擎；不写「必须用」会导致凭记忆编答案。
  • 不写「不要用」会让闲聊也触发检索，成本和延迟白涨。
  • 写明查询要用关键词短语、不带疑问词，比在检索侧清洗查询更根本。
  • 可选过滤参数要写「只在确定时才填」，填错会静默地把正确答案挡在外面。
  • 用一批负样本（闲聊、翻译、算术）做回归，断言工具没有被调用。

  Key points

  • The description is a prompt for the model, not a code comment: scope, when to call, when not to call, query shape.
  • Missing scope turns it into a web search; missing 'must call' produces confident answers from memory.
  • Missing 'do not call' makes small talk trigger retrieval, paying cost and latency for nothing.
  • Stating 'keyword phrase, no question words' fixes query pollution at the source.
  • Optional filters need 'only set when certain' - a wrong filter silently hides the right answer.
  • Regression-test with a negative suite and assert the tool was not invoked.

D14 综合项目与复盘：多租户企业知识库问答，一张 RAG 决策地图与面试专题

• 怎么向不懂技术的业务方证明你的检索系统真的变好了？How do you convince a non-technical stakeholder that your retrieval system actually got better?

  国内高频海外高频基础#evaluation#stakeholder-communication#abstention

  分析过程 · 先想清楚再作答

  1. 这题在考沟通，但拿分点在技术判断上：你选哪几个数字给业务方看，暴露了你自己有没有看懂这些指标。把召回率、nDCG、MRR 一股脑摊出去的答法会被判成不懂受众；只说「用户反馈变好了」又会被判成没有度量。
  2. 先立一条原则：**给业务方看的必须是他们能自己判断对错的东西**。归一化折损累计增益他们没法判断，而「这一百个真实问题里，系统答对了多少、答错了多少、老老实实说查不到了多少」他们一眼就能判断。所以对外的口径应该是三个数：答对率、答错率、拒答率，而且三个加起来是一百。
  3. 关键是把**答错和拒答分开**。这一条最能建立信任：查不到就说查不到不是故障，是正确输出；真正的故障是查不到还编一段。很多团队只报「准确率」，结果一个学会了一直拒答的系统能刷出满分——所以这三个数必须并排出现，缺一个都能被骗。
  4. 然后给可核对的证据，而不是只给数字：**挑十条真实问题做前后对照**，各贴出改动前和改动后的回答，每句结论后面挂着可点开的引用。业务方点开原文核对一遍，比看任何百分比都有说服力，而且这个动作顺带完成了一次人工抽检——你自己也需要它来校准模型裁判靠不靠谱。
  5. 本课里有一条要主动说的教训：**一列全是满分说明尺子坏了**。我们的题目是从语料反向出的，字面重合度过高，平均倒数排名恒为 1.0000。这个数字拿给业务方看，只会换来一次「那你们已经完美了」的误会，而它其实是指标饱和。指标撞天花板时该做的是把题目出难一点。
  6. 可预期的追问：那怎么让业务方参与进来？答一条很实用的：让他们提供题目。把线上答错的问题一条条补进标准答案集，评估集是长出来的，而不是一次性造好的；这样每一次改进都能指着「你上次提的那个问题现在答对了」，比任何汇报都直接。

  How to reason about it · think before answering

  1. This is a communication question whose scoring hinges on technical judgement: which numbers you choose to show reveals whether you understand the metrics yourself. Dumping recall, nDCG and MRR on a business stakeholder reads as tone-deaf; saying 'user feedback improved' reads as unmeasured.
  2. Start from a principle: show them something they can adjudicate themselves. They cannot judge normalized discounted cumulative gain, but they can absolutely judge 'out of these hundred real questions, how many did it answer correctly, how many wrongly, and how many did it honestly decline'. So the external framing is three numbers — correct, wrong, declined — and they sum to one hundred.
  3. The crucial move is separating wrong from declined, and it is the fastest way to earn trust: saying 'not found' is a correct output, not a failure; the failure is inventing an answer when nothing was found. Teams that report a single 'accuracy' number can be gamed by a system that learns to decline everything, which is why all three must appear side by side.
  4. Then supply checkable evidence rather than only numbers: take ten real questions and show before-and-after answers with clickable citations on every claim. A stakeholder who opens the source and verifies one claim is more convinced than by any percentage, and the exercise doubles as the human spot-check you need anyway to calibrate whether your model judge is trustworthy.
  5. There is a lesson from this course worth volunteering: a column of perfect scores means the ruler is broken. Our questions were written backwards from the corpus, lexical overlap is unusually high, and mean reciprocal rank sits at exactly 1.0000. Showing that to a stakeholder only invites the misreading that you are already perfect, when in fact the metric has saturated. When a metric hits the ceiling, the response is to make the questions harder.
  6. Expected follow-up: how do you get the business side involved? One very practical answer: let them supply questions. Every production miss gets appended to the golden set, so the evaluation set grows rather than being built once. Then each release can point at 'the question you raised last month now answers correctly', which lands better than any status report.

  答题要点

  • 对外只用三个他们能自己判断的数：答对率、答错率、拒答率，三者相加为一百。
  • 答错和拒答必须分开——查不到就说查不到是正确输出，只报一个准确率会被「一直拒答」刷满分。
  • 配十条真实问题的前后对照，每句结论挂可点开的引用，让他们自己核对原文。
  • 主动说明指标饱和：某一列恒为满分是尺子坏了，不是系统完美，该做的是把题目出难一点。
  • 让业务方提供题目，把线上答错的问题补进标准答案集——评估集是长出来的。

  Key points

  • Externally report three numbers they can adjudicate: correct, wrong, declined — summing to one hundred.
  • Keep wrong and declined separate; a single accuracy number is gamed by a system that learns to decline everything.
  • Pair it with ten before-and-after real questions, every claim carrying a citation they can open and verify.
  • Volunteer the saturation caveat: a column of perfect scores means a broken ruler, and the fix is harder questions.
  • Let stakeholders contribute questions; append every production miss to the golden set so it grows over time.