面试题库

14 天 RAG：从检索到可信回答

D8 评估先行：搭 golden set、算召回与排序指标、用模型当裁判判忠实度

• RAG 的评估集里为什么一定要放语料里没有答案的问题？不放会掩盖什么？Why must a RAG evaluation set include questions the corpus cannot answer, and what does leaving them out hide?

  国内高频海外高频基础#evaluation#abstention#golden-set

  分析过程 · 先想清楚再作答

  1. 这题看着简单，实际是在问「你有没有想过评估集本身也会说谎」。答成「为了测试拒答功能」只算及格，答出「不放会让某个故障在报表上完全不可见」才是满分。
  2. 推导只有一步：一个只会硬答的系统，在只有可答问题的评估集上能拿到很高的分——它每次都塞材料给模型，模型每次都编一段话，而评估集根本没有「应该拒答」这一栏。于是最危险的故障在报表上是不存在的。
  3. 结论：无答案问题是唯一能让「乱编」显形的东西。它不参与召回率，它的指标是拒答率——检索侧有没有把不够格的候选全挡下来，生成侧有没有真的说出「资料里没有」。
  4. 出题上有个必须说的细节：无答案问题必须留强干扰词，比如问「网页端支持哪些浏览器」而语料里恰好有一句「提交工单请附上浏览器与版本」。没有干扰词的无答案题检索器一条都捞不到，你测出来的是分词器不是系统。
  5. 可预期的追问是「拒答率低怎么办」。分两层查：先看检索侧的门槛是不是形同虚设（分数阈值定得太低，不相干的块也过关），再看生成侧的提示词有没有明确的拒答指令，两层都要有，只靠提示词兜是不牢的。

  How to reason about it · think before answering

  1. It looks easy but really asks whether you have considered that the eval set itself can lie. 'To test the refusal path' is a pass; 'without them the worst failure is invisible in the report' is a full mark.
  2. The derivation is one step: a system that always answers scores well on a set of answerable questions only. It stuffs context in, the model writes something, and the set has no column for 'should have refused'. The most dangerous failure simply does not appear.
  3. Conclusion: unanswerable questions are the only thing that makes fabrication visible. They are excluded from recall and scored on abstention instead - did retrieval gate out every weak candidate, and did generation actually say the material does not cover this.
  4. One authoring detail worth stating: unanswerable questions need strong distractor terms. Ask which browsers the web client supports when the corpus only says 'attach your browser and version when filing a ticket'. Without distractors retrieval returns nothing and you are testing your tokenizer, not your system.
  5. Expected follow-up: what if the abstention rate is low? Check two layers - whether the retrieval score gate is effectively a no-op, and whether the generation prompt carries an explicit refusal instruction. You need both; a prompt alone is not a reliable gate.

  答题要点

  • 只有可答问题的评估集，会让「不知道也硬答」这个故障完全不可见。
  • 无答案问题不算召回率，它的指标是拒答率，检索侧和生成侧各看一层。
  • 出题必须留强干扰词，否则检索器一条都捞不到，测的是分词器。
  • 建议无答案题占比不低于评估集的一成五，跟多跳题一起构成覆盖度底线。
  • 拒答率低要分两层查：检索门槛是否形同虚设，生成提示词有没有拒答指令。

  Key points

  • An all-answerable eval set makes 'answers confidently when it should not' completely invisible.
  • Unanswerable items are scored on abstention, not recall, and you check both the retrieval gate and the generation refusal.
  • Author them with strong distractor terms, or retrieval returns nothing and you are testing the tokenizer.
  • Keep them at roughly 15% or more of the set, alongside multi-hop items, as the coverage floor.
  • A low abstention rate splits into two causes: a no-op retrieval score gate, or a missing refusal instruction in the prompt.

D14 综合项目与复盘：多租户企业知识库问答，一张 RAG 决策地图与面试专题

• 怎么向不懂技术的业务方证明你的检索系统真的变好了？How do you convince a non-technical stakeholder that your retrieval system actually got better?

  国内高频海外高频基础#evaluation#stakeholder-communication#abstention

  分析过程 · 先想清楚再作答

  1. 这题在考沟通，但拿分点在技术判断上：你选哪几个数字给业务方看，暴露了你自己有没有看懂这些指标。把召回率、nDCG、MRR 一股脑摊出去的答法会被判成不懂受众；只说「用户反馈变好了」又会被判成没有度量。
  2. 先立一条原则：**给业务方看的必须是他们能自己判断对错的东西**。归一化折损累计增益他们没法判断，而「这一百个真实问题里，系统答对了多少、答错了多少、老老实实说查不到了多少」他们一眼就能判断。所以对外的口径应该是三个数：答对率、答错率、拒答率，而且三个加起来是一百。
  3. 关键是把**答错和拒答分开**。这一条最能建立信任：查不到就说查不到不是故障，是正确输出；真正的故障是查不到还编一段。很多团队只报「准确率」，结果一个学会了一直拒答的系统能刷出满分——所以这三个数必须并排出现，缺一个都能被骗。
  4. 然后给可核对的证据，而不是只给数字：**挑十条真实问题做前后对照**，各贴出改动前和改动后的回答，每句结论后面挂着可点开的引用。业务方点开原文核对一遍，比看任何百分比都有说服力，而且这个动作顺带完成了一次人工抽检——你自己也需要它来校准模型裁判靠不靠谱。
  5. 本课里有一条要主动说的教训：**一列全是满分说明尺子坏了**。我们的题目是从语料反向出的，字面重合度过高，平均倒数排名恒为 1.0000。这个数字拿给业务方看，只会换来一次「那你们已经完美了」的误会，而它其实是指标饱和。指标撞天花板时该做的是把题目出难一点。
  6. 可预期的追问：那怎么让业务方参与进来？答一条很实用的：让他们提供题目。把线上答错的问题一条条补进标准答案集，评估集是长出来的，而不是一次性造好的；这样每一次改进都能指着「你上次提的那个问题现在答对了」，比任何汇报都直接。

  How to reason about it · think before answering

  1. This is a communication question whose scoring hinges on technical judgement: which numbers you choose to show reveals whether you understand the metrics yourself. Dumping recall, nDCG and MRR on a business stakeholder reads as tone-deaf; saying 'user feedback improved' reads as unmeasured.
  2. Start from a principle: show them something they can adjudicate themselves. They cannot judge normalized discounted cumulative gain, but they can absolutely judge 'out of these hundred real questions, how many did it answer correctly, how many wrongly, and how many did it honestly decline'. So the external framing is three numbers — correct, wrong, declined — and they sum to one hundred.
  3. The crucial move is separating wrong from declined, and it is the fastest way to earn trust: saying 'not found' is a correct output, not a failure; the failure is inventing an answer when nothing was found. Teams that report a single 'accuracy' number can be gamed by a system that learns to decline everything, which is why all three must appear side by side.
  4. Then supply checkable evidence rather than only numbers: take ten real questions and show before-and-after answers with clickable citations on every claim. A stakeholder who opens the source and verifies one claim is more convinced than by any percentage, and the exercise doubles as the human spot-check you need anyway to calibrate whether your model judge is trustworthy.
  5. There is a lesson from this course worth volunteering: a column of perfect scores means the ruler is broken. Our questions were written backwards from the corpus, lexical overlap is unusually high, and mean reciprocal rank sits at exactly 1.0000. Showing that to a stakeholder only invites the misreading that you are already perfect, when in fact the metric has saturated. When a metric hits the ceiling, the response is to make the questions harder.
  6. Expected follow-up: how do you get the business side involved? One very practical answer: let them supply questions. Every production miss gets appended to the golden set, so the evaluation set grows rather than being built once. Then each release can point at 'the question you raised last month now answers correctly', which lands better than any status report.

  答题要点

  • 对外只用三个他们能自己判断的数：答对率、答错率、拒答率，三者相加为一百。
  • 答错和拒答必须分开——查不到就说查不到是正确输出，只报一个准确率会被「一直拒答」刷满分。
  • 配十条真实问题的前后对照，每句结论挂可点开的引用，让他们自己核对原文。
  • 主动说明指标饱和：某一列恒为满分是尺子坏了，不是系统完美，该做的是把题目出难一点。
  • 让业务方提供题目，把线上答错的问题补进标准答案集——评估集是长出来的。

  Key points

  • Externally report three numbers they can adjudicate: correct, wrong, declined — summing to one hundred.
  • Keep wrong and declined separate; a single accuracy number is gamed by a system that learns to decline everything.
  • Pair it with ten before-and-after real questions, every claim carrying a citation they can open and verify.
  • Volunteer the saturation caveat: a column of perfect scores means a broken ruler, and the fix is harder questions.
  • Let stakeholders contribute questions; append every production miss to the golden set so it grows over time.
• 如果预算只够做三件事来提升一个已有 RAG 系统的效果，你选哪三件？为什么是这三件？If you could only fund three changes to improve an existing RAG system, which three would you pick and why those three?

  国内高频海外高频进阶#prioritization#evaluation#abstention

  分析过程 · 先想清楚再作答

  1. 这题在考优先级判断，而不是知识面。答成「上重排、上混合检索、上查询改写」这类手法清单几乎必然掉分——因为它跳过了一个前提：**你凭什么知道这三件对你的系统有用？** 面试官等的就是这句话。
  2. 所以第一件必须是**建评估**，而且理由要具体到不可反驳：没有秤，剩下两件做完你也说不清是变好还是变坏；有了秤，后面每一笔钱都能算回报。而且它便宜——检索侧三个指标是纯本地计算、几秒钟、零成本，能挂进每次提交；花时间的只是给题目标答案文档那一次。顺带说清评估集的配比：多跳与无答案各占一成以上，缺了无答案那一类，一个只会硬答的系统在报表上就是满分。
  3. 第二件是**把拒答从提示词搬进代码**，这一件的性价比通常最高而最容易被跳过。提示词里写十遍「找不到就说找不到」增益接近于零；而引用编号是一个闭集，判它存不存在只要一行代码，再加一道「这句话与被引块的实质重合度」就能拦住「编号是真的、内容是假的」那一类。我们实验里的基线拒答率是 0.0%——四道语料里根本没有答案的题一道都没闭嘴，这类缺陷在只报召回率的报表上完全不可见。
  4. 第三件要**先看失败案例再决定**，这才是这道题真正的答案。看完面板你会落到其中之一：多跳题占比高就补桥接检索或改索引结构；换个说法就捞不到，说明该上向量那一路或混合检索；答案捞到了却排不进上下文，那是重排或者预算的活。**先有失败案例，再有手法**——我们试过五种高级索引结构，没有一种跑赢基线，因为我们的系统压根没有那些结构要补的短板。
  5. 为什么不选那些看起来更亮的：Agentic 检索的收益集中在多跳题上而代价摊给全部问题；「全开」所有查询侧手法在我们的实测里召回率和默认配置一模一样，模型调用却是 2.5 倍、检索次数 4.3 倍。**堆手法很容易，说清楚为什么关掉某几项才是本事。**
  6. 可预期的追问：三件做完怎么证明钱花对了？答：每一项单独开关各跑一遍，报三笔账——指标涨了多少、延迟涨了多少、钱涨了多少。只报第一笔的提案不该被批准，包括你自己的。

  How to reason about it · think before answering

  1. This tests prioritisation, not breadth. Answering with a list of techniques — add reranking, add hybrid retrieval, add query rewriting — almost always loses points, because it skips a prerequisite: how do you know those three help your system? That is precisely the sentence the interviewer is waiting for.
  2. So the first item has to be building evaluation, with a reason specific enough to be unarguable: without a scale, you cannot tell whether the other two helped or hurt; with one, every subsequent spend has a measurable return. It is also cheap — the three retrieval metrics are pure local computation, run in seconds, cost nothing, and can gate every commit; the only real effort is labelling answer documents once. Include the composition rule: multi-hop and unanswerable each above ten percent, because without the unanswerable class a system that only ever guesses scores perfectly on your report.
  3. Second, move abstention out of the prompt and into code — usually the best return per unit of effort, and the item most often skipped. Writing 'say you don't know' ten times in a prompt buys almost nothing. Citation numbers are a closed set, so checking existence is one line, and adding a substantive-overlap check catches the harder forgery where the number is real but the content is not. Our baseline abstention rate was 0.0 percent: four questions with no answer in the corpus, zero of them declined — a defect that is completely invisible on a report that only shows recall.
  4. Third, look at the failure cases before deciding, which is the actual answer to this question. After reading the panel you land on one of a few branches: a high share of multi-hop means bridging retrieval or a different index structure; queries that miss when phrased differently mean you need the vector route or hybrid retrieval; answers retrieved but never packed into context means reranking or budget. Failure cases first, technique second — we tried five advanced index structures and not one beat the baseline, because our system simply did not have the weakness they address.
  5. Why not the flashier options: agentic retrieval concentrates its gains on multi-hop while spreading cost across every question, and in our measurements turning on every query-side technique produced exactly the same recall as the default configuration while using 2.5 times the model calls and 4.3 times the retrievals. Stacking techniques is easy; explaining why you switched several off is the skill.
  6. Expected follow-up: once the three are done, how do you prove the money was well spent? Toggle each one individually and report three ledgers — how much the metric moved, how much latency moved, how much cost moved. A proposal that reports only the first should not be approved, including your own.

  答题要点

  • 第一件是建评估：没有秤，另外两件做完也说不清变好还是变坏；检索侧指标零成本可挂进每次提交。
  • 评估集必须含无答案那一类，否则一个只会硬答的系统在报表上就是满分。
  • 第二件是把拒答从提示词搬进代码：编号是闭集，再加实质重合度就能拦住「编号真、内容假」。
  • 第三件由失败案例决定，不由手法清单决定——先有失败案例，再有索引结构或检索手法。
  • 每一项单独开关跑一遍并报三笔账：指标、延迟、钱。只报第一笔的提案不该被批准。

  Key points

  • First, build evaluation: without a scale the other two changes are unverifiable, and the retrieval metrics are cheap enough to gate every commit.
  • The golden set must include unanswerable questions, or a system that only ever guesses scores perfectly on your report.
  • Second, move abstention from the prompt into code: citation numbers are a closed set, and a substantive-overlap check catches real-number-fake-content forgeries.
  • Third is chosen by the failure cases, not by a list of techniques — failure cases first, index structure or retrieval trick second.
  • Toggle each change individually and report three ledgers: metric, latency, cost. A proposal reporting only the first should not be approved.