面试题库

14 天 RAG：从检索到可信回答

D14 综合项目与复盘：多租户企业知识库问答，一张 RAG 决策地图与面试专题

• 怎么向不懂技术的业务方证明你的检索系统真的变好了？How do you convince a non-technical stakeholder that your retrieval system actually got better?

  国内高频海外高频基础#evaluation#stakeholder-communication#abstention

  分析过程 · 先想清楚再作答

  1. 这题在考沟通，但拿分点在技术判断上：你选哪几个数字给业务方看，暴露了你自己有没有看懂这些指标。把召回率、nDCG、MRR 一股脑摊出去的答法会被判成不懂受众；只说「用户反馈变好了」又会被判成没有度量。
  2. 先立一条原则：**给业务方看的必须是他们能自己判断对错的东西**。归一化折损累计增益他们没法判断，而「这一百个真实问题里，系统答对了多少、答错了多少、老老实实说查不到了多少」他们一眼就能判断。所以对外的口径应该是三个数：答对率、答错率、拒答率，而且三个加起来是一百。
  3. 关键是把**答错和拒答分开**。这一条最能建立信任：查不到就说查不到不是故障，是正确输出；真正的故障是查不到还编一段。很多团队只报「准确率」，结果一个学会了一直拒答的系统能刷出满分——所以这三个数必须并排出现，缺一个都能被骗。
  4. 然后给可核对的证据，而不是只给数字：**挑十条真实问题做前后对照**，各贴出改动前和改动后的回答，每句结论后面挂着可点开的引用。业务方点开原文核对一遍，比看任何百分比都有说服力，而且这个动作顺带完成了一次人工抽检——你自己也需要它来校准模型裁判靠不靠谱。
  5. 本课里有一条要主动说的教训：**一列全是满分说明尺子坏了**。我们的题目是从语料反向出的，字面重合度过高，平均倒数排名恒为 1.0000。这个数字拿给业务方看，只会换来一次「那你们已经完美了」的误会，而它其实是指标饱和。指标撞天花板时该做的是把题目出难一点。
  6. 可预期的追问：那怎么让业务方参与进来？答一条很实用的：让他们提供题目。把线上答错的问题一条条补进标准答案集，评估集是长出来的，而不是一次性造好的；这样每一次改进都能指着「你上次提的那个问题现在答对了」，比任何汇报都直接。

  How to reason about it · think before answering

  1. This is a communication question whose scoring hinges on technical judgement: which numbers you choose to show reveals whether you understand the metrics yourself. Dumping recall, nDCG and MRR on a business stakeholder reads as tone-deaf; saying 'user feedback improved' reads as unmeasured.
  2. Start from a principle: show them something they can adjudicate themselves. They cannot judge normalized discounted cumulative gain, but they can absolutely judge 'out of these hundred real questions, how many did it answer correctly, how many wrongly, and how many did it honestly decline'. So the external framing is three numbers — correct, wrong, declined — and they sum to one hundred.
  3. The crucial move is separating wrong from declined, and it is the fastest way to earn trust: saying 'not found' is a correct output, not a failure; the failure is inventing an answer when nothing was found. Teams that report a single 'accuracy' number can be gamed by a system that learns to decline everything, which is why all three must appear side by side.
  4. Then supply checkable evidence rather than only numbers: take ten real questions and show before-and-after answers with clickable citations on every claim. A stakeholder who opens the source and verifies one claim is more convinced than by any percentage, and the exercise doubles as the human spot-check you need anyway to calibrate whether your model judge is trustworthy.
  5. There is a lesson from this course worth volunteering: a column of perfect scores means the ruler is broken. Our questions were written backwards from the corpus, lexical overlap is unusually high, and mean reciprocal rank sits at exactly 1.0000. Showing that to a stakeholder only invites the misreading that you are already perfect, when in fact the metric has saturated. When a metric hits the ceiling, the response is to make the questions harder.
  6. Expected follow-up: how do you get the business side involved? One very practical answer: let them supply questions. Every production miss gets appended to the golden set, so the evaluation set grows rather than being built once. Then each release can point at 'the question you raised last month now answers correctly', which lands better than any status report.

  答题要点

  • 对外只用三个他们能自己判断的数：答对率、答错率、拒答率，三者相加为一百。
  • 答错和拒答必须分开——查不到就说查不到是正确输出，只报一个准确率会被「一直拒答」刷满分。
  • 配十条真实问题的前后对照，每句结论挂可点开的引用，让他们自己核对原文。
  • 主动说明指标饱和：某一列恒为满分是尺子坏了，不是系统完美，该做的是把题目出难一点。
  • 让业务方提供题目，把线上答错的问题补进标准答案集——评估集是长出来的。

  Key points

  • Externally report three numbers they can adjudicate: correct, wrong, declined — summing to one hundred.
  • Keep wrong and declined separate; a single accuracy number is gamed by a system that learns to decline everything.
  • Pair it with ten before-and-after real questions, every claim carrying a citation they can open and verify.
  • Volunteer the saturation caveat: a column of perfect scores means a broken ruler, and the fix is harder questions.
  • Let stakeholders contribute questions; append every production miss to the golden set so it grows over time.