面试题库

14 天 RAG：从检索到可信回答

D8 评估先行：搭 golden set、算召回与排序指标、用模型当裁判判忠实度

• 让你从零给一个公司知识库的 RAG 系统建评估集，你会怎么做？多少题才算够用？You need to build an evaluation set from scratch for a RAG system over a company knowledge base. How would you do it, and how many questions are enough?

  国内高频海外高频进阶#evaluation#golden-set#rag

  分析过程 · 先想清楚再作答

  1. 这题的区分度在「出题方向」和「规模的理由」两处。开口就说「找几百个用户真实问题」的，多半没真做过——真实问题的答案在哪篇文档里，没人标得出来。
  2. 先给方向：从语料反向出题，打开每一篇读它能回答什么，出题的那一刻答案文档就已经确定了，标注成本几乎为零。反方向（先想问题再找答案）会得到一堆自己都不知道答案的题。
  3. 再给结构：每题记问题、答案文档列表、类型三个字段；类型至少分单文档、多跳、无答案三类，并说明多跳必须全部答案文档命中才算命中，无答案不参与召回率而是考拒答。
  4. 规模的理由要给出来，不能只报一个数字：20 题能把「完全不能用」和「基本能用」分开，够做冒烟；100 到 200 题才有资格判断「涨了两个点」是真的还是噪声。上线之后每次线上出问题就把那个问题补进集合——评估集是长出来的。
  5. 补一句成本与保鲜：出题是人力活，20 题两小时是正常量级；语料更新后要复核答案文档还在不在，否则集合会悄悄腐烂，指标下跌你会误以为是系统坏了。
  6. 可预期的追问是「怎么防止评估集被过拟合」。答案是留一份不参与调优的保留集，并且定期从线上真实问题里补充新题，只用来验收不用来调参。

  How to reason about it · think before answering

  1. The discriminator here is the direction you generate questions in, and whether you can justify a size rather than name one.
  2. Go corpus-first: read each document and write the questions it can answer. The answer document is fixed at authoring time, so labeling is nearly free. Question-first gives you items whose answers nobody can locate.
  3. Give the schema: question, answer document ids, and a type. At minimum three types - single-document, multi-hop, and unanswerable. Multi-hop counts as a hit only when every answer document makes it into the context; unanswerable items are scored on abstention, not recall.
  4. Justify the size: 20 items separate 'broken' from 'usable' and are enough for a smoke gate; 100 to 200 are needed before a two-point delta means anything. Then grow the set - every production failure becomes a new item.
  5. Mention cost and decay: roughly two hours for 20 items, and answer labels must be rechecked whenever the corpus changes, or the set rots and you misread the drop as a system regression.
  6. Expected follow-up: how do you avoid overfitting to the eval set? Keep a held-out slice that never informs tuning, and refresh it from real production questions.

  答题要点

  • 从语料反向出题，出题时答案文档就已确定，标注成本最低。
  • 每题标类型：单文档、多跳、无答案，三类缺一不可。
  • 多跳要求全部答案文档命中；无答案不算召回率，考的是拒答。
  • 20 题够冒烟，100 到 200 题才能判断小幅变化；线上故障持续补题。
  • 留一份不参与调优的保留集，防止对评估集过拟合。

  Key points

  • Author corpus-first so the answer document is known at authoring time.
  • Label every item with a type: single-document, multi-hop, unanswerable.
  • Multi-hop requires all answer documents; unanswerable items score abstention, not recall.
  • 20 items for a smoke gate, 100 to 200 to trust small deltas, and keep growing it from production failures.
  • Hold out a slice that never informs tuning to avoid overfitting the set.
• 召回率、平均倒数排名、归一化折损累计增益，这三个检索指标分别在什么故障下会先掉下来？只盯一个会漏掉什么？Recall, mean reciprocal rank, and normalized discounted cumulative gain - which failure mode does each one catch first, and what do you miss by watching only one?

  国内高频海外高频进阶#retrieval-metrics#evaluation#ranking

  分析过程 · 先想清楚再作答

  1. 这题考的是「知不知道指标之间的盲区」，不是背定义。能把三者按「有没有 / 靠不靠前 / 整体好不好」分层的，基本就答对了一半。
  2. 推导链是这样的：召回率是布尔的——答案文档在不在最终上下文里。它对「压根没捞到」最敏感，但答案从第 1 名掉到第 8 名它一动不动，只要还在预算内。
  3. 倒数排名只看第一条相关结果的名次，所以「答案还在但被挤到后面」它立刻掉。反过来它有个盲区：前十条里有一条命中还是五条命中，它给的分完全一样。
  4. 归一化折损累计增益把前 k 名里每一条相关结果都按名次折算再累加，所以它对「整体排序质量」敏感，是重排最直接的优化目标。它的盲区是不告诉你「有没有」——召回率为零时它也是零，看不出是没捞到还是排得差。
  5. 结论：三个一起看才能定位故障层。召回率掉说明检索或切块出了问题，要动召回策略；召回率不动而倒数排名掉，说明排序退化，该上重排；两者都稳而 nDCG 掉，说明前几名里混进了更多噪声。
  6. 可预期的追问是「指标顶格了怎么办」。真实答案是把题目做难：指标撞天花板说明评估集失去区分度，这时候继续优化系统是在瞎调。

  How to reason about it · think before answering

  1. This tests whether you know each metric's blind spot, not whether you can recite definitions. Layer them as 'did it show up / how high / how good overall' and you are halfway there.
  2. Recall is boolean: is the answer document in the final context. It catches 'never retrieved', but it does not move when the answer slips from rank 1 to rank 8, as long as it still fits the budget.
  3. MRR looks only at the rank of the first relevant hit, so ranking degradation shows up immediately. Its blind spot: one relevant item in the top ten scores exactly the same as five.
  4. nDCG discounts every relevant hit in the top k by its position, so it tracks overall ranking quality and is the direct optimization target for reranking. Its blind spot is existence - it is zero both when nothing was retrieved and when ranking is terrible.
  5. Conclusion: together they localize the failure. Recall drops means retrieval or chunking; recall flat but MRR down means ranking degraded, reach for a reranker; both stable but nDCG down means more noise crept into the top results.
  6. Expected follow-up: what if a metric saturates? Make the questions harder - a saturated metric means the eval set lost its discriminative power, and further tuning is blind.

  答题要点

  • 召回率管「有没有进上下文」，对完全没捞到最敏感，对名次变化不敏感。
  • 平均倒数排名管「第一条排第几」，对排序退化最敏感，但分不清命中一条还是五条。
  • 归一化折损累计增益管「前 k 名整体质量」，是重排的直接优化目标，但看不出有没有。
  • 三者组合才能定位故障在召回层、排序层还是噪声层。
  • 命中口径要说清：按 token 预算装上下文，不是按固定条数取前 k。

  Key points

  • Recall answers 'did it make it into the context', sensitive to total misses, blind to rank shifts.
  • MRR answers 'how high is the first hit', sensitive to ranking degradation, blind to how many hits there are.
  • nDCG answers 'how good is the top k overall', the direct target for reranking, blind to existence.
  • Only the combination localizes the failure to retrieval, ranking, or noise.
  • State the hit criterion: context is packed against a token budget, not a fixed top-k.
• 用模型当裁判来评 RAG 的忠实度，有哪些系统性偏差？你怎么发现它们、又怎么证明你的裁判可信？What systematic biases does an LLM judge have when scoring RAG faithfulness, and how do you detect them and prove your judge is trustworthy?

  国内高频海外高频深入#llm-as-judge#evaluation#faithfulness

  分析过程 · 先想清楚再作答

  1. 这题的题眼在后半句。能背出「位置偏好、长度偏好、自我偏好」三个名词的人很多，能说出「怎么证明可信」的很少——面试官要的是后者。
  2. 先把三个偏差和各自的缓解手段一一对应：位置偏好用逐条独立打分代替两两比较，非要比较就交换顺序跑两遍、结论不一致判平局；长度偏好用逐句判定加比例计分，写得越长分母越大，长度红利自动消失；自我偏好用跨供应商或跨档位的模型评判，生成和评判不同源。
  3. 再补两条提示词层面的：给死评分锚点，1.0 / 0.6 / 0.3 各自是什么必须写明，否则同一份输入不同天给的分都不一样；强制结构化输出并要求把没支撑的句子原样列出，这是人工复核的抓手。
  4. 证明可信只有一条路：人工抽检算一致率。分层抽十到三十条——各类型都要有、命中和没命中都要有、裁判给高分和低分都要有，只判一个二元问题（有没有材料外的内容），跟裁判的结论比对。低于 0.8 就不能拿它的分数做拦合并这类决策。
  5. 一个能加分的细节：一致率很高不一定是好消息。如果抽的十条都是「答案原样抄自材料」的简单题，判对是理所当然的，这时候 100% 说明的是抽检没难度，不是裁判可靠。
  6. 可预期的追问是「裁判本身会不会坏」。答案是给裁判写探针：喂几组已知正确答案的输入（照抄材料的、明显编造的），每次跑评估都验一遍——评估系统坏掉的方式最阴险，分数照常输出，只是不再有意义。

  How to reason about it · think before answering

  1. The second half of the question is the discriminator. Plenty of people can name position, length, and self-preference bias; few can say how they prove the judge is trustworthy.
  2. Pair each bias with its mitigation: position bias - score pointwise instead of pairwise, and if you must compare, swap the order and call disagreement a tie; length bias - decompose into claims and score a ratio, so a longer answer grows its own denominator; self-preference - judge with a different vendor or tier than the generator.
  3. Add two prompt-level requirements: fixed rubric anchors (spell out what 1.0, 0.6 and 0.3 mean, or the same input scores differently on different days) and forced structured output that quotes the unsupported sentences verbatim, which is what makes human review possible.
  4. Proving trust has exactly one route: human spot-checks and an agreement rate. Stratify ten to thirty items across types, hits and misses, high and low judge scores; answer one binary question only - is anything here not in the material - and compare. Below 0.8 the judge's scores cannot gate a merge.
  5. A detail that scores points: a very high agreement rate may mean your spot-check was too easy. If all ten sampled answers copy the material verbatim, agreeing is trivial and 100% says nothing about the judge.
  6. Expected follow-up: can the judge itself break? Add probes - fixed inputs with known verdicts, one faithful and one obviously fabricated, checked on every run. An evaluation system fails silently: the numbers keep coming, they just stop meaning anything.

  答题要点

  • 三个偏差：位置偏好、偏爱长答案、自己评自己，各自有对应的缓解手段。
  • 逐条独立打分代替两两比较；逐句判定按比例计分抵消长度红利；生成与评判不同源。
  • 提示词要给死评分锚点，并强制结构化输出、列出没支撑的句子。
  • 可信度靠人工分层抽检算一致率，低于 0.8 不能用它做拦合并的决策。
  • 给裁判本身写探针，每次跑评估都验一遍它有没有坏。

  Key points

  • Three biases: position, verbosity, and self-preference, each with a matching mitigation.
  • Score pointwise rather than pairwise; decompose into claims and score a ratio to kill the length premium; never let the generator judge itself.
  • Pin rubric anchors in the prompt and force structured output that quotes unsupported sentences.
  • Establish trust through stratified human spot-checks and an agreement rate; below 0.8 the judge cannot gate merges.
  • Add probes with known verdicts so a broken judge is caught on every run.
• RAG 的评估集里为什么一定要放语料里没有答案的问题？不放会掩盖什么？Why must a RAG evaluation set include questions the corpus cannot answer, and what does leaving them out hide?

  国内高频海外高频基础#evaluation#abstention#golden-set

  分析过程 · 先想清楚再作答

  1. 这题看着简单，实际是在问「你有没有想过评估集本身也会说谎」。答成「为了测试拒答功能」只算及格，答出「不放会让某个故障在报表上完全不可见」才是满分。
  2. 推导只有一步：一个只会硬答的系统，在只有可答问题的评估集上能拿到很高的分——它每次都塞材料给模型，模型每次都编一段话，而评估集根本没有「应该拒答」这一栏。于是最危险的故障在报表上是不存在的。
  3. 结论：无答案问题是唯一能让「乱编」显形的东西。它不参与召回率，它的指标是拒答率——检索侧有没有把不够格的候选全挡下来，生成侧有没有真的说出「资料里没有」。
  4. 出题上有个必须说的细节：无答案问题必须留强干扰词，比如问「网页端支持哪些浏览器」而语料里恰好有一句「提交工单请附上浏览器与版本」。没有干扰词的无答案题检索器一条都捞不到，你测出来的是分词器不是系统。
  5. 可预期的追问是「拒答率低怎么办」。分两层查：先看检索侧的门槛是不是形同虚设（分数阈值定得太低，不相干的块也过关），再看生成侧的提示词有没有明确的拒答指令，两层都要有，只靠提示词兜是不牢的。

  How to reason about it · think before answering

  1. It looks easy but really asks whether you have considered that the eval set itself can lie. 'To test the refusal path' is a pass; 'without them the worst failure is invisible in the report' is a full mark.
  2. The derivation is one step: a system that always answers scores well on a set of answerable questions only. It stuffs context in, the model writes something, and the set has no column for 'should have refused'. The most dangerous failure simply does not appear.
  3. Conclusion: unanswerable questions are the only thing that makes fabrication visible. They are excluded from recall and scored on abstention instead - did retrieval gate out every weak candidate, and did generation actually say the material does not cover this.
  4. One authoring detail worth stating: unanswerable questions need strong distractor terms. Ask which browsers the web client supports when the corpus only says 'attach your browser and version when filing a ticket'. Without distractors retrieval returns nothing and you are testing your tokenizer, not your system.
  5. Expected follow-up: what if the abstention rate is low? Check two layers - whether the retrieval score gate is effectively a no-op, and whether the generation prompt carries an explicit refusal instruction. You need both; a prompt alone is not a reliable gate.

  答题要点

  • 只有可答问题的评估集，会让「不知道也硬答」这个故障完全不可见。
  • 无答案问题不算召回率，它的指标是拒答率，检索侧和生成侧各看一层。
  • 出题必须留强干扰词，否则检索器一条都捞不到，测的是分词器。
  • 建议无答案题占比不低于评估集的一成五，跟多跳题一起构成覆盖度底线。
  • 拒答率低要分两层查：检索门槛是否形同虚设，生成提示词有没有拒答指令。

  Key points

  • An all-answerable eval set makes 'answers confidently when it should not' completely invisible.
  • Unanswerable items are scored on abstention, not recall, and you check both the retrieval gate and the generation refusal.
  • Author them with strong distractor terms, or retrieval returns nothing and you are testing the tokenizer.
  • Keep them at roughly 15% or more of the set, alongside multi-hop items, as the coverage floor.
  • A low abstention rate splits into two causes: a no-op retrieval score gate, or a missing refusal instruction in the prompt.

D9 混合检索与重排：两路召回、倒数排名融合，再用交叉编码器把前几名重新排一遍

• 你把纯向量检索换成了混合检索加重排，上线之后评估指标反而掉了。你会怎么排查？You replaced pure vector retrieval with hybrid search plus reranking, and after shipping it your eval metrics went down. How do you investigate?

  国内高频海外高频深入#hybrid-search#evaluation

  分析过程 · 先想清楚再作答

  1. 这题考的是你有没有真的做过分阶段归因。答「调一下权重再看看」就输了——那是在猜，不是在查。
  2. 第一步是拆档跑，不是改代码：纯关键词、纯向量、混合、混合加重排四档在**同一份评估集、同一个上下文预算**下各跑一遍。指标掉在哪一档就在哪一档找原因，这一步能立刻区分「融合坏了」和「重排坏了」。
  3. 第二步问一个具体问题：掉的是召回率还是排序指标？召回率掉说明答案根本没进上下文，是候选池或者预算的问题；排序指标掉而召回率没动，说明答案还在、只是被挤到了后面，那是融合权重或重排模型的问题。这两类故障的解法完全不同。
  4. 第三个常见根因是召回深度。每路取多少条这个旋钮方向反直觉：取深了不是更保险，是把噪声也一起投了票。我在一份 134 块的语料上实测过，每路从取 50 收到取 5，混合那一档的召回率从 87.5% 回到 93.8%、多跳档从 50% 回到 75%，而 nDCG 反而掉了近 0.1——两个指标会打架，先想清楚业务要哪个。
  5. 第四个根因是评估口径被悄悄改了。上下文预算、命中判定、候选池深度只要动过一个，新旧数字就不可比，这时候「掉了」可能根本不是真的掉了。
  6. 可预期的追问：怎么防止下次再踩？答把四档对照做成一条命令、把上一版报告存成基线、指标退步就以非 0 退出码拦住合并——这就是评估要先于优化的原因。

  How to reason about it · think before answering

  1. This question tests whether you have actually done stage-by-stage attribution. Answering `I would tune the weights and see` loses — that is guessing, not investigating.
  2. Step one is to run the stages apart, not to change code: pure keyword, pure vector, hybrid, and hybrid plus rerank, all on the **same eval set with the same context budget**. Whichever stage the drop appears in is where you look, and this alone separates `fusion is broken` from `reranking is broken`.
  3. Step two asks a specific question: did recall drop, or did the ranking metrics drop? A recall drop means the answer never entered the context at all — a candidate-pool or budget problem. Ranking metrics dropping while recall holds means the answer is still there but pushed down — a fusion-weight or rerank-model problem. The two failures have completely different fixes.
  4. A third common root cause is recall depth. This knob runs against intuition: going deeper is not safer, it lets noise vote too. On a 134-chunk corpus I measured that narrowing each route from 50 to 5 took hybrid recall from 87.5% back to 93.8% and multi-hop from 50% to 75%, while nDCG fell by almost 0.1. The metrics fight each other, so decide which one the product needs first.
  5. A fourth root cause is that the eval protocol quietly changed. Touch the context budget, the hit rule, or the candidate depth, and the old and new numbers stop being comparable — in which case the `drop` may not be a drop at all.
  6. Expected follow-up: how do you avoid this next time? Make the four-way comparison a single command, store the previous report as a baseline, and fail the build with a non-zero exit code on regression. That is precisely why evaluation comes before optimization.

  答题要点

  • 先拆档跑四种配置，在同一份评估集和同一个上下文预算下归因，不要一上来就调参。
  • 区分召回率掉与排序指标掉：前者是候选池或预算问题，后者是融合或重排问题。
  • 查召回深度：每路取太深会把噪声也投进融合，收窄反而可能救回召回率。
  • 确认评估口径没被改：预算、命中判定、候选池深度动过一个，新旧数字就不可比。
  • 把四档对照固化成一条命令加一份基线报告，指标退步直接拦住合并。

  Key points

  • Run all four configurations separately for attribution, on one eval set with one context budget, before touching any parameter.
  • Separate a recall drop from a ranking drop: the first is a candidate-pool or budget issue, the second is a fusion or rerank issue.
  • Check recall depth: taking too many per route lets noise vote, and narrowing it can bring recall back.
  • Confirm the eval protocol did not change; touching budget, hit rule, or candidate depth makes old and new numbers incomparable.
  • Freeze the four-way comparison into one command plus a baseline report, and block merges on regression.

D11 高级索引：父子文档、摘要索引、上下文检索，以及树状聚合与图检索的取舍

• 同一份语料建了三套索引，检索时你怎么决定走哪一套？You have built three different indexes over the same corpus. How do you decide which one a query goes to?

  国内高频海外高频进阶#index-routing#evaluation#architecture

  分析过程 · 先想清楚再作答

  1. 这题是送分还是丢分，取决于你有没有先反问一句『真的需要三套吗』。上来就答路由策略的人，默认了一个没被验证的前提。
  2. 第一步是承认多数情况下答案是『都不走，走默认那套』。我们在 30 篇语料上把五种索引结构各测一遍，**召回率全部停在 93.8%，没有一种跑赢基线**；唯一动了的是 nDCG@10（块头把它从 0.6438 抬到 0.7218），而两段式的摘要索引还掉到了 87.5%。每种结构补的都是一个特定短板，你没有那个短板时它只带来成本。
  3. 第二步才是路由，而判据不是『哪套准』——那是离线评估该回答的问题，不是运行时能知道的。运行时能拿到的只有**问题的形状**：细节型（答案落在某一段）、概括型（要全库的一个概括）、多跳型（要跨实体串联）。按形状分流，正好对应块级索引、树状聚合索引、图索引。
  4. 实现上就是一个轻量意图分类器，跟前一天的意图路由是同一套东西，不必再造一个。分类结果作为元数据带进请求，方便事后拿评估集回看分错了多少。
  5. 兜底策略要说清楚：分类错了**回落到默认那一套**，不要并行全查一遍再融合。并行看着稳，实际上把延迟和成本按索引套数翻倍，而多出来的那两路大概率一条都进不了上下文预算。
  6. 可预期的追问是『怎么知道分类器分对了』。答案是把路由决策记进日志，定期拿标准答案集回放：对每个问题分别走三套索引，看分类器选的那套是不是指标最好的那套。这是一个能持续跑的离线作业，不需要人工标注。

  How to reason about it · think before answering

  1. Whether this is an easy point or a lost one depends on whether you first ask 'do we actually need three?'. Jumping straight to routing accepts an unverified premise.
  2. Step one is admitting the answer is usually 'none of them — use the default'. Across 30 documents we measured five index structures and every one landed at 93.8% recall, none beating the baseline. The only metric that moved was nDCG@10, which headers lifted from 0.6438 to 0.7218, while the two-stage summary index fell to 87.5%. Each structure patches one specific weakness; without that weakness it is pure overhead.
  3. Step two is routing, and the criterion is not 'which index is more accurate' — that is an offline evaluation question, not something you know at request time. What you do have at request time is the shape of the question: detail-seeking, summarizing, or entity-chaining. Those map onto the chunk index, the tree-summary index and the graph index.
  4. Implementation is a lightweight intent classifier — the same one from the previous day's intent routing, no need to invent another. Carry the decision as request metadata so you can replay it later.
  5. Spell out the fallback: on a misclassification, fall back to the default index rather than fanning out across all three and fusing. Fan-out looks safe but multiplies latency and cost by the number of indexes, and the extra routes usually never make it into the context budget anyway.
  6. Expect 'how do you know the classifier is right'. Log every routing decision and replay the golden set periodically: run each question through all three indexes and check whether the classifier picked the best-scoring one. It is a standing offline job that needs no human labelling.

  答题要点

  • 先反问是否真需要三套：实测五种索引结构召回率全部持平在 93.8%，没有对应短板就是纯成本。
  • 运行时的判据是问题的形状——细节型、概括型、多跳型，分别对应块级、树状摘要、图索引。
  • 复用前一天的意图路由做分类，把路由决策记进请求元数据。
  • 分类错了回落到默认索引，不要并行全查再融合——延迟和成本按套数翻倍。
  • 用标准答案集定期回放，检验分类器选的那套是不是指标最好的那套。

  Key points

  • First challenge the premise: all five index structures landed at the same 93.8% recall in our measurement, so an index without a matching weakness is pure cost.
  • At request time the usable signal is question shape — detail, summary, or entity-chaining — mapping to chunk, tree-summary and graph indexes.
  • Reuse the previous day's intent router for classification and record the routing decision as request metadata.
  • Fall back to the default index on misclassification instead of fanning out and fusing, which multiplies latency and cost.
  • Replay the golden set periodically to check whether the classifier picks the best-scoring index.

D14 综合项目与复盘：多租户企业知识库问答，一张 RAG 决策地图与面试专题

• 怎么向不懂技术的业务方证明你的检索系统真的变好了？How do you convince a non-technical stakeholder that your retrieval system actually got better?

  国内高频海外高频基础#evaluation#stakeholder-communication#abstention

  分析过程 · 先想清楚再作答

  1. 这题在考沟通，但拿分点在技术判断上：你选哪几个数字给业务方看，暴露了你自己有没有看懂这些指标。把召回率、nDCG、MRR 一股脑摊出去的答法会被判成不懂受众；只说「用户反馈变好了」又会被判成没有度量。
  2. 先立一条原则：**给业务方看的必须是他们能自己判断对错的东西**。归一化折损累计增益他们没法判断，而「这一百个真实问题里，系统答对了多少、答错了多少、老老实实说查不到了多少」他们一眼就能判断。所以对外的口径应该是三个数：答对率、答错率、拒答率，而且三个加起来是一百。
  3. 关键是把**答错和拒答分开**。这一条最能建立信任：查不到就说查不到不是故障，是正确输出；真正的故障是查不到还编一段。很多团队只报「准确率」，结果一个学会了一直拒答的系统能刷出满分——所以这三个数必须并排出现，缺一个都能被骗。
  4. 然后给可核对的证据，而不是只给数字：**挑十条真实问题做前后对照**，各贴出改动前和改动后的回答，每句结论后面挂着可点开的引用。业务方点开原文核对一遍，比看任何百分比都有说服力，而且这个动作顺带完成了一次人工抽检——你自己也需要它来校准模型裁判靠不靠谱。
  5. 本课里有一条要主动说的教训：**一列全是满分说明尺子坏了**。我们的题目是从语料反向出的，字面重合度过高，平均倒数排名恒为 1.0000。这个数字拿给业务方看，只会换来一次「那你们已经完美了」的误会，而它其实是指标饱和。指标撞天花板时该做的是把题目出难一点。
  6. 可预期的追问：那怎么让业务方参与进来？答一条很实用的：让他们提供题目。把线上答错的问题一条条补进标准答案集，评估集是长出来的，而不是一次性造好的；这样每一次改进都能指着「你上次提的那个问题现在答对了」，比任何汇报都直接。

  How to reason about it · think before answering

  1. This is a communication question whose scoring hinges on technical judgement: which numbers you choose to show reveals whether you understand the metrics yourself. Dumping recall, nDCG and MRR on a business stakeholder reads as tone-deaf; saying 'user feedback improved' reads as unmeasured.
  2. Start from a principle: show them something they can adjudicate themselves. They cannot judge normalized discounted cumulative gain, but they can absolutely judge 'out of these hundred real questions, how many did it answer correctly, how many wrongly, and how many did it honestly decline'. So the external framing is three numbers — correct, wrong, declined — and they sum to one hundred.
  3. The crucial move is separating wrong from declined, and it is the fastest way to earn trust: saying 'not found' is a correct output, not a failure; the failure is inventing an answer when nothing was found. Teams that report a single 'accuracy' number can be gamed by a system that learns to decline everything, which is why all three must appear side by side.
  4. Then supply checkable evidence rather than only numbers: take ten real questions and show before-and-after answers with clickable citations on every claim. A stakeholder who opens the source and verifies one claim is more convinced than by any percentage, and the exercise doubles as the human spot-check you need anyway to calibrate whether your model judge is trustworthy.
  5. There is a lesson from this course worth volunteering: a column of perfect scores means the ruler is broken. Our questions were written backwards from the corpus, lexical overlap is unusually high, and mean reciprocal rank sits at exactly 1.0000. Showing that to a stakeholder only invites the misreading that you are already perfect, when in fact the metric has saturated. When a metric hits the ceiling, the response is to make the questions harder.
  6. Expected follow-up: how do you get the business side involved? One very practical answer: let them supply questions. Every production miss gets appended to the golden set, so the evaluation set grows rather than being built once. Then each release can point at 'the question you raised last month now answers correctly', which lands better than any status report.

  答题要点

  • 对外只用三个他们能自己判断的数：答对率、答错率、拒答率，三者相加为一百。
  • 答错和拒答必须分开——查不到就说查不到是正确输出，只报一个准确率会被「一直拒答」刷满分。
  • 配十条真实问题的前后对照，每句结论挂可点开的引用，让他们自己核对原文。
  • 主动说明指标饱和：某一列恒为满分是尺子坏了，不是系统完美，该做的是把题目出难一点。
  • 让业务方提供题目，把线上答错的问题补进标准答案集——评估集是长出来的。

  Key points

  • Externally report three numbers they can adjudicate: correct, wrong, declined — summing to one hundred.
  • Keep wrong and declined separate; a single accuracy number is gamed by a system that learns to decline everything.
  • Pair it with ten before-and-after real questions, every claim carrying a citation they can open and verify.
  • Volunteer the saturation caveat: a column of perfect scores means a broken ruler, and the fix is harder questions.
  • Let stakeholders contribute questions; append every production miss to the golden set so it grows over time.
• 如果预算只够做三件事来提升一个已有 RAG 系统的效果，你选哪三件？为什么是这三件？If you could only fund three changes to improve an existing RAG system, which three would you pick and why those three?

  国内高频海外高频进阶#prioritization#evaluation#abstention

  分析过程 · 先想清楚再作答

  1. 这题在考优先级判断，而不是知识面。答成「上重排、上混合检索、上查询改写」这类手法清单几乎必然掉分——因为它跳过了一个前提：**你凭什么知道这三件对你的系统有用？** 面试官等的就是这句话。
  2. 所以第一件必须是**建评估**，而且理由要具体到不可反驳：没有秤，剩下两件做完你也说不清是变好还是变坏；有了秤，后面每一笔钱都能算回报。而且它便宜——检索侧三个指标是纯本地计算、几秒钟、零成本，能挂进每次提交；花时间的只是给题目标答案文档那一次。顺带说清评估集的配比：多跳与无答案各占一成以上，缺了无答案那一类，一个只会硬答的系统在报表上就是满分。
  3. 第二件是**把拒答从提示词搬进代码**，这一件的性价比通常最高而最容易被跳过。提示词里写十遍「找不到就说找不到」增益接近于零；而引用编号是一个闭集，判它存不存在只要一行代码，再加一道「这句话与被引块的实质重合度」就能拦住「编号是真的、内容是假的」那一类。我们实验里的基线拒答率是 0.0%——四道语料里根本没有答案的题一道都没闭嘴，这类缺陷在只报召回率的报表上完全不可见。
  4. 第三件要**先看失败案例再决定**，这才是这道题真正的答案。看完面板你会落到其中之一：多跳题占比高就补桥接检索或改索引结构；换个说法就捞不到，说明该上向量那一路或混合检索；答案捞到了却排不进上下文，那是重排或者预算的活。**先有失败案例，再有手法**——我们试过五种高级索引结构，没有一种跑赢基线，因为我们的系统压根没有那些结构要补的短板。
  5. 为什么不选那些看起来更亮的：Agentic 检索的收益集中在多跳题上而代价摊给全部问题；「全开」所有查询侧手法在我们的实测里召回率和默认配置一模一样，模型调用却是 2.5 倍、检索次数 4.3 倍。**堆手法很容易，说清楚为什么关掉某几项才是本事。**
  6. 可预期的追问：三件做完怎么证明钱花对了？答：每一项单独开关各跑一遍，报三笔账——指标涨了多少、延迟涨了多少、钱涨了多少。只报第一笔的提案不该被批准，包括你自己的。

  How to reason about it · think before answering

  1. This tests prioritisation, not breadth. Answering with a list of techniques — add reranking, add hybrid retrieval, add query rewriting — almost always loses points, because it skips a prerequisite: how do you know those three help your system? That is precisely the sentence the interviewer is waiting for.
  2. So the first item has to be building evaluation, with a reason specific enough to be unarguable: without a scale, you cannot tell whether the other two helped or hurt; with one, every subsequent spend has a measurable return. It is also cheap — the three retrieval metrics are pure local computation, run in seconds, cost nothing, and can gate every commit; the only real effort is labelling answer documents once. Include the composition rule: multi-hop and unanswerable each above ten percent, because without the unanswerable class a system that only ever guesses scores perfectly on your report.
  3. Second, move abstention out of the prompt and into code — usually the best return per unit of effort, and the item most often skipped. Writing 'say you don't know' ten times in a prompt buys almost nothing. Citation numbers are a closed set, so checking existence is one line, and adding a substantive-overlap check catches the harder forgery where the number is real but the content is not. Our baseline abstention rate was 0.0 percent: four questions with no answer in the corpus, zero of them declined — a defect that is completely invisible on a report that only shows recall.
  4. Third, look at the failure cases before deciding, which is the actual answer to this question. After reading the panel you land on one of a few branches: a high share of multi-hop means bridging retrieval or a different index structure; queries that miss when phrased differently mean you need the vector route or hybrid retrieval; answers retrieved but never packed into context means reranking or budget. Failure cases first, technique second — we tried five advanced index structures and not one beat the baseline, because our system simply did not have the weakness they address.
  5. Why not the flashier options: agentic retrieval concentrates its gains on multi-hop while spreading cost across every question, and in our measurements turning on every query-side technique produced exactly the same recall as the default configuration while using 2.5 times the model calls and 4.3 times the retrievals. Stacking techniques is easy; explaining why you switched several off is the skill.
  6. Expected follow-up: once the three are done, how do you prove the money was well spent? Toggle each one individually and report three ledgers — how much the metric moved, how much latency moved, how much cost moved. A proposal that reports only the first should not be approved, including your own.

  答题要点

  • 第一件是建评估：没有秤，另外两件做完也说不清变好还是变坏；检索侧指标零成本可挂进每次提交。
  • 评估集必须含无答案那一类，否则一个只会硬答的系统在报表上就是满分。
  • 第二件是把拒答从提示词搬进代码：编号是闭集，再加实质重合度就能拦住「编号真、内容假」。
  • 第三件由失败案例决定，不由手法清单决定——先有失败案例，再有索引结构或检索手法。
  • 每一项单独开关跑一遍并报三笔账：指标、延迟、钱。只报第一笔的提案不该被批准。

  Key points

  • First, build evaluation: without a scale the other two changes are unverifiable, and the retrieval metrics are cheap enough to gate every commit.
  • The golden set must include unanswerable questions, or a system that only ever guesses scores perfectly on your report.
  • Second, move abstention from the prompt into code: citation numbers are a closed set, and a substantive-overlap check catches real-number-fake-content forgeries.
  • Third is chosen by the failure cases, not by a list of techniques — failure cases first, index structure or retrieval trick second.
  • Toggle each change individually and report three ledgers: metric, latency, cost. A proposal reporting only the first should not be approved.