Interview Bank

RAG in 14 Days: From Retrieval to Trustworthy Answers

D10 Query-Side Optimization: Rewriting, Hypothetical Document Embeddings, Multi-Query, Step-Back Prompting, and Intent Routing

• What happens when intent routing misclassifies, and how would you design the fallback?意图路由判错了会怎样？你会怎么设计兜底？

  Common in ChinaCommon overseasIntermediate#intent-routing#fallback#observability

  How to reason about it · think before answering

  1. This tests whether you have thought about the direction of the error. A router is a classifier and classifiers misfire; "add more training data" is not a fallback design.
  2. Break the errors down by direction — that is the backbone of the answer. Across three routes (direct answer, single-hop, multi-hop) the six confusions carry wildly asymmetric costs. Routing a retrieval-worthy question to a direct answer leaves the model with no material at all, so it fabricates: the most expensive error. Routing chit-chat to single-hop merely wastes one retrieval. Routing multi-hop to single-hop just yields an incomplete answer.
  3. The conclusion follows: bias the fallback toward spending a little more, and default to single-hop retrieval whenever the classifier is unsure. Single-hop is the cheapest error to make, and it is recoverable — with partial material the model can still say it only found half the answer; with no material it can only invent one.
  4. Add a runtime fallback, which beats better up-front classification: after a direct-answer routing, if the draft reply contains figures, amounts or dates that need a source, fall back to retrieval and answer again; after a single-hop routing, if no candidate clears the admission gate, escalate to multi-hop or abstain. Correcting the earlier decision with the later observation is the single most useful pattern in routing systems.
  5. Mention observability: log every routing decision with the raw question, the label, and whether a fallback fired. Without that log you know neither how accurate the router is nor what to train the next version on.
  6. Expect "when should you skip routing entirely?" When chit-chat is a small share of traffic and multi-hop questions are rare, the classification call costs more than it saves. In our 30-document lab the real gain from routing was not saved retrievals but the ability to give recognised multi-hop questions a larger context budget.

  分析过程 · 先想清楚再作答

  1. 这题在考「有没有想过错误的方向」。路由是分类器，分类器一定会错；只答「多加训练数据提高准确率」的，等于没回答兜底怎么设计。
  2. 先把错误按方向拆开，这一步是整题的骨架：三条路（直接回答、单跳检索、多跳检索）两两误判，代价完全不对称。把该检索的判成直接回答，模型手里一点材料都没有，只能编，这是最贵的一种错；把闲聊判成单跳，只是白花一次检索；把多跳判成单跳，只是少查一轮、答得不全。
  3. 结论顺势就出来了：**兜底方向要偏向「多花一点钱」，判不出来一律退回单跳检索。** 单跳是三条路里错得最轻的一条，而且它的错误是可恢复的——材料不全模型还能说「资料里只查到一半」，材料为空它就只能编。
  4. 再补一层运行时兜底，比事前分类更管用：分类成直接回答之后，如果模型的回答里出现了具体数字、金额、日期这类需要出处的内容，就回退去检索一次再答；分类成单跳之后，如果检索侧一条都没过门槛，就升级走多跳或直接拒答。**用后一步的观测结果纠正前一步的判断**，这是路由系统最实用的一条设计。
  5. 还要提一句可观测性：路由的每一次判定都要落日志，带上原始问题、判定结果、后续是否发生了兜底升级。没有这份日志，你既不知道路由准不准，也没法攒出下一版的训练集。
  6. 可预期的追问是「什么时候干脆别做路由」。答：流量里闲聊占比很低、且多跳问题很少时，路由省下的钱还不够付分类调用的钱，这时候直接全部走单跳更划算——我们在 30 篇语料的实验里就看到，路由真正的收益并不在省检索，而在于认出多跳之后给它更高的上下文预算。

  Key points

  • The three routes have asymmetric error costs: sending a retrieval-worthy question to a direct answer is the worst, while routing chit-chat to single-hop only wastes one retrieval.
  • Bias the fallback toward spending more: default to single-hop whenever the classifier is unsure, since that error is the mildest and is recoverable.
  • Add runtime fallbacks: re-retrieve if a direct answer contains figures that need a source; escalate or abstain if no single-hop candidate clears the gate.
  • Log every routing decision — raw question, label, whether a fallback fired — for both monitoring and the next training set.
  • When chit-chat and multi-hop are both rare, the classification call costs more than it saves; route everything to single-hop instead.

  答题要点

  • 三条路的误判代价不对称：把该检索的判成直接回答最贵（模型没材料只能编），把闲聊判成单跳只是白花一次检索。
  • 兜底方向偏向多花钱：判不出来一律退回单跳检索，它是错得最轻且可恢复的一条路。
  • 加运行时兜底：直接回答里出现需要出处的数字就补一次检索；单跳检索一条都没过门槛就升级或拒答。
  • 每一次路由判定都落日志（原始问题、判定结果、是否触发兜底），既用于监控也用于攒下一版训练集。
  • 闲聊与多跳占比都很低时，路由省的钱付不起分类调用，直接全走单跳更划算。