Interview Bank

RAG in 14 Days: From Retrieval to Trustworthy Answers

D10 Query-Side Optimization: Rewriting, Hypothetical Document Embeddings, Multi-Query, Step-Back Prompting, and Intent Routing

• Query rewriting adds a model call per question and doubles end-to-end latency. How do you decide whether it is worth paying?上线查询改写之后每问多了一次模型调用，端到端延迟涨了一倍，你怎么判断这笔开销值不值？

  Common in ChinaCommon overseasDeep dive#cost-tradeoff#latency#query-rewriting

  How to reason about it · think before answering

  1. This question is about turning an engineering judgement into numbers. "Rewriting obviously helps quality" is a fail — the candidate never measured the gain.
  2. Start with one question that nearly settles it: which slice of traffic does the gain land on? Query rewriting buys almost nothing on single-turn questions (we measured identical metrics with it on and off across 20 single-turn items); the entire payoff is in follow-up turns. So step one is to pull the share of multi-turn sessions from production logs.
  3. Step two is to lay out all three ledgers, because one alone cannot support a decision: how much the metrics moved on a fixed golden set, how much latency grew (a rewrite is a short-output task, so a cheap fast model often costs a few hundred milliseconds rather than doubling anything), and how many extra calls were added — one model call for rewriting versus one call plus several retrievals for multi-query is a completely different cost shape.
  4. Step three is to price the cheaper variants before deciding: rewrite only when a trigger fires (short query, pronoun, ellipsis), cache rewrites per session, and run a small model instead of the main one. These usually remove most of the cost while keeping the gain.
  5. Land on a usable rule: gain times affected traffic share, divided by added latency and cost, ranked against your other candidate optimisations. Rewriting usually ranks high because its failure mode is a false "not found" immediately after the user's own question — an abandonment-grade experience bug, not a few metric points.
  6. Expect "what if the latency genuinely is unacceptable?" Fire the rewrite and the first retrieval in parallel: search with the raw query immediately, search again when the rewrite returns, and fuse both rankings. You pay a max instead of a sum, at the cost of one extra retrieval.

  分析过程 · 先想清楚再作答

  1. 这题考的是「能不能把工程判断落到数字上」。凡是回答「改写当然要做，能提升效果」的，一律判为没做过——他连收益是多少都没量。
  2. 先问自己一句：**收益出现在哪一类流量上**。这一条几乎决定了答案。查询改写在单轮问答上的收益接近零（我们在 20 道单轮题上实测开关它指标一模一样），收益全在多轮追问。所以第一步是去线上日志里查多轮会话占比，占比很低的话这笔钱不该花在全量流量上。
  3. 第二步是把三笔账摆齐，缺一笔就不能下判断：指标涨了多少（用固定的标准答案集跑，不要用感觉）、延迟涨了多少（改写是短输出任务，可以换便宜快的那一档模型，往往只多两三百毫秒而不是翻倍）、多了几次调用（改写是一次，多路查询是一次调用加几次检索，成本结构完全不同，别混着算）。
  4. 第三步是找**便宜的替代路径**再比一次：只在命中触发条件时才改写（短问题、含指代词、含省略），纯新话题直接跳过；改写结果按会话缓存；用小模型跑改写而不是主模型。这三招通常能把这笔开销压掉一大半，而收益几乎不掉。
  5. 结论要落成一条可执行的判据：**收益乘以受影响流量占比，除以增加的延迟与成本**，跟你手上其他候选优化排个序。改写通常能排到很前面，因为它的失败方式是「用户明明追问同一件事却被告知查不到」，那是会直接导致弃用的体验故障，不只是指标掉几个点。
  6. 可预期的追问是「延迟真的不能接受怎么办」。答：把改写和第一次检索**并行发**，用原查询先检索一路，改写回来后再补一路，两路用倒数排名融合合起来——延迟只多一个 max 而不是一个加法，代价是多一次检索。

  Key points

  • Locate the gain first: rewriting is near-zero on single-turn traffic and pays off on follow-ups, so start from the share of multi-turn sessions.
  • All three ledgers are mandatory: metric delta on a golden set, added latency, added calls and token cost.
  • Try the cheap variants before deciding: conditional triggering, per-session caching, and a small model for the rewrite.
  • Decide on gain times affected traffic share over added latency and cost, then rank it against your other optimisations.
  • If latency is a hard constraint, fire the rewrite in parallel with the first retrieval and fuse both rankings, turning a sum into a max.

  答题要点

  • 先定位收益落在哪一类流量：改写在单轮上接近零收益，价值全在多轮追问，先查多轮会话占比。
  • 三笔账缺一不可：标准答案集上的指标变化、增加的延迟、增加的调用次数与 token 成本。
  • 先试便宜的替代路径：条件触发、按会话缓存、用小模型跑改写，通常能压掉大半开销。
  • 判据是「收益 × 受影响流量占比 ÷ 增加的延迟与成本」，再和其他候选优化排序。
  • 延迟真的卡死时，把改写与首次检索并行发，两路名次用倒数排名融合，延迟从加法变成取最大值。