Interview Bank

RAG in 14 Days: From Retrieval to Trustworthy Answers

D9 Hybrid Search and Reranking: Two-Path Retrieval, Reciprocal Rank Fusion, Then Re-Ranking the Top Results With a Cross-Encoder

• Adding a reranker costs you 200 ms of extra latency per question plus a per-search fee. How do you decide whether that spend is worth it?加上重排之后每次提问多了两百毫秒延迟，还多了一笔按次计费的开销。你怎么判断这笔钱该不该付？

  Common in ChinaCommon overseasDeep dive#rerank#cost-tradeoff

  How to reason about it · think before answering

  1. This question tests whether you can translate a technical choice into a business judgment. Answering `check whether the metrics went up` covers only a third of it.
  2. Split it into three ledgers: how much the metrics moved, how much latency grew, and how much money it costs. All three must be reported together; a proposal with only the first will not survive review.
  3. For the first ledger, be specific about **which** metric reranking improves. Reranking changes the order, not the candidate set — it cannot fix `the answer was never retrieved`. If your recall is the bottleneck, add a retrieval route or adjust recall depth first; the 200 ms buys nothing.
  4. For the second, ask where those 200 ms land. They sit synchronously between retrieval and generation, with the user waiting; but if a streaming generation follows and time-to-first-token is already a second or two, the relative cost is small. In an as-you-type search box, 200 ms is fatal.
  5. For the third, note the billing unit: rerankers usually charge per search rather than per token, so sending a few more candidates barely changes the bill — what is expensive is the number of questions. That points optimization at reducing query volume (caching, intent routing) rather than at trimming the candidate list.
  6. Expected follow-up: what if you simply cannot afford it? Three paths — rerank only queries classified as hard (intent routing), cache results, or self-host an open-weights cross-encoder to convert per-call fees into fixed compute cost.

  分析过程 · 先想清楚再作答

  1. 这题考的是你会不会把技术选择翻译成业务判断。只答「看指标涨没涨」只答了三分之一。
  2. 先把账拆成三笔：指标涨了多少、延迟涨了多少、钱涨了多少。三笔必须一起报，只报第一笔的方案在评审会上过不去。
  3. 第一笔要问清楚重排改善的是**哪个**指标。重排改的是顺序，不是候选集合——它救不了「答案压根没被召回」这种故障。如果你的召回率本来就不够，先去加召回路数或者调召回深度，重排这两百毫秒是白花的。
  4. 第二笔要看这两百毫秒落在哪。它是同步卡在检索之后、生成之前的，用户全程在等；但如果后面接的是一个流式生成、首字节本来就要一两秒，这两百毫秒的相对占比就小得多。反过来，如果这是一个自动补全式的即时搜索框，两百毫秒就是致命的。
  5. 第三笔要注意计价单位：重排普遍按检索次数计价而不是按 token，所以「多送几条给它排」几乎不涨钱，真正贵的是提问次数本身。这直接决定了优化方向是压提问量（缓存、意图路由）而不是压候选数。
  6. 可预期的追问：如果就是付不起怎么办？答三条路——只对判定为复杂的查询走重排（意图路由）、把结果缓存起来、或者换成自部署的开源交叉编码器把按次付费变成固定的算力成本。

  Key points

  • Report all three ledgers together: metric gain, latency growth, cost growth; a proposal missing one is incomplete.
  • Confirm whether the bottleneck is ordering or recall first; reranking only reorders and cannot rescue an answer that was never retrieved.
  • Judge the latency by where it lands: it is small relative to a streaming generation, but fatal in an as-you-type search box.
  • Rerankers bill per search rather than per token, so cost scales with question volume, not candidate count.
  • If it is unaffordable: route only hard queries to the reranker, cache results, or self-host an open-weights cross-encoder.

  答题要点

  • 三笔账一起报：指标增量、延迟增量、成本增量，缺一笔方案就不完整。
  • 先确认瓶颈是排序还是召回：重排只改顺序，救不了没被召回的答案。
  • 延迟要看落在哪：流式生成场景下相对占比小，即时搜索框里两百毫秒就是致命的。
  • 重排按检索次数计价而不是按 token，涨钱的是提问量而不是候选条数。
  • 付不起时的三条路：意图路由只对难查询重排、结果缓存、换自部署的开源交叉编码器。