Interview Bank

RAG in 14 Days: From Retrieval to Trustworthy Answers

D12 Agentic RAG: Turning Retrieval Into a Tool So the Model Decides Whether to Search, How Many Times, and Whether to Start Over

• When would you refuse to make a RAG system agentic, and what data would you use to convince your team?什么情况下你会拒绝把一个 RAG 系统做成 Agentic 的？拿什么数据说服你的团队？

  Common in ChinaCommon overseasIntermediate#agentic-rag#cost#engineering-judgement

  How to reason about it · think before answering

  1. This tests engineering judgement and whether you can do arithmetic. Anyone who says 'agentic is more advanced so we should ship it' is out. The interviewer wants you to name the cost and draw the boundary with numbers.
  2. Decompose it: identify which question types actually benefit, then check how much of your traffic they represent. Agentic gains concentrate in multi-hop questions and retrieval retries; single-document questions are answered by one lookup and every extra round is waste.
  3. So the criterion is the evaluation set, not intuition. On a 20-item set we measured multi-hop recall going from 75% to 100% while overall answerable recall moved only from 93.8% to 100%, at the cost of average retrieval calls going from 1 to 1.75 plus the same number of assessment calls - you pay for 100% of traffic so that 5% of it improves.
  4. Three clear refusals: latency-sensitive surfaces, where each round adds a retrieval plus a model round trip and roughly doubles time to first token; fixed question patterns, where nine in ten questions are single-document and the gain is near zero; and tight cost budgets, where a real model is less disciplined than an offline stand-in and the variance, not the mean, is what breaks your capacity plan.
  5. Finish with the alternative: route. Use one cheap check to decide whether a question looks multi-hop, and only then enter the loop. Nine tenths take a single retrieval, one tenth loops, and the economics change completely. Looping is a capability, not a default.
  6. Expected follow-up: how do you know which questions look multi-hop? Mine the eval set and production logs for patterns - two facts requested in one sentence, or a question about the person behind a role - start with rules, and reach for a small classifier only when rules stop working.

  分析过程 · 先想清楚再作答

  1. 这题在考工程判断力，也在考你会不会算账。凡是答「Agentic 更先进所以要上」的，直接出局；面试官想听的是你能主动说出它的代价，并且用数字划出适用边界。
  2. 怎么拆：先承认收益来自哪一类问题，再看这类问题在你的流量里占多大比例。Agentic 的收益几乎全部集中在多跳和检索失败重试上，单文档可答的问题一次检索就够了，多查一轮纯属浪费。
  3. 所以判据不是感觉，是评估集：跑一遍，看 multi 那一档占多少题、涨了多少个点，再对照总调用次数涨了多少倍。在一份 20 题的集合上，我们量到的是多跳召回从 75% 涨到 100%，可答题整体只从 93.8% 涨到 100%，代价是平均检索调用从 1 次涨到 1.75 次、外加同样次数的自评调用——为 100% 的问题付钱，只有 5% 的问题拿到好处。
  4. 三类明确不上：延迟敏感（每多一轮就是一次检索加一次模型往返，首字延迟拉长一到两倍）；问题模式固定（九成是单文档可答，收益接近零）；成本吃紧（真实模型不像离线替身那样老实，成本方差比均值更难受，按均值做的容量规划会在长尾上被打穿）。
  5. 给出替代方案才算完整：分流。先用一次便宜的判断看这一问像不像多跳，像才进循环，不像走固定流程。九成走一次检索、一成走循环，账完全不一样。这也说明循环是一种能力，不是默认值。
  6. 可预期的追问是「那你怎么知道哪些问题像多跳」。答案是从评估集和线上日志里找模式（问句里同时问了两个事实、问的是某个角色背后的人），先用规则跑，跑不动再上小模型分类——顺序不要反。

  Key points

  • Gains concentrate in multi-hop and retry cases; single-document questions gain almost nothing.
  • Settle it with the evaluation set: multi-hop delta against the multiplier on total calls.
  • One measured set: multi-hop recall 75% to 100%, overall 93.8% to 100%, retrieval calls 1 to 1.75 plus the same number of assessment calls.
  • Refuse when latency-sensitive, when question patterns are fixed, or when cost is tight - variance hurts more than the mean.
  • Route instead: a cheap check up front, and only multi-hop-looking questions enter the loop.
  • Looping is a capability, not a default.

  答题要点

  • 收益集中在多跳与检索失败重试，单文档可答的问题上收益接近零。
  • 用评估集算账：multi 档涨了多少点，对照总调用次数涨了多少倍。
  • 实测过的一组数字：多跳召回 75% 到 100%，整体 93.8% 到 100%，检索调用 1 次到 1.75 次外加等量自评调用。
  • 三类不上：延迟敏感、问题模式固定、成本吃紧（方差比均值更难受）。
  • 替代方案是分流：便宜的判断先过滤，像多跳才进循环。
  • 循环是一种能力，不是默认值。