Agentic RAG: Turning Retrieval Into a Tool So the Model Decides Whether to Search, How Many Times, and Whether to Start Over

Interview questions

• You are exposing retrieval to a model as a tool. How do you write the tool description, and what concrete failure modes appear when you write it badly?把检索包成一个工具交给模型，这个工具的描述该怎么写？写不好会导致哪些具体的错误行为？

  Common in ChinaCommon overseasBasic#tool-design#agentic-rag#prompting

  How to reason about it · think before answering

  1. The discriminator is whether you can name concrete failure modes. Reciting 'the description should be clear' signals you have never shipped one.
  2. Give the structure first: a usable description answers four things - what is and is not in the corpus, when the tool must be called, when it must not be called, and what shape the query string should take.
  3. Attach a failure to each: no scope and the model treats it as a web search; no 'must call' and it answers policy questions from memory, convincingly; no 'must not call' and greetings or translations each burn a retrieval; no query shape and the model pastes the raw user sentence in, dragging interrogative words into the index.
  4. The query-shape line is the cheapest win: one sentence saying 'keyword phrase, no question words' beats ten heuristics for query cleaning on the retrieval side.
  5. Production angle: optional filter parameters such as department need an explicit 'only set this when you are certain'. Models like to fill optional fields, and a wrong filter hides the correct answer while the logs only show 'no results'.
  6. Expected follow-up: how do you verify the description works? Run a negative suite - small talk, translation, arithmetic, follow-ups already answered in the conversation - and assert the tool was not called. That regression is automatable.

  分析过程 · 先想清楚再作答

  1. 这题的题眼是「具体的错误行为」。只会背「描述要写清楚工具的用途」的，一句话就暴露了没上过线——面试官想听的是描述里少一句话，线上就多一类工单。
  2. 先给结构：一段合格的工具描述要回答四件事——库里有什么和没有什么、什么时候必须用、什么时候不要用、查询串写成什么形状。四条各对应一类事故，逐条挂钩着说最有说服力。
  3. 逐条挂钩：不写范围，模型拿它当搜索引擎，问天气也去查；不写「必须用」，涉及公司制度的问题被模型凭记忆编答案，而且编得非常像真的；不写「不要用」，闲聊和翻译都触发一次无谓检索，成本和延迟白涨；不写查询形状，模型把用户整句问话塞进 query，「叫什么名字」这种疑问词进了检索，纯噪声。
  4. 最后一条最值钱也最容易漏：在描述里加一句「写成关键词短语，不要带疑问词」，比在检索侧做十种查询清洗都管用——问题在源头，就在源头修。
  5. 补一个生产视角：参数里的过滤字段（比如部门）要写明「只在确定时才填」。模型倾向于把可选参数填满，填错一个部门就把正确答案挡在库外，而这种错误在日志里看不出来，表现是「检索没结果」。
  6. 可预期的追问是「怎么验证描述写对了」。答案是拿一批负样本跑：闲聊、翻译、算术、以及答案已在对话里的追问，看模型有没有多调一次工具；这类回归是能自动化的。

  Key points

  • The description is a prompt for the model, not a code comment: scope, when to call, when not to call, query shape.
  • Missing scope turns it into a web search; missing 'must call' produces confident answers from memory.
  • Missing 'do not call' makes small talk trigger retrieval, paying cost and latency for nothing.
  • Stating 'keyword phrase, no question words' fixes query pollution at the source.
  • Optional filters need 'only set when certain' - a wrong filter silently hides the right answer.
  • Regression-test with a negative suite and assert the tool was not invoked.

  答题要点

  • 描述是写给模型看的提示词，不是注释；四段式：范围、什么时候用、什么时候不用、查询写成什么形状。
  • 不写范围会被当成搜索引擎；不写「必须用」会导致凭记忆编答案。
  • 不写「不要用」会让闲聊也触发检索，成本和延迟白涨。
  • 写明查询要用关键词短语、不带疑问词，比在检索侧清洗查询更根本。
  • 可选过滤参数要写「只在确定时才填」，填错会静默地把正确答案挡在外面。
  • 用一批负样本（闲聊、翻译、算术）做回归，断言工具没有被调用。
• Self-reflective retrieval rewrites the query and retries. How do you guarantee it terminates instead of spinning on the same query forever?自反思式检索会反复改写查询重试。你怎么保证它一定会停下来，而不是在同一个查询上原地打转？

  Common in ChinaCommon overseasIntermediate#agentic-rag#self-reflection#reliability

  How to reason about it · think before answering

  1. This checks whether you have actually run such a loop. 'Set a max iteration count' is half an answer: it stops one failure mode and lets two others through.
  2. Split runaway behaviour into three shapes and give each its own brake. Progress that never completes is capped by max rounds. Per-round budgets that pass individually but blow up in aggregate need a cumulative token budget - four rounds of 600 tokens each never trips a per-round check yet quadruples what reaches the model. Spinning in place needs duplicate-query detection.
  3. Two implementation details prove you have written it: the duplicate check belongs before the retrieval call, otherwise you pay for a call to learn you are looping; and queries must be normalized to a set of terms, or 'failover approval' and 'approval failover' count as two distinct queries and the loop keeps turning.
  4. Say what happens after it stops: stop reasons must be recorded as distinct categories - satisfied, gave up, hit round cap, hit token budget, duplicate query. Collapsing them into 'loop finished' hides how often the system simply surrendered.
  5. An easy miss: installing a brake is not testing it. If the default token budget sits far above real usage it never fires, which is the same as not having one. Every brake needs a case that trips it.
  6. Expected follow-up: what if the model says 'not enough' when it actually is? Make the assessment structured - which elements are covered, which are missing - and treat an empty missing list as sufficient, so the decision is auditable rather than a bare boolean.

  分析过程 · 先想清楚再作答

  1. 这题在考「有没有真让循环跑过」。只答「设一个最大轮数」的能拿一半分，因为最大轮数只拦住了一类失控，剩下两类照样漏出去。
  2. 怎么拆：把失控分成三种形态，每种配一道闸。一是「每轮都在推进但永远推进不完」，用最大轮数拦；二是「每轮都不超标但累计爆掉」，用累计 token 预算拦——四轮各读 600 token 没有一轮超标，可送进模型的材料已经是单轮的四倍；三是「原地打转」，用重复查询检测拦。
  3. 重复查询检测有两个实现细节，答出来就说明真写过：一是要放在检索之前，否则要白花一次调用才发现自己在转圈；二是判重要对查询做归一化，只看词的集合，否则「主备切换 审批」和「审批 主备切换」会被当成两个不同的查询，圈照转不误。
  4. 还要说清停下来之后怎么办：停止原因必须分类记录，「查够了」「主动认输」「撞到轮数」「撞到预算」「原地打转」是五种不同的结局。把它们混成一个「循环结束」，你就永远看不见系统在多大比例的问题上其实是放弃了。
  5. 一个容易被忽略的点：闸门装了不等于验过。默认预算如果比实际用量高一大截，跑多少遍都踩不响它，等于没装。每一道闸都要构造一个用例把它踩响，这是验收的一部分。
  6. 可预期的追问是「模型自己说不够，但其实已经够了怎么办」。答案是自评要给结构化输出（覆盖了哪些要素、缺哪些），缺失项为空却仍判不够时按「够了」处理——让判断可审计，而不是信一个布尔值。

  Key points

  • Three brakes, none optional: max rounds, cumulative token budget, duplicate-query detection.
  • The cumulative budget catches rounds that each pass but blow up together - the round cap cannot see that.
  • Check for duplicates before retrieving, and normalize the query to a term set before comparing.
  • Record stop reasons as distinct categories rather than one 'finished' bucket.
  • Every brake needs a case that actually trips it; an untested brake is no brake.
  • Have the assessor emit covered and missing elements so 'not enough' is auditable.

  答题要点

  • 三道闸缺一不可：最大轮数、累计 token 预算、重复查询检测。
  • 累计预算拦的是「每轮都不超但加起来爆掉」，轮数闸看不见这件事。
  • 重复查询检测要放在检索之前，且查询要归一化成词的集合再判重。
  • 停止原因分类记录：查够了、主动认输、撞轮数、撞预算、原地打转是五种结局。
  • 每一道闸都要构造用例踩响，装了没验过等于没装。
  • 自评输出结构化的覆盖与缺失项，让「不够」这个判断可审计。
• In multi-hop retrieval a wrong first hop poisons every hop after it. How would you design for that?多跳检索里第一跳查错了，后面全跟着错。你会怎么设计容错？

  Common in ChinaCommon overseasDeep dive#multi-hop#error-propagation#agentic-rag

  How to reason about it · think before answering

  1. This is about error propagation. 'Add a retry' is not enough - retries help when one path fails, but the multi-hop problem is walking confidently down the wrong path.
  2. Separate two failure kinds first, because the fixes are opposite. Either the answer document never entered the candidate pool - no amount of loosening helps, only a new query term does, which is the multi-hop path - or it was retrieved and then dropped by your own admission threshold, where extra hops are useless and only relaxing the gate recovers it. Coverage plus the answer slot tells them apart: low coverage means wrong direction (broaden), high coverage with an empty slot means halfway there (hop).
  3. Then give the mechanism. Do not let the model freestyle the next query: pick a bridge phrase from the sentence that best matches the question - a concrete noun the question never mentioned that also appears in another document. 'Not in the question' makes it new information; 'appears elsewhere' guarantees there is somewhere to hop to.
  4. Fault tolerance has three layers: keep the earlier hop's material, so a bad second hop does not destroy the evidence you already had; trace each hop separately so you can locate where it went wrong; and surrender explicitly when there is no lead left, handing 'insufficient evidence' to the generation-side refusal.
  5. The overlooked trap is worth points: the hop succeeds but the metric does not move. The second hop really did retrieve the target document, yet if you merge both hops' candidates and pack by score, the first hop's higher lexical overlap fills the budget and the target never enters the context. Allocate the context budget round-robin across hops - that is where multi-hop gains are actually realized.
  6. Expected follow-up: how do you know the first hop was wrong? From the structured self-assessment, not from the final answer. By the time the answer is wrong the chain is three hops deep and much more expensive to debug.

  分析过程 · 先想清楚再作答

  1. 这题考的是错误传播意识。只答「加个重试」是不够的——重试只在「同一条路走不通」时有用，而多跳的问题是走上了错误的路还越走越远。
  2. 先把两类失败分开，这是整题的骨架：一类是根本没捞到（答案文档在候选池里一次都没出现，放宽门槛毫无用处，只能靠新的查询词重查，也就是多跳），一类是捞到了却被自己的过滤器扔了（排在第二名但没过准入门槛，这一类跳多少跳都没用，只能降级放宽门槛重判）。判据是要素覆盖率加答案槽位：覆盖率低是方向错了走放宽，覆盖率高但槽位空是只查到半路走多跳。判错类型，容错就完全用反了。
  3. 然后给具体机制。判断下一跳查什么，不能凭模型自由发挥，要有可解释的判据：从最贴题的那一句里挑出问题没提过、且在别的文档里也出现过的具体名词当作桥接短语。「问题没提过」保证它是新信息，「别的文档里也有」保证真的有下一跳可跳——只在这一篇里出现的短语，查了只会把同一篇再捞回来。
  4. 结论层面，容错有三层：不要丢掉上一跳的材料（第二跳查错了，第一跳的证据还在）；每一跳独立记录轨迹，事后能定位是哪一跳歪的；追不动时主动认输，把「材料不足」交给生成侧的拒答，而不是硬凑一个答案。
  5. 有一个非常容易被忽略的坑，说出来会加分：跳成功了，命中却没变。第二跳确实把目标文档检索回来了，但如果把两跳的候选混在一起按名次装上下文，第一跳的材料字面重合度更高，会把预算占满，目标文档根本挤不进去。上下文预算必须按跳轮转分配——多跳的收益是在这一步兑现的，不是在检索那一步。
  6. 可预期的追问是「怎么知道第一跳错了」。答案是靠自评的结构化输出，而不是靠最终答案对不对；等到答案错了再回头找，链路已经断了三跳，定位成本高得多。

  Key points

  • Classify first: never retrieved needs a new query term (a hop); retrieved-then-filtered needs a relaxed gate. The fixes are opposite.
  • Derive the next query from a bridge phrase - a concrete noun absent from the question that also appears in another document.
  • Keep the previous hop's material so a failed hop does not discard existing evidence.
  • Trace every hop separately so you can pinpoint which one drifted.
  • Allocate context budget round-robin across hops, or a successful hop still fails to change the metric.
  • Surrender explicitly when no lead remains and hand it to the generation-side refusal.

  答题要点

  • 先分类：根本没捞到只能靠多跳换查询词，捞到了被门槛扔了只能靠降级放宽，两者修法相反。
  • 下一跳的查询用桥接短语：问题没提过、且别的文档里也出现过的具体名词。
  • 保留上一跳的材料，第二跳失败时第一跳的证据仍在。
  • 每一跳独立记轨迹，能定位是哪一跳歪的。
  • 上下文预算按跳轮转分配，否则跳成功了命中也不会变。
  • 追不动时主动认输，把材料不足交给生成侧拒答，不硬凑答案。
• When would you refuse to make a RAG system agentic, and what data would you use to convince your team?什么情况下你会拒绝把一个 RAG 系统做成 Agentic 的？拿什么数据说服你的团队？

  Common in ChinaCommon overseasIntermediate#agentic-rag#cost#engineering-judgement

  How to reason about it · think before answering

  1. This tests engineering judgement and whether you can do arithmetic. Anyone who says 'agentic is more advanced so we should ship it' is out. The interviewer wants you to name the cost and draw the boundary with numbers.
  2. Decompose it: identify which question types actually benefit, then check how much of your traffic they represent. Agentic gains concentrate in multi-hop questions and retrieval retries; single-document questions are answered by one lookup and every extra round is waste.
  3. So the criterion is the evaluation set, not intuition. On a 20-item set we measured multi-hop recall going from 75% to 100% while overall answerable recall moved only from 93.8% to 100%, at the cost of average retrieval calls going from 1 to 1.75 plus the same number of assessment calls - you pay for 100% of traffic so that 5% of it improves.
  4. Three clear refusals: latency-sensitive surfaces, where each round adds a retrieval plus a model round trip and roughly doubles time to first token; fixed question patterns, where nine in ten questions are single-document and the gain is near zero; and tight cost budgets, where a real model is less disciplined than an offline stand-in and the variance, not the mean, is what breaks your capacity plan.
  5. Finish with the alternative: route. Use one cheap check to decide whether a question looks multi-hop, and only then enter the loop. Nine tenths take a single retrieval, one tenth loops, and the economics change completely. Looping is a capability, not a default.
  6. Expected follow-up: how do you know which questions look multi-hop? Mine the eval set and production logs for patterns - two facts requested in one sentence, or a question about the person behind a role - start with rules, and reach for a small classifier only when rules stop working.

  分析过程 · 先想清楚再作答

  1. 这题在考工程判断力，也在考你会不会算账。凡是答「Agentic 更先进所以要上」的，直接出局；面试官想听的是你能主动说出它的代价，并且用数字划出适用边界。
  2. 怎么拆：先承认收益来自哪一类问题，再看这类问题在你的流量里占多大比例。Agentic 的收益几乎全部集中在多跳和检索失败重试上，单文档可答的问题一次检索就够了，多查一轮纯属浪费。
  3. 所以判据不是感觉，是评估集：跑一遍，看 multi 那一档占多少题、涨了多少个点，再对照总调用次数涨了多少倍。在一份 20 题的集合上，我们量到的是多跳召回从 75% 涨到 100%，可答题整体只从 93.8% 涨到 100%，代价是平均检索调用从 1 次涨到 1.75 次、外加同样次数的自评调用——为 100% 的问题付钱，只有 5% 的问题拿到好处。
  4. 三类明确不上：延迟敏感（每多一轮就是一次检索加一次模型往返，首字延迟拉长一到两倍）；问题模式固定（九成是单文档可答，收益接近零）；成本吃紧（真实模型不像离线替身那样老实，成本方差比均值更难受，按均值做的容量规划会在长尾上被打穿）。
  5. 给出替代方案才算完整：分流。先用一次便宜的判断看这一问像不像多跳，像才进循环，不像走固定流程。九成走一次检索、一成走循环，账完全不一样。这也说明循环是一种能力，不是默认值。
  6. 可预期的追问是「那你怎么知道哪些问题像多跳」。答案是从评估集和线上日志里找模式（问句里同时问了两个事实、问的是某个角色背后的人），先用规则跑，跑不动再上小模型分类——顺序不要反。

  Key points

  • Gains concentrate in multi-hop and retry cases; single-document questions gain almost nothing.
  • Settle it with the evaluation set: multi-hop delta against the multiplier on total calls.
  • One measured set: multi-hop recall 75% to 100%, overall 93.8% to 100%, retrieval calls 1 to 1.75 plus the same number of assessment calls.
  • Refuse when latency-sensitive, when question patterns are fixed, or when cost is tight - variance hurts more than the mean.
  • Route instead: a cheap check up front, and only multi-hop-looking questions enter the loop.
  • Looping is a capability, not a default.

  答题要点

  • 收益集中在多跳与检索失败重试，单文档可答的问题上收益接近零。
  • 用评估集算账：multi 档涨了多少点，对照总调用次数涨了多少倍。
  • 实测过的一组数字：多跳召回 75% 到 100%，整体 93.8% 到 100%，检索调用 1 次到 1.75 次外加等量自评调用。
  • 三类不上：延迟敏感、问题模式固定、成本吃紧（方差比均值更难受）。
  • 替代方案是分流：便宜的判断先过滤，像多跳才进循环。
  • 循环是一种能力，不是默认值。

Comments

Sign in to join the discussion

No comments yet — be the first.