Interview Bank

From Frontend Engineer to Agent Engineer in 30 Days

D5 The Tool System and Event-Driven Design: Parameter Validation, Feeding Errors Back for Self-Correction, Event Subscription (dg P05/P06/M05/M07)

• Which lifecycle events does an agent runtime typically expose, and why is waiting for the final return value not enough?Agent 的事件系统一般会暴露哪些生命周期事件？为什么不能只等最终返回值？

  Common in ChinaCommon overseasIntermediate#event-driven#observability

  How to reason about it · think before answering

  1. It looks like a listing question but it really tests whether you have shipped an agent with a UI. Reciting event names without saying what each one is for reads as documentation-deep only.
  2. Start with the motivation: a tool-using loop runs from seconds to minutes, calling models and tools and sometimes retrying, while the return value is just the final sentence. Everything in between is a black box to the caller, who cannot tell whether to keep waiting.
  3. List them with a purpose each: run:start, run:end and run:error mark the turn and its two endings; model:delta carries text fragments for the typewriter effect; tool:proposed fires when the model has chosen a tool but has not executed it, which is where the approval gate hangs; tool:start, tool:end and tool:error are the three exits of execution, with duration on tool:end; approval:required tells the UI to show a confirmation card.
  4. Then name the real payoff: one event stream feeds three consumers — the UI renders progress, logging gets distributed tracing, and metering reads token counts off run:end. One stream instead of three instrumentation layers is an architecture answer, not an API listing.
  5. Add two implementation rules that separate candidates: every event carries a runId and a monotonic sequence number because ordering is not guaranteed once events cross processes, and listeners must contain no business logic and never let an exception escape into the main loop. Events are a side channel, not the trunk.
  6. Expect the follow-up: isn't one event per token too many? Yes, so batch on a time window — flush every 50ms, which is imperceptible to users and cuts message volume by an order of magnitude.

  分析过程 · 先想清楚再作答

  1. 这题看起来是背清单，实际考的是你有没有做过带界面的 Agent。只报事件名不解释用途，会被判成看过文档但没接过前端。
  2. 先说动机：一次带工具的循环短则几秒长则几分钟，中间要调模型、调工具、可能还失败重试，而返回值只有最后一句话。对调用方来说中间全是黑盒——不知道它在干什么，也不知道该不该再等。
  3. 再报清单并各配一句用途：run:start / run:end / run:error 是一轮的开始与两种结束；model:delta 是模型吐出的文本片段，前端拿它做打字机效果；tool:proposed 是模型决定要调工具但还没执行，权限确认就挂在这个事件上；tool:start / tool:end / tool:error 是工具执行的三个出口，tool:end 带耗时；approval:required 让界面弹确认框。
  4. 然后说出这套设计真正的价值：同一条事件流同时喂三个消费者——界面渲染进度、日志系统做链路追踪、计量系统拿 run:end 的 token 数算成本。不为三件事写三套埋点，这是架构判断而不是 API 罗列。
  5. 补两条实现纪律，能显著拉开差距：事件必须带 runId 和自增序号，因为跨进程传输后顺序不保证；监听器里不写业务逻辑，且监听器抛错不能炸掉主循环——事件是旁路不是主干。
  6. 可以预期的追问：model:delta 一个 token 一条事件会不会太多？会，所以要按时间窗合批，攒 50 毫秒推一次，用户感知不到差别而消息量掉一个数量级。

  Key points

  • Motivation: a turn takes seconds to minutes and only returns the final sentence, so the caller cannot tell whether to keep waiting
  • Typical events: run:start/end/error, model:delta, tool:proposed, tool:start/end/error, approval:required
  • One stream serves the UI, distributed tracing and cost metering — no need for three instrumentation layers
  • Every event carries a runId and a sequence number since ordering is not guaranteed across processes
  • Listeners hold no business logic and must not throw into the main loop; batch model:delta on a 50ms window

  答题要点

  • 动机：一轮循环几秒到几分钟，返回值只有最后一句话，中间全是黑盒，调用方无法判断该不该继续等
  • 常见事件：run:start / run:end / run:error、model:delta、tool:proposed、tool:start / tool:end / tool:error、approval:required
  • 同一条事件流同时喂界面、日志链路追踪和成本计量三个消费者，不用写三套埋点
  • 事件要带 runId 和自增序号，跨进程后顺序不保证，消费端要能自己排序
  • 监听器不写业务逻辑，且抛错不能影响主循环；model:delta 要按 50 毫秒时间窗合批

D13 Cron Scheduling (Central Scheduler → Stream Delivery) + Cost Metering (Token → USD Ledger, Usage Report)

• How would you design a token cost metering and ledger system from scratch?让你从零设计一套 token 成本计量和台账系统，你会怎么做？

  Common in ChinaCommon overseasIntermediate#cost#observability#data-modeling

  How to reason about it · think before answering

  1. This question tests whether you have ever reconciled a bill. The discriminators are the numeric type you store money in, and whether cost is stored or computed at query time. A design missing either gets rejected by finance within a quarter.
  2. Set the criterion first: a ledger is not a log. Logs exist for debugging and can be dropped; a ledger has to reconcile against the vendor invoice and answer why the bill grew 40% this month, so every charge must trace back to who, which run, which model, and how many tokens. Every field falls out of that.
  3. Then walk the fields with reasons: user_id says whose budget it hits; run_id says which execution it belongs to and is nullable because some spend is system-level batch work; model records the one actually used, since fallback routes the same workload to different providers; kind separates chat from embedding because their volumes and growth curves differ completely; prompt_tokens and completion_tokens are stored separately because input and output differ three- to four-fold in price, and a single total can neither reproduce the amount nor tell you whether the prompt is bloated or the model is verbose.
  4. Now the two judgments that show experience. First, money uses fixed-point: numeric in the database, Decimal or BigDecimal in code, never accumulated in binary floats, or the total will diverge from the sum of rows after a hundred thousand entries. Second, cost is computed at write time and stored redundantly, not recomputed from the current price table — prices change, and history must not change with them. That is the essential difference between a ledger and a report.
  5. Volunteer the timing and transaction boundary: record at the moment you receive the usage field, not at business success, because failed calls still cost money and a fallback spans two or three billable calls per business operation. Ledger writes need not share the business transaction — losing a row costs fractions of a cent, while locking the ledger table stalls user conversations — so write asynchronously with retries and a uniqueness constraint on run id plus call index. The exception is quota enforcement: if the product caps spend, the decrement must be transactional or concurrent requests will blow through the cap.
  6. Expect: what happens to history when the vendor changes prices? The price table itself needs effective dates and a version, and the ledger stores both the computed amount and the price version, so recomputation and audit both have a basis.

  分析过程 · 先想清楚再作答

  1. 这题在考「你有没有真的对过账」。区分度在两个地方：金额用什么类型存，以及金额是冗余存还是查询时现算。答不到这两点的方案，上线三个月就会被财务打回来。
  2. 先立判据：台账不是日志。日志是给排查问题用的，删了就删了；台账要拿去对账、要回答「这个月为什么涨了 40%」，所以每一笔钱都必须能追回到「谁、因为哪一次执行、用哪个模型、花了多少 token」。字段设计全部由这条判据推出来。
  3. 然后给字段和理由，一一对应：user_id 回答该算谁头上、run_id 回答属于哪次执行（允许为空，因为有系统级批量开销）、model 存调用当时那一个（fallback 会让同一段业务落到不同模型上）、kind 区分 chat 和 embedding（两者量级和增长曲线完全不同）、prompt_tokens 与 completion_tokens 分开存（输入输出单价差三到四倍，只存 total 就算不回金额，也看不出是提示词太长还是模型太啰嗦）。
  4. 接着是两个最能体现经验的判断。第一，金额用定点类型：数据库用 numeric，代码里用 Decimal 或 BigDecimal，绝不用双精度浮点累加，否则十万条之后总额和逐条相加对不上。第二，cost_usd 要在写入那一刻算好并冗余存，不要查询时用当前价格表现算——价格会变，历史账单不能跟着一起变，这是台账和报表最本质的区别。
  5. 还要主动说记账的时机和事务边界：记账放在「拿到 usage 字段」那一刻，而不是「业务成功」那一刻，因为失败的调用同样产生费用，尤其 fallback 会一次业务跨两三次收费调用。台账写入不必和业务同事务（丢一条只是几厘钱，锁住台账表却会卡住用户对话），可以异步加重试，用 run_id 加调用序号做唯一约束防重；但如果产品有额度限制，配额扣减必须同事务，否则用户能靠并发把额度刷穿。
  6. 可以预期的追问：厂商调价了历史数据怎么办？答案是价格表本身要有生效时间和版本号，台账里既存算好的金额也可以存价格版本，这样重算和审计都有依据。

  Key points

  • A ledger is not a log: every charge must trace to a user, a run, a model and a token count, and the schema follows from that
  • Store prompt and completion tokens separately, since input and output prices differ three- to four-fold and a single total can neither reproduce the amount nor localize the problem
  • Use fixed-point money (numeric in the database, Decimal or BigDecimal in code); float accumulation makes totals disagree with the sum of rows
  • Compute cost at write time and store it, rather than recomputing from today's price table, so history stays stable when prices change
  • Record at the moment usage is returned, not at business success — failed calls and fallbacks still cost money; ledger writes can be async with retries, but quota decrements must be transactional

  答题要点

  • 台账不是日志：每一笔钱要能追回到谁、哪一次 run、哪个模型、多少 token，字段设计全由这条判据推出
  • prompt_tokens 与 completion_tokens 必须分开存，因为输入输出单价差三到四倍，只存 total 既算不回金额也看不出问题出在哪一侧
  • 金额用定点类型（数据库 numeric、代码 Decimal/BigDecimal），不要用浮点累加，否则总额和逐条相加对不上
  • cost_usd 在写入那一刻算好并冗余存，不要查询时按当前价格现算——价格会变，历史账单不能跟着变
  • 记账时机是拿到 usage 字段那一刻而不是业务成功那一刻，失败调用和 fallback 同样产生费用；台账可异步写入加重试，但配额扣减必须和业务同事务
• Which dimensions should a usage report for an LLM product cover, and what decision does each one drive?一份 LLM 应用的 usage report 通常要覆盖哪些维度？这些维度分别用来做什么决策？

  Common in ChinaCommon overseasIntermediate#observability#cost#reporting

  How to reason about it · think before answering

  1. The trap is listing dimensions: by user, by day, by model, by feature. Length signals you have not thought about it. The discriminator is the second half — which action each dimension drives. No action means you built reports but never used one.
  2. Give three primary dimensions with their action type: by user is a commercial action (who to reprice, who is abusing, whether tiering covers cost); by day is a debugging action (align with the release timeline to find which deploy stepped the cost up); by model and call kind is an optimization action (did tiered routing actually save money, is embedding volume running away). Three dimensions, three different dashboard audiences.
  3. Then go up a level: absolute dollars carry no information. What matters are unit-economics ratios with a denominator — cost per run (monthly cost over run count), cost per active user per month, and business actions completed per dollar. The first two say whether pricing covers cost; the third says whether the system deserves further investment.
  4. Prove you have used it with a concrete pattern: cost per run is a ruler. If user count is flat but cost per run climbs, it is almost always a deploy that lengthened the prompt or a tool whose response body grew. That signal usually appears days before latency alerts, which is why mature teams put the cost curve next to error rate and latency on the on-call dashboard.
  5. Add the dimension most people miss: failures and fallbacks. Failed calls are still billed, and a fallback spans two or three billable calls per business operation. Without slicing that out, your gap against the vendor invoice concentrates exactly during incidents, when you most need cost clarity.
  6. Expect: how fresh does the report need to be? Tier it — daily rollups can run offline, but quota and budget guardrails need near-real-time month-to-date totals, usually from an incrementally updated per-user monthly summary table rather than scanning the detail rows on every request.

  分析过程 · 先想清楚再作答

  1. 这题最容易答成罗列维度：按用户、按天、按模型、按功能……列得越全越显得没想过。区分度在后半句——每个维度对应的是哪一类行动。列不出行动，说明你只做过报表没用过报表。
  2. 先给三个主维度和它们各自的行动类型：按用户切是商业动作（谁该涨价、谁在滥用、定价分层能不能覆盖成本）；按天切是排障动作（对齐发布时间线，找出是哪次上线让成本跳了台阶）；按模型和调用类型切是优化动作（验证分层路由有没有真省到钱、embedding 的量是不是失控了）。三个维度对应三个不同的看板受众。
  3. 然后升一层，指出绝对金额没有信息量，真正有用的是带分母的单位经济学指标：每次执行成本（当月总成本除以 run 数）、每用户月成本（除以活跃用户数）、每美元产出（完成的业务动作数除以总成本）。前两个用来判断定价能不能覆盖成本，第三个用来判断这套系统值不值得继续投入。
  4. 举一个能落地的用法证明你真用过：每次执行成本这个比值是把尺子。如果用户数没涨而单次成本涨了，几乎一定是某次上线让提示词变长了，或者某个工具的返回体膨胀了——这个信号通常比超时告警早好几天出现，所以成熟团队会把成本曲线和错误率、延迟并排挂在值班大盘上。
  5. 最后补一个大多数人会漏的维度：失败与降级。失败的调用照样收费，fallback 会让一次业务操作跨两三次收费调用。报表里不单独切出这一块，你和厂商账单的差额就会恰好集中在故障期，也就是最需要看清成本的时候。
  6. 可以预期的追问：报表要做到什么实时度？答案是分层——按天的汇总离线跑就够，但配额和预算护栏需要近实时的当月累计，通常用一张按用户按月的汇总表增量更新，而不是每次请求都扫一遍明细。

  Key points

  • By user drives commercial decisions, by day drives debugging, and by model or call kind drives optimization — three dimensions, three audiences
  • Absolute dollars say nothing; use ratios with a denominator: cost per run, cost per active user per month, and business actions per dollar
  • Cost per run is a ruler: flat users with rising per-run cost usually means a longer prompt or a bloated tool response, and it shows days before latency alerts
  • Slice out failed and fallback calls, or your gap against the vendor invoice concentrates during incidents
  • Tier the freshness: daily rollups offline, near-real-time month-to-date totals from an incremental summary table for budget guardrails

  答题要点

  • 按用户切是商业动作（定价分层、异常账号），按天切是排障动作（对齐发布找成本跳变），按模型和调用类型切是优化动作（验证分层路由、盯 embedding 用量）
  • 绝对金额没有信息量，要看带分母的指标：每次执行成本、每用户月成本、每美元产出
  • 每次执行成本是把尺子：用户数没涨而单次成本涨了，通常是提示词变长或工具返回体膨胀，比超时告警早好几天出现
  • 必须单独切出失败与降级的开销，否则和厂商账单的差额会集中在故障期
  • 实时度要分层：按天汇总可离线跑，预算护栏需要近实时的当月累计，用增量汇总表而不是每次扫明细

D14 Deployment and Operations: Multi-Worker Compose, Heartbeats, Health Checks, Graceful Shutdown, Dev/Prod Isolation; Week Two Retrospective

• With multiple replicas, how do you design heartbeats and health checks? Are they the same thing?多实例部署下，怎么设计心跳和健康检查？两者是同一件事吗？

  Common in ChinaCommon overseasIntermediate#observability#deployment#distributed-systems

  How to reason about it · think before answering

  1. The hinge is are they the same thing. Answering both check liveness loses the point — the interviewer wants to see you split one word into three distinct questions, because conflating them causes real outages.
  2. Separate them: a liveness probe answers should this process be restarted, a readiness probe answers can you send me traffic now, and a heartbeat dashboard answers what is the cluster's state. The audiences differ: the first two are for the orchestrator, the third is for a human.
  3. Then say why heartbeats are not optional: the orchestrator only sees process liveness, but a worker can be alive while doing no work at all — a blocked event loop, an exhausted connection pool timing out every read, a noisy neighbour saturating host CPU. This kind of zombie is exactly what the orchestrator cannot see, and only an application-level heartbeat catches it.
  4. Get the direction right too: replicas push their own heartbeat rather than the gateway polling each one. Containers change IP and hostname constantly, so a poller needs a roster that is always changing — and maintaining that roster is what heartbeats are for, so the logic is circular. Report at least three things: a timestamp for liveness, in-flight count to distinguish idle from overloaded, and a version so you can watch old and new replicas during a rollout.
  5. The sharpest point is isolation: do not query downstream dependencies inside a readiness probe. One worker going quiet would turn every gateway's readiness red, and the orchestrator would pull the entire ingress layer — turning a non-critical fault into a full outage. In reality that worker's absence does not stop intake at all: messages sit in the stream, unacked ones get claimed by someone else, and its lease changes hands when the TTL expires.
  6. Expect: so how does the gateway decide whether a worker is usable? Answer that it does not, and does not need to — the gateway never assigns work to a specific worker; the consumer group and the lease decide that. Heartbeat data is for observability and alerting, not routing. Getting here shows you actually understand the layering.

  分析过程 · 先想清楚再作答

  1. 题眼是「两者是同一件事吗」。答「都是探活」直接失分——面试官想看你能不能把一个词拆成三个不同的问题，因为混起来会造成真事故。
  2. 先拆问题：存活探针回答「这进程要不要被重启」，就绪探针回答「现在能不能给我发流量」，心跳面板回答「集群此刻是什么状态」。三者的读者不同：前两个给编排系统，第三个给人。
  3. 再说心跳为什么不可省：编排系统只能看到进程存活，而 Worker 完全可以进程活着而活儿全停——事件循环被死循环占住、连接池耗尽后取消息全超时、宿主机 CPU 被邻居打满。这类假死恰好是编排系统看不见的那种，只有业务自己上报的心跳能发现。
  4. 方向也要答对：心跳是副本自己 push，不是 Gateway 逐个 pull。因为容器随时换 IP 和主机名，去问的一方需要一份永远在变的名单，而那份名单本身就得靠心跳维护，逻辑绕回来了。上报内容至少三样：时间戳判活、在跑任务数区分闲和忙、版本号在滚动发布时看新旧两批各剩几个。
  5. 最关键的一刀是隔离性：**不要把下游依赖查进就绪探针**。一个 Worker 失联导致所有 Gateway 的就绪探针同时转红，编排系统会把整个接入层摘光——一个非核心故障被自己升级成全站不可用。而实际上那个 Worker 失联根本不影响接单：消息还在流里，没确认的会被别人接手，它的租约会因 TTL 到期而易主。
  6. 可以预期的追问：那 Gateway 怎么判断某个 Worker 可不可用？答「它不判断，也不需要判断」——Gateway 从不指定某个 Worker 干活，派活由消费组和租约决定，心跳的用途是观测和告警，不是路由。答到这里就说明你真的想清楚了分层。

  Key points

  • Split one word into three questions: liveness (restart me?), readiness (send me traffic?), heartbeat dashboard (what is the cluster doing?) — first two for the orchestrator, third for humans
  • The orchestrator sees process liveness but not zombies (blocked loop, exhausted pool, stolen CPU), so an application-level heartbeat is mandatory
  • Heartbeats must be pushed by replicas, not polled by the gateway: containers change IP constantly and polling needs a roster that heartbeats themselves maintain
  • Report timestamp, in-flight count and version — for liveness, load, and rollout progress respectively
  • Never query downstream dependencies in a readiness probe, or one quiet worker pulls the whole ingress layer and escalates a minor fault into an outage
  • The gateway does not judge worker availability — the consumer group and lease assign work; heartbeats are for observability, not routing

  答题要点

  • 一个词要拆成三个问题：存活探针（要不要重启）、就绪探针（能不能发流量）、心跳面板（集群什么状态），前两个给编排系统、第三个给人
  • 编排系统只看得见进程存活，看不见假死（事件循环卡住、连接池耗尽、CPU 被抢），所以业务层心跳不可省
  • 心跳必须是副本 push 而不是 Gateway pull：容器随时换 IP，pull 需要一份靠心跳才能维护的名单，逻辑绕回来了
  • 上报时间戳、在跑任务数、版本号三样，分别用于判活、区分忙闲、观察滚动发布进度
  • 不要把下游依赖查进就绪探针，否则一个 Worker 失联会让整个接入层被摘掉，把非核心故障升级成全站不可用
  • Gateway 不判断 Worker 可用性——派活由消费组和租约决定，心跳只用于观测告警，不用于路由

D16 Dynamic Routing With a Supervisor: Structured-Output Routing, Override, routingReason

• What is a field like routingReason actually worth in production? Is it just logging?routingReason 这类调试信息在生产系统里有什么价值？只是打日志而已吗？

  Common in ChinaCommon overseasIntermediate#observability#routing#debugging

  How to reason about it · think before answering

  1. This looks like a throwaway question but it screens for whether you have ever been on call. Anyone who stops at it helps with debugging has not.
  2. Start with the fact you cannot design around: the routing decision is made by a model, and models are not reproducible. The same sentence may be judged differently next time, so you cannot re-run to see what it was thinking. The reason must be captured at decision time or it is gone forever — that is what turns this field from a log line into the only audit evidence you have.
  3. Then give three concrete uses. One, it separates a wrong model judgement from a parsing or fallback problem, provided the prefix carries a cause code. Two, it is raw material for the next prompt revision: group a week of fallbacks by cause and the missing intent descriptions jump out. Three, it feeds offline evaluation — a golden set should score routing accuracy, not just the final answer, and that is only scorable if the decision and its reason were recorded.
  4. Mention the shape: a structured prefix wrapping a human sentence. The prefix (fallback plus cause, override plus target) is what you aggregate on; the sentence is what you read for one specific case. Making the whole field prose puts you right back in the failure mode this chapter argues against.
  5. Add the detail people skip: neither a fallback nor a human override should erase the model's original judgement — carry it into the reason. Otherwise nobody can later tell whether the model got it wrong or a human redirected it. Twenty extra characters save an afternoon of archaeology.
  6. Expect: do these fields create privacy or cost problems? Yes, so record the basis for the decision rather than the user's raw text, cap the length, and reuse the same run identifier as your tracing instead of inventing a parallel one.

  分析过程 · 先想清楚再作答

  1. 这题看着像水题，其实在筛「有没有真的排查过线上问题」。答「方便调试」就结束的人，基本没值过班。
  2. 先给一条不可回避的事实：**路由决策是模型做的，而模型不可复现**。同一句话下次未必给同样的判断，你没法重跑一遍去看「当时是怎么想的」。所以理由必须在当时就写下来，否则那次判断永远丢了。这一条把 routingReason 从「日志」抬到了「唯一的审计证据」。
  3. 然后给三个具体用途，每个都要能落地：一是把「模型判错了」和「解析或兜底出错了」分开，前缀写成 fallback 加原因码，一眼就能分辨；二是攒下一版提示词的素材，把一周内落进兜底的请求按原因分组，会看到集中的几类意图缺描述；三是它是离线评估的输入——标准样本集要评的不只是最终回答，还有分诊准不准，而这件事只有当时记了判断和理由才评得了。
  4. 写法上有个细节值得主动说：**结构化的壳加自然语言的芯**。前缀（fallback 加原因、override 加目标）用来聚合统计，后面那句人话用来看具体这一单。整条都写成自然语言，就退回成本章批判的那种东西了。
  5. 再补一条容易被忽略的：兜底和人工改派都不要擦掉模型的原判，原样拼进理由里。否则一周后没人说得清这一单是模型判错了还是本来就被人改过——多写二十个字符，省掉一次翻遍代码的排查。
  6. 可以预期的追问：这些字段会不会带来隐私或成本问题？答案是会，所以理由里只写判断依据不写用户原文，长度设上限（比如 120 字），并且和链路追踪共用同一个 run 标识，别另起一套。

  Key points

  • The decision comes from a model and is not reproducible, so the reason must be captured at decision time — it is the only audit evidence you get
  • Use one: it separates a wrong model judgement from a parsing or fallback failure, via a cause code in the prefix
  • Use two: grouping a week of fallbacks by cause tells you exactly what the next routing prompt is missing
  • Use three: it feeds offline evaluation, since routing accuracy can only be scored if the decision and reason were recorded
  • Shape it as a structured prefix around a human sentence: aggregate on the prefix, read the sentence for one case
  • Keep the model's original judgement through fallbacks and overrides; store the basis rather than raw user text, cap the length, and reuse the tracing run id

  答题要点

  • 路由决策由模型做出且不可复现，理由必须在当时写下来，否则那次判断永远丢了——它是唯一的审计证据
  • 用途一：把「模型判错」和「解析或兜底出错」分开，靠 fallback 加原因码一眼分辨
  • 用途二：把一周内落进兜底的请求按原因分组，直接得到下一版分诊提示词该补什么
  • 用途三：它是离线评估的输入，分诊准确率这个指标只有记了当时的判断与理由才评得了
  • 写法是结构化的壳加自然语言的芯：前缀用于聚合统计，人话用于看具体这一单
  • 兜底与人工改派都要保留模型原判；理由只写判断依据不写用户原文，长度设上限，并复用链路追踪的 run 标识

D21 Evaluation and Observability: a Golden Set, LLM-as-Judge, Tracing, a Failure-Rate/Cost Dashboard; Pi vs. LangGraph Summary; Week Three Retrospective

• What does observability look like for a multi-agent system, and how does it differ from a single agent?多 Agent 系统的可观测性要看哪些东西？和单 Agent 有什么不一样？

  Common in ChinaCommon overseasIntermediate#observability#tracing#distributed-systems

  How to reason about it · think before answering

  1. The hinge is differ. Saying add logs and metrics is a non-answer; name the structural difference.
  2. In one sentence: a single agent's call is a line, a multi-agent request is a tree. One request goes supervisor routing, planner splitting into three, three executors in parallel, a critic rejecting one, that one rerunning, then aggregation — flattened by time you cannot see nesting or which two ran concurrently.
  3. So spans must carry a parent pointer; that is the whole game. With it you have a tree, without it a flat list where you know what happened but not what triggered what. A span needs surprisingly few fields — id, parent, name, start and end, a few attributes — to reconstruct the entire tree.
  4. How the parent propagates is itself an interview point: do not thread a parentSpanId parameter through every function, because each new node then changes a signature and one omission breaks the chain. Use the language's implicit context — AsyncLocalStorage in JS, contextvars in Python, TaskLocal in Swift, and ScopedValue or ThreadLocal with explicit propagation across thread pools in Java.
  5. Then the four questions a dashboard must answer: how much is wrong (pass rate, routing accuracy, degradation rate, fallback rate), where is it slow (p50/p95), what did it cost, and which role spent the money (cost attributed per node). That last one is multi-agent specific and the most actionable — measured, executor nodes took over a third of spend, telling you immediately where to optimise.
  6. One foundational point: the dashboard is not a second instrumentation layer, it is an aggregation of traces. The same raw data read across is a tree and stacked up is a dashboard. Two separate sources will eventually disagree, after which nobody trusts either.
  7. Finally, tie back to routing: the routing decision is made by a model and the same sentence may route differently next time, so the routing rationale must be recorded — if you do not capture it then, that judgement is gone forever. It is the easiest thing to omit and the thing most needing post-hoc audit.

  分析过程 · 先想清楚再作答

  1. 题眼在「不一样」。答「加日志加监控」等于没答，要说清结构上的差别。
  2. 结构差别一句话：**单 Agent 的一次调用是一条线，多 Agent 是一棵树。** 一次请求走监督者路由、规划者拆三件、三个执行者并行、评审者打回一件、那件重跑、最后汇总——按时间平铺看不出谁在谁里面，也看不出哪两个是并行的。
  3. 所以 span 必须带**父指针**，这是全部关键：有它才是树，没它只是一张平铺列表，你知道发生过什么，却不知道谁触发了谁。一条 span 的字段少得出奇——id、父指针、名字、起止时刻、几个属性，就够还原整棵树。
  4. 父子关系怎么传下去也是个考点：**不要在每个函数上加一个 parentSpanId 参数**，每加一个节点都要改签名、漏一处断一截。用语言自带的隐式上下文——JS 的 AsyncLocalStorage、Python 的 contextvars、Swift 的 TaskLocal，Java 用 ScopedValue 或 ThreadLocal 配合线程池的显式传播。
  5. 然后说面板要回答哪四个问题：错了多少（通过率、路由准确率、降级率、兜底率）、慢在哪（p50/p95）、花了多少、**钱花在哪个角色身上**（按节点分摊）。最后一样是多 Agent 特有的，也最有用——实测执行者节点占了成本三分之一强，一眼就知道压成本先压哪儿。
  6. 还有一条地基性的：**面板不是另一套埋点，是 trace 的聚合**。同一份原始数据横着看是树、竖着堆是面板。两套数据来源迟早会对不上，然后没有人相信任何一个。
  7. 最后回指路由：路由决策是模型做的，同一句话下次未必给同样的答案，所以必须把**路由理由**一起记下来——当时不记，那次判断就永远丢了。这是多 Agent 里最容易漏、又最需要事后审计的一条。

  Key points

  • Structural difference: a single agent call is a line, multi-agent is a tree (route, split, parallel execute, critic reject, rerun, aggregate)
  • Spans need a parent pointer, or you have a flat list showing neither nesting nor parallelism
  • Propagate parentage through implicit context (AsyncLocalStorage / contextvars / TaskLocal), not a parameter on every signature
  • The dashboard answers four questions: how much is wrong, where it is slow, what it cost, and which role spent it — the last is multi-agent specific and most actionable
  • The dashboard must be an aggregation of traces, not separate instrumentation; two sources will disagree
  • Record the routing rationale: routing is a model decision, and uncaptured it is lost forever

  答题要点

  • 结构差别：单 Agent 一次调用是一条线，多 Agent 是一棵树（路由→拆分→并行执行→评审打回→重跑→汇总）
  • span 必须带父指针，否则只是平铺列表，看不出嵌套关系也看不出并行
  • 父子关系用语言自带的隐式上下文传（AsyncLocalStorage / contextvars / TaskLocal），不要在每个函数签名上加参数
  • 面板回答四个问题：错了多少、慢在哪、花了多少、钱花在哪个角色身上（最后一个是多 Agent 特有且最有用）
  • 面板必须是 trace 的聚合而不是另一套埋点，两套数据源迟早对不上
  • 路由理由必须记下来：路由是模型做的决策，当时不记那次判断就永远丢了

D22 Security: Prompt Injection, Least Privilege for Tools, Sandboxing Approaches, Secret Management

• How should secrets be managed in an agent system? Where must they never appear, and how do you rotate them without downtime?Agent 系统里的密钥应该怎么管理？它绝对不能出现在哪些地方，轮换要怎么做才能不停机？

  Common in ChinaCommon overseasIntermediate#secrets-management#security#observability

  How to reason about it · think before answering

  1. It reads like a giveaway, but there is one answer point specific to agents, and missing it makes you sound like a generic backend engineer: secrets must never enter the LLM context. The interviewer asked about an agent system, and that is the line he is waiting for.
  2. Give the four 'nevers', one line each. Never in code — hardcoding hands the secret to everyone with read access, and deleting the line does not remove it from git history. Never in logs — the highest-frequency leak channel; nobody prints a secret on purpose, but 'log the whole request header so we can debug' is universal. Never in the LLM context. Never in error messages — responses to the frontend and exceptions thrown upstream are both outbound channels.
  3. Expand the third one, since it is what differentiates the answer: once a secret is in the context it will be sent to the model vendor, stored in conversation history, written into traces, and eventually read out loud by some prompt injection. What the agent needs is the capability to call an API, not the key itself — the key stays inside the tool implementation, and the model only ever sees the tool name and its arguments.
  4. Then the mechanics: redact at a single logging exit rather than trusting callers. Relying on everyone to mask by hand guarantees a miss. Do it in the one place logs leave the process, with two passes — replace known secret values from the environment, then catch the rest with generic shape patterns. Route the exception path through the same exit, because stack traces routinely carry connection strings with credentials.
  5. Storage and rotation: dotenv plus gitignore locally; in production a secret manager the process reads at startup under its own workload identity, never values baked into an image or a deployment manifest. Rotate dual-key: accept old and new simultaneously, shift traffic to the new one, confirm the old one has no remaining callers, then revoke. A single-shot swap always leaves a failure window on some replica.
  6. Expect the follow-up: how often do you rotate? The interval is secondary — what you should actually rehearse is whether you can revoke and replace a suspected-leaked key within five minutes. Saying that shows you are thinking about incident response rather than a compliance checkbox.

  分析过程 · 先想清楚再作答

  1. 这题看着是送分题，但有一个专属于 Agent 的答案点，答不出来就只是通用后端水平：密钥不能进 LLM 上下文。面试官问的是 Agent 系统，这一条就是他在等的。
  2. 先给四不入，一条一句：不入代码（写死在源码里等于给了所有有仓库读权限的人，而且删掉那一行 git 历史里还在）；不入日志（最高频的泄漏渠道，没人故意打印密钥，但「把请求头整个打出来方便排查」每个团队都干过）；不入 LLM 上下文；不入错误信息（返回给前端的报错和抛给上游的异常都是对外出口）。
  3. 把第三条展开，这是本题的差异点：密钥一旦进了上下文，就意味着它会被送到模型厂商、被存进会话历史、被写进 trace，然后在某一次提示词注入里被完整地念出来。正确的形态是 Agent 需要的是「能调用某个 API」这个能力，而不是那把钥匙本身——密钥留在工具的实现里，模型只看得到工具名和参数。
  4. 再给落地手段：日志出口统一脱敏，不靠调用方自觉。靠每个人写日志时记得手动打码，一定会漏。做法是在唯一的日志出口做替换，两条路一起用——进程里已知的密钥值整段替换，再用通用形状兜底那些不是从环境变量来的密钥。异常处理那一支也要走同一个出口，堆栈里经常夹着带密钥的连接串。
  5. 存储与轮换：本地开发用 .env 加 gitignore；线上走密钥管理服务，进程启动时按自己的身份去取，不要把值烤进镜像或写进部署清单。轮换要双活——同时允许新旧两把 key，流量切到新 key、观察到没有旧 key 的调用了再吊销，一次性替换必然在某个副本上留下失败窗口。
  6. 可以预期的追问：轮换周期定多久？周期是次要的，真正要演练的是「能不能在 5 分钟内换掉一把疑似泄漏的 key」。答得出这一句，说明你想的是事故响应而不是合规打卡。

  Key points

  • Four nevers: never in code, never in logs, never in the LLM context, never in error messages
  • The agent-specific one is the context — anything there reaches the vendor, the history and the traces, and can be read out by an injection
  • The agent needs the capability to call an API, not the key; the key stays inside the tool implementation
  • Redact at one logging exit instead of trusting callers, and route the exception path through it too
  • Use a secret manager with workload identity in production, and rotate dual-key: accept both, shift traffic, verify no old callers, then revoke

  答题要点

  • 四不入：不入代码、不入日志、不入 LLM 上下文、不入错误信息
  • Agent 特有的一条是不入上下文——进了上下文就会被送到厂商、存进历史、写进 trace，并可能被注入念出来
  • Agent 需要的是「能调用某个 API」的能力而不是钥匙本身，密钥留在工具实现里
  • 日志出口统一 redact，不靠调用方自觉；异常路径走同一个出口，堆栈里常夹着连接串
  • 线上走密钥管理服务按身份拉取；轮换用双活，新旧同时有效、切流量、确认无旧调用再吊销

MCP in 7 Days: Wire Tools Into Any Agent

D7 Productionizing and Retrospective: Writing Evals for Tools, Versioning, Publishing to npm and a Registry, Observability, and a Capstone Project

• A user reports that one of your MCP tools is 'always getting it wrong' in production. In what order do you investigate?线上有人反馈某个 MCP 工具「总是调不对」。你按什么顺序排查？

  Common in ChinaCommon overseasIntermediate#observability#debugging

  How to reason about it · think before answering

  1. This tests ordering, not breadth. Anyone who opens with logs and stack traces gets asked how they know the problem is server-side at all.
  2. Step zero is translating 'getting it wrong' into three mutually exclusive symptoms, without which everything after is guesswork: it was never called, it was called with wrong arguments, or it was called correctly and returned the wrong thing. Asking whether it did nothing or did the wrong thing, or simply checking whether a call was logged, separates them.
  3. Each symptom has its own path. Never called means the description is at fault: run the eval set and see whether happy paths or edges dropped, since happy-path drops mean a vague description and edge drops mean the sentence distinguishing similar tools is missing. Wrong arguments means the schema is at fault: ambiguous field names, unstated formats, wrong required markers. The signal is a persistently high tool-execution error rate, which usually means the schema is unclear rather than the model being dumb. Only a wrong result is a code problem, and only then do unit tests and logs matter.
  4. State the metric ordering too: error rate first, selection accuracy second. A normal error rate with unhappy users almost always means the tool is not being chosen; a spiking error rate sends you to the code and upstream. Watch P95, not the mean, because a remote tool degrading from 200 milliseconds to 8 seconds barely moves an average.
  5. Volunteer two commonly missed causes. Aggregation collisions: the client is connected to several servers, two tools share a name, and the model picked the other one, so your server was never called and investigating it will never find anything. And version or caching: list results carry ttlMs cache hints, so the client may hold a stale tool list, and refresh depends on a listChanged notification.
  6. Conclusion: the chain has four links, description, schema, aggregation and caching, and implementation. Walk it in the order the model sees it, because the earliest links produce no error logs and are therefore the ones people skip.
  7. Likely follow-up: how do you keep evidence? Emit one structured log per call with tool name, an argument digest plus field names, the isError flag, duration, and whether a human confirmed the call, that last column being the only way to distinguish user intent from the model acting on its own.

  分析过程 · 先想清楚再作答

  1. 这题考的是排查的**顺序**，不是知识点的多少。上来就贴日志和堆栈的人会被追问「你怎么知道问题在服务端」。
  2. 第零步是把「调不对」翻译成三种互斥的现象，这一步不做后面全是猜：一是**没被调**（模型压根没选这个工具）；二是**调了但参数错**；三是**调了参数也对，但结果不对**。问一句「那次它是没动，还是动了但做错了」，或者直接去日志里看有没有这条调用记录，就能分开。
  3. 对应三条不同的路。没被调，问题在**描述**：去跑评估集，看正例还是边界掉了；正例掉说明描述写糊，边界掉说明缺了区分相似工具的那句话。参数错，问题在 **schema**：看字段名是不是有歧义、描述里有没有写清格式、必填项是不是标对了；这类问题的信号是错误率里工具执行错误持续偏高——那通常不是模型笨，是 schema 没说清。结果不对才是代码问题，这时候才轮到单元测试和日志。
  4. 指标层面的顺序也说一下：**先看错误率，再看选中率**。错误率正常但用户说不好用，八成是选不中；错误率飙了才去看代码和上游。耗时看 P95 不看平均值，远程服务端上一个工具从 200 毫秒退化到 8 秒，平均值可能只动一点点。
  5. 还有两条容易被忽略但很常见的原因，要主动提。一是**聚合冲突**：客户端连了多个服务端，两个工具重名，模型选中的是另一个服务端的那个——这时候「你的工具」根本没被调，查你的服务端永远查不出来。二是**版本或缓存**：列表结果带 ttlMs 缓存提示，客户端可能拿着旧的工具清单；工具清单变了要靠 listChanged 通知才会重新拉。
  6. 结论：这条链上有四个环节——描述、schema、聚合与缓存、实现。**按模型看得见的顺序从前往后查**，因为越靠前的环节越不产生错误日志，也就越容易被跳过。
  7. 可预期的追问：怎么留证据？每次调用记一条结构化日志，字段里要有工具名、参数摘要与字段名、是否 isError、耗时、以及这次调用有没有经过人工确认；最后那一栏是事后区分「用户授意」和「模型自作主张」的唯一依据。

  Key points

  • First split 'getting it wrong' into never called, wrong arguments, or wrong result; the split decides where to look
  • Never called points at the description and the eval set; wrong arguments at the schema; only a wrong result at the code
  • Check error rate before selection accuracy, and read P95 rather than the mean
  • Do not miss aggregation collisions, where another server's same-named tool was chosen, or a stale cached tool list

  答题要点

  • 先把「调不对」分成没被调、参数错、结果错三种互斥现象，再决定查哪里
  • 没被调查描述并跑评估集；参数错查 schema；结果错才轮到代码与日志
  • 指标顺序是先错误率再选中率；耗时看 P95 不看平均值
  • 别漏掉聚合重名（选中的是别的服务端的同名工具）和工具清单缓存这两类原因

RAG in 14 Days: From Retrieval to Trustworthy Answers

D10 Query-Side Optimization: Rewriting, Hypothetical Document Embeddings, Multi-Query, Step-Back Prompting, and Intent Routing

• What happens when intent routing misclassifies, and how would you design the fallback?意图路由判错了会怎样？你会怎么设计兜底？

  Common in ChinaCommon overseasIntermediate#intent-routing#fallback#observability

  How to reason about it · think before answering

  1. This tests whether you have thought about the direction of the error. A router is a classifier and classifiers misfire; "add more training data" is not a fallback design.
  2. Break the errors down by direction — that is the backbone of the answer. Across three routes (direct answer, single-hop, multi-hop) the six confusions carry wildly asymmetric costs. Routing a retrieval-worthy question to a direct answer leaves the model with no material at all, so it fabricates: the most expensive error. Routing chit-chat to single-hop merely wastes one retrieval. Routing multi-hop to single-hop just yields an incomplete answer.
  3. The conclusion follows: bias the fallback toward spending a little more, and default to single-hop retrieval whenever the classifier is unsure. Single-hop is the cheapest error to make, and it is recoverable — with partial material the model can still say it only found half the answer; with no material it can only invent one.
  4. Add a runtime fallback, which beats better up-front classification: after a direct-answer routing, if the draft reply contains figures, amounts or dates that need a source, fall back to retrieval and answer again; after a single-hop routing, if no candidate clears the admission gate, escalate to multi-hop or abstain. Correcting the earlier decision with the later observation is the single most useful pattern in routing systems.
  5. Mention observability: log every routing decision with the raw question, the label, and whether a fallback fired. Without that log you know neither how accurate the router is nor what to train the next version on.
  6. Expect "when should you skip routing entirely?" When chit-chat is a small share of traffic and multi-hop questions are rare, the classification call costs more than it saves. In our 30-document lab the real gain from routing was not saved retrievals but the ability to give recognised multi-hop questions a larger context budget.

  分析过程 · 先想清楚再作答

  1. 这题在考「有没有想过错误的方向」。路由是分类器，分类器一定会错；只答「多加训练数据提高准确率」的，等于没回答兜底怎么设计。
  2. 先把错误按方向拆开，这一步是整题的骨架：三条路（直接回答、单跳检索、多跳检索）两两误判，代价完全不对称。把该检索的判成直接回答，模型手里一点材料都没有，只能编，这是最贵的一种错；把闲聊判成单跳，只是白花一次检索；把多跳判成单跳，只是少查一轮、答得不全。
  3. 结论顺势就出来了：**兜底方向要偏向「多花一点钱」，判不出来一律退回单跳检索。** 单跳是三条路里错得最轻的一条，而且它的错误是可恢复的——材料不全模型还能说「资料里只查到一半」，材料为空它就只能编。
  4. 再补一层运行时兜底，比事前分类更管用：分类成直接回答之后，如果模型的回答里出现了具体数字、金额、日期这类需要出处的内容，就回退去检索一次再答；分类成单跳之后，如果检索侧一条都没过门槛，就升级走多跳或直接拒答。**用后一步的观测结果纠正前一步的判断**，这是路由系统最实用的一条设计。
  5. 还要提一句可观测性：路由的每一次判定都要落日志，带上原始问题、判定结果、后续是否发生了兜底升级。没有这份日志，你既不知道路由准不准，也没法攒出下一版的训练集。
  6. 可预期的追问是「什么时候干脆别做路由」。答：流量里闲聊占比很低、且多跳问题很少时，路由省下的钱还不够付分类调用的钱，这时候直接全部走单跳更划算——我们在 30 篇语料的实验里就看到，路由真正的收益并不在省检索，而在于认出多跳之后给它更高的上下文预算。

  Key points

  • The three routes have asymmetric error costs: sending a retrieval-worthy question to a direct answer is the worst, while routing chit-chat to single-hop only wastes one retrieval.
  • Bias the fallback toward spending more: default to single-hop whenever the classifier is unsure, since that error is the mildest and is recoverable.
  • Add runtime fallbacks: re-retrieve if a direct answer contains figures that need a source; escalate or abstain if no single-hop candidate clears the gate.
  • Log every routing decision — raw question, label, whether a fallback fired — for both monitoring and the next training set.
  • When chit-chat and multi-hop are both rare, the classification call costs more than it saves; route everything to single-hop instead.

  答题要点

  • 三条路的误判代价不对称：把该检索的判成直接回答最贵（模型没材料只能编），把闲聊判成单跳只是白花一次检索。
  • 兜底方向偏向多花钱：判不出来一律退回单跳检索，它是错得最轻且可恢复的一条路。
  • 加运行时兜底：直接回答里出现需要出处的数字就补一次检索；单跳检索一条都没过门槛就升级或拒答。
  • 每一次路由判定都落日志（原始问题、判定结果、是否触发兜底），既用于监控也用于攒下一版训练集。
  • 闲聊与多跳占比都很低时，路由省的钱付不起分类调用，直接全走单跳更划算。

D14 Capstone Project and Retrospective: A Multi-Tenant Enterprise Knowledge-Base Q&A, a RAG Decision Map, and an Interview Deep Dive

• Users report that your live RAG system 'answers inaccurately'. What is your triage order?RAG 系统上线后用户反馈「答得不准」，你的排查顺序是什么？

  Common in ChinaCommon overseasIntermediate#debugging#failure-modes#observability

  How to reason about it · think before answering

  1. This one is almost guaranteed to be asked, and most people answer with a flat list of possibilities: maybe chunking, maybe the prompt, maybe the model. A list is not triage. Triage means an order, a decision rule at each step, and each step eliminating half the search space.
  2. First decompose the complaint. 'Inaccurate' hides at least four distinct failures whose fixes do not transfer: off-topic answers, partial answers, misaligned citations, and stale content. So the first action is not to change a setting, it is to obtain the specific question and answer and classify it into one of those four.
  3. Then give the order along with its justification: read the pipeline right to left, fix it left to right. Right to left because the generated answer is what you see first; left to right because upstream errors are amplified downstream — no prompt can recover a document retrieval never fetched. Concretely: dump the candidate pool and the final context for that question, and check whether the answer document is in the pool at all. Absent means a retrieval debt; present but below the admission gate means a gate debt; admitted but never packed into the context budget means chunks too large or budget too small; all present and still unused means it is finally a generation problem.
  4. One detail worth volunteering because it is easy to get wrong: for multi-hop questions, diagnose the documents that are missing, not whether any one of them was retrieved. In our experiment one question needed two documents; the first ranked first every time and the second never entered the candidate pool at all. Judging by 'any of them' labels it a budget problem, and you can spend a full day tuning budgets to no effect. This distinction only occurs to someone who has actually triaged question by question.
  5. The fourth class, stale content, happens outside the question path and has its own rule: first check whether reconciliation even noticed the edit (was the content hash computed after line-ending normalization?), then check whether the cache key includes the index version and the permission scope. 'When must this expire' is equivalent to 'is that thing part of the key' — leave something out of the key and changes to it will never invalidate the entry.
  6. Expected follow-up: how do you stop relying on manual triage? Build the classification into the evaluation panel so every missed question is automatically labelled with one of the four classes, and report it per tenant. A global average dilutes one customer's collapse across the whole population, and that customer is exactly the one who will file the complaint.

  分析过程 · 先想清楚再作答

  1. 这题几乎是必考题，而绝大多数人答成一堆并列的可能性：可能是切块问题、可能是提示词问题、可能是模型不行。并列不是排查，排查的意思是**有顺序、有判据、每一步能把可能性砍掉一半**。
  2. 先把「答得不准」这四个字拆开——它至少塞了四种病，而且修法互不通用：答非所问、只答得出片段、引用错位、更新不生效。所以第一个动作不是改配置，是**拿到具体的问题和回答，把它归到这四类里的一类**。
  3. 然后给顺序，而且要说清顺序的理由：**排查从右往左看、修复从左往右修**。从右往左是因为你最先看到的是生成结果；从左往右是因为上游的错会被下游放大——检索没捞到的东西，再好的提示词也救不回来。具体走法是：打印这一问的候选池和最终上下文，先看答案文档在不在候选池里。不在，是检索的债；在候选池但没过准入门槛，是门槛的债；过了门槛却没装进上下文预算，是块太大或预算太小；都进了而模型没用上，才轮到生成侧。
  4. 这里有一个容易写错的细节值得主动讲：**多跳题的诊断对象是缺的那几篇，不是「有没有捞到任意一篇」**。我们实验里有一道题要同时命中两篇，第一篇稳稳排第一、第二篇一次都没进候选池；用「任意一篇」去判会把它归成预算问题，然后你去调预算，调一整天也没用。这一条区分度很高，因为它只有真的按题排查过才想得到。
  5. 第四类「更新不生效」发生在问答之外，判据是另一条：先看对账认没认出这篇改了（内容指纹算之前有没有做换行归一化），再看缓存的 key 里有没有把索引版本和权限范围算进去。「什么时候必须失效」等价于「key 里有没有把那样东西算进去」，key 少放一样，那样东西变了缓存就不会失效。
  6. 可预期的追问：怎么让这套排查不靠人肉？答案是把分类做进评估面板——每一道没中的题自动标出它属于四类中的哪一类，并按租户分开统计。全局平均会把单个客户的塌方按人头摊薄，而线上会投诉的恰恰是那个客户。

  Key points

  • Classify the complaint into four failures first — off-topic, partial, misaligned citation, stale — because their fixes do not transfer.
  • Read right to left, fix left to right: dump the candidate pool and final context and find which layer the answer document stalls at.
  • The four rules in order: never retrieved, retrieved but below the gate, admitted but squeezed out of the budget, packed but unused by the model.
  • For multi-hop, diagnose only the missing documents; judging by 'any one retrieved' mislabels a never-retrieved case as a budget problem.
  • For stale content, check reconciliation and the cache key: what must expire is exactly what the key must contain.

  答题要点

  • 先把「答得不准」归类成四种病：答非所问、只答得出片段、引用错位、更新不生效——修法互不通用。
  • 排查从右往左看、修复从左往右修：先打印候选池与最终上下文，看答案文档卡在哪一层。
  • 四层判据依次是：没进候选池、进了没过门槛、过了没装进预算、都进了模型没用上。
  • 多跳题只诊断缺的那几篇；用「有没有捞到任意一篇」会把「根本没捞到」误判成预算问题。
  • 「更新不生效」查对账与缓存 key：什么时候必须失效，等价于 key 里有没有算进那样东西。