Interview Bank

From Frontend Engineer to Agent Engineer in 30 Days

D1 LLM API Basics: messages/roles, Tokens, Streaming, Temperature; What an Agent Actually Is

• A streaming reply is cut off mid-way. What do the client and server each do, and can EventSource auto-reconnect help?流式回复到一半网络断了，前端和后端各要做什么？EventSource 的自动重连能用上吗？

  Common in ChinaCommon overseasIntermediate#streaming#reliability#sse

  How to reason about it · think before answering

  1. The trap is the second half: people who memorized 'SSE reconnects automatically' answer yes, which is wrong.
  2. Native EventSource does auto-reconnect per spec, sending Last-Event-ID, with the server marking events via id: and setting the interval via retry: — but it only issues GET and requires Content-Type text/event-stream.
  3. LLM chat APIs require POST because messages go in the body, so real clients use fetch plus hand-written SSE parsing, where none of that machinery applies.
  4. So the client owns detection, retry and buffering of what arrived; the server's job is making retries safe — resumable output and idempotent side effects.
  5. Give the continuation strategy and its limits: feed the received prefix back as context, but tool-use and thinking blocks cannot be partially recovered — resume from the last complete text block.
  6. Follow-up to expect: does a non-200 reconnect? Per spec no — a non-200 status or wrong Content-Type fails the connection, and a 204 tells the browser to stop reconnecting.

  分析过程 · 先想清楚再作答

  1. 这题的陷阱在后半句。很多人背过「SSE 自带重连」，就直接答自动重连能救——那是错的，必须先分清两种 SSE 用法。
  2. 浏览器原生 EventSource 确实按规范自动重连：重连时带 Last-Event-ID 请求头，服务器用 id: 打点、用 retry: 设间隔；但它只能发 GET，且要求响应 Content-Type 是 text/event-stream。
  3. 而 LLM chat API 必须 POST（messages 要放在请求体里），所以实际用的是 fetch 加手写 SSE 解析——EventSource 那套自动重连一行都用不上。
  4. 于是前端职责变成：自己判定断流、自己重试、自己保存已收到的部分。后端职责是让重试是安全的——响应可续、副作用幂等。
  5. 给出续写策略并说清边界：把已收到的内容作为上下文构造续写请求；但工具调用块和思考块无法部分恢复，只能从最近的完整文本块续。
  6. 可预期追问：非 200 响应会重连吗？按规范不会——状态码不是 200 或 Content-Type 不对，连接直接判定失败；服务器还可以用 204 主动叫停重连。

  Key points

  • Separate the two SSE modes: native EventSource auto-reconnects with Last-Event-ID but is GET-only; LLM APIs use POST and cannot rely on it
  • The client must therefore detect the break, retry itself, and keep whatever text already arrived
  • Continuation: send the received prefix as context so the model resumes rather than restarting the turn
  • Limits: tool_use and thinking blocks cannot be partially recovered; resume from the last complete text block
  • The server must make retries safe: resumable responses, idempotent tool side effects, correct billing for tokens already produced

  答题要点

  • 先区分两种 SSE：浏览器原生 EventSource 自动重连并带 Last-Event-ID，但只能 GET；LLM API 走 POST，用不上这套
  • 所以前端要自己检测断流、自己重试，并保留已收到的部分内容
  • 续写策略：把已收到的内容作为上下文发起新请求，让模型接着写，而不是整轮重来
  • 边界：tool_use 和 thinking 块无法部分恢复，只能从最近的完整文本块续
  • 后端要保证重试安全：响应可续、工具副作用幂等，并对已产生的用量正确计费
• The user backgrounds the app or closes the tab. How do you restore a reply that was still being generated?用户切到后台或者直接关掉网页，回来后怎么恢复那条还在生成的回复？

  Common in ChinaCommon overseasDeep dive#streaming#reliability#architecture

  How to reason about it · think before answering

  1. First separate this from a dropped connection: the client is gone, so no client-side retry will ever run.
  2. That leaves one option — the generation must outlive the client, which means persisting the stream server-side.
  3. Concretely: assign a stream id per generation; the server pushes tokens to the live connection while also writing them to storage such as Redis, and the chat record stores that activeStreamId.
  4. Recovery is a separate GET endpoint: the client asks with the chat id, the server locates the stream by activeStreamId and resumes; with no active stream it returns 204.
  5. Name the costs, not just the design: extra storage, expiry/cleanup, and concurrency when several connections consume the same stream.
  6. Extension: this differs from ordinary message persistence because the reply is still being produced — you need a resumable stream, not a static row.

  分析过程 · 先想清楚再作答

  1. 先识别这题和「网络断了」不是同一个问题：客户端已经不存在了，任何写在前端的重试逻辑都不会执行。
  2. 由此推出唯一出路：生成过程必须能脱离这个客户端独立存活，也就是把流本身放到服务端持久化。
  3. 落到具体架构：发起请求时给这轮生成分配一个流 id，服务端一边把 token 推给当前连接，一边把同样的内容写进 Redis 之类的存储；会话记录里保存这个 activeStreamId。
  4. 恢复路径是另开一个 GET 端点：客户端带着会话 id 请求，服务端按 activeStreamId 找到那条流并接着推；找不到活跃流就返回 204，让前端知道没有需要恢复的东西。
  5. 说清代价，别只说方案：多了一份存储、一套过期清理、以及「同一条流可能被多个连接消费」的并发问题。
  6. 延伸：这套结构和普通聊天产品的「消息已持久化，重进会话直接读库」不同——区别在于回复还在生成中，需要的是可续的流而不是一条静态记录。

  Key points

  • The client is gone, so recovery must live server-side: the generation has to outlive the connection
  • Assign a stream id at start; the server writes tokens to Redis while streaming, and the chat stores activeStreamId
  • Resume through a dedicated GET endpoint that replays the active stream, returning 204 when there is none
  • Costs: extra storage, expiry and cleanup, and concurrent consumers of one stream
  • It differs from plain message persistence because the reply is still in flight, so you need a resumable stream

  答题要点

  • 客户端已经不在了，前端重试无从谈起，必须让生成过程在服务端独立存活
  • 发起生成时分配流 id，服务端边推送边把内容写进 Redis，会话里记录 activeStreamId
  • 恢复走单独的 GET 端点：按会话 id 找到活跃流接着推，没有活跃流就返回 204
  • 代价：额外存储、过期清理，以及同一条流被多个连接消费的并发处理
  • 与「消息持久化后重新读库」的区别在于回复仍在生成中，需要的是可续的流
• After a retry, how do you avoid double billing and re-executing tool calls that already ran?断线重试之后，怎么保证不重复计费、也不重复执行已经做过的工具调用？

  Common in ChinaCommon overseasDeep dive#reliability#tools#idempotency

  How to reason about it · think before answering

  1. Split it in two: billing is a bookkeeping problem, tool side effects are an execution problem, and they have different fixes.
  2. Billing: meter server-side by tokens actually produced, not by request count. Tokens produced before the break are real cost; so are retry tokens. The point is not to count the same batch twice.
  3. That needs a stable identifier: give each generation a run id and dedupe usage records by run id plus sequence.
  4. Tools: the danger is side-effecting tools — transfers, messages, orders. The fix is an idempotency key derived from the call arguments, checked before execution.
  5. Add the state-machine view: record each call as pending / running / done and replay only what is unfinished.
  6. Follow-up: who generates the idempotency key? The caller must, and pass it along — a server-generated key cannot stay stable across retries.

  分析过程 · 先想清楚再作答

  1. 先把问题拆成两半：计费是「记录问题」，工具副作用是「执行问题」，两者的解法不同，混在一起答会含糊。
  2. 计费侧：用量应该在服务端按实际收到的 token 记账，而不是按「请求次数」。断在中途已经产生的 token 是真实成本，要照记；重试产生的是新成本，也要照记——关键是别把同一批 token 记两遍。
  3. 为此需要一个稳定的标识：给每轮生成一个 run id，用量记录以 run id + 序号去重，重放同一段不会重复入账。
  4. 工具侧：真正危险的是有副作用的工具（转账、发消息、下单）。解法是幂等键——由调用参数派生一个稳定的 key，执行前先查这个 key 是否已有结果，有就直接返回旧结果。
  5. 补一层状态机视角：把每次工具调用记为「待执行 / 执行中 / 已完成」，重试时只重放未完成的部分，已完成的直接取结果，这也是恢复中断任务的通用做法。
  6. 常见追问：幂等键该谁生成？应由客户端或调度侧生成并随请求传递，服务端自己生成就没法跨重试保持一致。

  Key points

  • Separate billing (bookkeeping) from tool side effects (execution); they need different mechanisms
  • Meter by tokens actually produced, deduped by run id plus sequence so one batch is never counted twice
  • Guard side-effecting tools with an idempotency key derived from the call arguments
  • Model each tool call as pending / running / done and replay only unfinished work
  • The caller must generate and pass the idempotency key so it stays stable across retries

  答题要点

  • 拆成两个问题：计费是记账问题，工具副作用是执行问题，解法不同
  • 计费按服务端实际产生的 token 记，用 run id 加序号去重，避免同一批 token 重复入账
  • 有副作用的工具用幂等键：由调用参数派生稳定 key，执行前先查是否已有结果
  • 把每次工具调用记成待执行/执行中/已完成的状态机，重试只重放未完成的部分
  • 幂等键要由调用方生成并随请求传递，服务端自行生成无法跨重试保持一致
• On mobile, connectivity is flaky. How would you design the reconnection strategy for a chat feature?移动端 App 里的对话，网络频繁抖动，你会怎么设计重连策略？

  Common in ChinaCommon overseasIntermediate#reliability#mobile#streaming

  How to reason about it · think before answering

  1. Start with what makes mobile different: network switches between WiFi and cellular, the OS suspends apps, background time is limited.
  2. Use exponential backoff with jitter; jitter is the commonly missed part that prevents a thundering herd when a wide outage clears.
  3. Set ceilings: max attempts and max interval, then surface an explicit reload action instead of retrying silently forever.
  4. Distinguish a brief blip from being genuinely offline: subscribe to OS connectivity events, stop retrying when offline, and reconnect on the restore event — far cheaper on battery than blind timers.
  5. Combine with server-side persistence: after the OS kills the app, resume by chat id rather than reconstructing from local cache.
  6. Finally the send path: queue outgoing messages while offline and replay them in order, each with an idempotency key.

  分析过程 · 先想清楚再作答

  1. 先说明移动端和浏览器的差别：网络在 WiFi 与蜂窝之间切换、App 会被系统挂起、后台执行时间受限，所以不能照搬网页那套。
  2. 重试节奏用指数退避加随机抖动。抖动这一条常被忽略，但它是防止大面积断网恢复后所有客户端同时涌上来把服务打垮的关键。
  3. 要设上限：最大重试次数与最大退避间隔，超过就转成显式的「重新加载」按钮交给用户，而不是无限静默重试。
  4. 区分「短暂抖动」和「真的没网」：监听系统的网络状态变化，没网时直接停止重试并进入离线态，等网络恢复事件再立刻重连，比盲目定时重试省电得多。
  5. 结合上一题的服务端持久化：App 被系统杀掉后重进，靠会话 id 请求恢复端点，而不是指望本地缓存拼出完整回复。
  6. 最后补发送侧：用户在离线时发出的消息进本地队列，恢复后按序重发，且每条带幂等键，避免重复发送。

  Key points

  • Mobile differs: network handoffs, OS suspension, limited background time — do not copy the web strategy
  • Exponential backoff with jitter, where jitter prevents a reconnect storm when an outage clears
  • Cap attempts and interval, then hand the user an explicit reload instead of retrying forever
  • Listen to OS connectivity events: stop while offline, reconnect on restore, which saves battery over polling
  • Resume replies via server-side persistence by chat id; queue outgoing messages with idempotency keys

  答题要点

  • 移动端特殊性：WiFi 与蜂窝切换、App 被挂起、后台执行时间受限，不能照搬网页策略
  • 指数退避加随机抖动，抖动用于避免大面积恢复时的重连风暴
  • 设最大重试次数与最大间隔，超过后转为显式的重新加载入口，不做无限静默重试
  • 监听系统网络状态：离线直接停重试进入离线态，收到恢复事件再重连，比定时轮询省电
  • 回复恢复依赖服务端持久化，靠会话 id 请求恢复端点；发送侧用本地队列加幂等键按序重发
• A user pressing stop and a dropped connection both look like a closed connection server-side. How do you tell them apart?用户主动点「停止生成」和网络意外断开，在服务端看起来都是连接没了，怎么区分处理？

  Common in ChinaCommon overseasDeep dive#streaming#reliability#ux

  How to reason about it · think before answering

  1. Say why it matters: stop means the user no longer wants the output, so free compute and end the run; a drop means they still want it, so preserve the result for resumption.
  2. Connection state alone cannot distinguish them — it looks identical — so you need an explicit signal.
  3. Give stop its own endpoint: the client calls it with the run id before closing, and the server marks the run as user-cancelled and aborts the upstream call.
  4. Treat a bare connection close as an unexpected drop: keep persisting output and hold the stream for resumption.
  5. Add the real-world caveat: the stop request itself may fail to send when the network is down, so the server needs a fallback — end a stream with no consumer after a timeout.
  6. Extend to billing: both cases still owe for tokens already produced, since the upstream provider has charged; they differ only in whether output is retained.

  分析过程 · 先想清楚再作答

  1. 先点破为什么要区分：主动停止是「用户不想要了」，应当立即释放算力并结束这轮；意外断开是「用户还想要」，理应保留结果供恢复。处理反了，用户要么白花钱，要么回来发现内容没了。
  2. 所以不能只靠 TCP 连接状态判断——它对两种情况的表现是一样的。必须有一个显式信号。
  3. 做法是给「停止」单独一个接口：前端点停止时先调这个接口，带上 run id，服务端据此把该轮标记为「用户取消」，再中止上游模型调用。
  4. 而单纯的连接关闭一律按「意外断开」处理：继续把已生成内容落盘、保留可恢复的流，等客户端回来续。
  5. 补一个现实约束：停止请求本身也可能因为断网而发不出去。所以服务端还需要兜底——比如流没有任何消费者超过一定时间就自行结束，避免算力空转。
  6. 延伸到计费：两种情况都要为已经产生的 token 计费，因为上游厂商已经收了钱；区别只在于要不要保留结果和是否继续生成。

  Key points

  • The semantics are opposite: stop frees compute immediately, a drop preserves output for resumption
  • Connection state cannot distinguish them, so add an explicit stop endpoint carrying the run id
  • Treat a bare close as an unexpected drop: keep persisting and hold the stream for resume
  • Fallback: the stop call may itself fail to send, so end streams with no consumer after a timeout
  • Both still bill for tokens already produced; they differ only in retention and whether generation continues

  答题要点

  • 两者语义相反：主动停止要立即释放算力并结束，意外断开要保留结果等待恢复
  • TCP 连接状态无法区分，必须有显式信号：给停止单独一个接口，带 run id 标记为用户取消
  • 只收到连接关闭一律按意外断开处理，继续落盘并保留可恢复的流
  • 兜底：停止请求本身也可能发不出去，服务端需对长时间无消费者的流自行结束
  • 计费上两者都要为已产生的 token 记账，区别只在于是否保留结果、是否继续生成

D2 How Tool Calling Works: JSON Schema, the tool_use Loop; Hand-Writing an Agent Loop With No Framework

• How do you keep an agent loop from running forever — is a max-step counter enough?怎么防止 Agent 循环停不下来？只加一个最大步数够吗？

  Common in ChinaCommon overseasDeep dive#agent-loop#reliability#cost

  How to reason about it · think before answering

  1. The second half is an open trap. 'Add a counter' is the passing grade; what they want is whether you know what a counter cannot catch.
  2. Explain why it runs away first: the finish reason stays tool_calls because the tool results are not moving the model forward — empty results, fields that do not answer the question, error text that never says what correct looks like. So the first line of defense is not a guard rail at all; it is writing tool results and error messages that carry information.
  3. Then three complementary hard limits: a step cap is the obvious one; a token and cost budget catches 'few steps, all of them expensive'; a per-step wall-clock timeout catches 'one call hung for two minutes'. A system with only a step cap can still blow its budget on a single enormous context.
  4. Add a semantic guard: detect repeats. The same tool with identical arguments twice in a row is almost always spinning. Cut it short and tell the model so — 'you already called this tool with exactly these arguments' — which usually converges faster than waiting for the counter to run out.
  5. Hitting the cap needs an honest ending: never return an empty string, give the user a sentence they can act on, and record cap hits as a metric. A rising cap-hit rate usually means a tool's description or return value needs fixing, not that the cap should be raised.
  6. Expect: what number do you pick? There is no universal one. Chat-style tasks usually fit in five to ten steps; retrieval-heavy tasks need more. Read the production distribution, take p99 plus headroom, and remember that the tighter the cap, the closer your system sits to a fixed workflow rather than an agent.

  分析过程 · 先想清楚再作答

  1. 后半句是明摆着的陷阱。只答「加一个计数器」是及格线，面试官真正想听的是你知道计数器拦不住什么。
  2. 先解释它为什么会停不下来：停止原因一直是 tool_calls，通常是因为工具返回的东西没帮模型前进——结果为空、字段答非所问、错误文案没说清该怎么改，于是它换个参数一试再试。所以第一层其实不是护栏，是把工具的返回值和错误文案写得有信息量。
  3. 再给硬护栏，三条互补：步数上限最直接；token 与成本预算拦的是「步数不多但每步都很贵」；单轮的墙上时钟超时拦的是「一步就卡了两分钟」。只有步数上限的系统，照样会被一次超长上下文的调用打爆预算。
  4. 语义层面再加一条：检测重复调用。同一个工具、同一份参数连续出现两次以上，几乎可以断定它在原地打转，直接截断并把「你已经用完全相同的参数调过这个工具了，换个思路或者告诉用户你做不到」回传给模型，往往比等步数耗尽更快收敛。
  5. 触顶之后必须有交代：不能静默返回空字符串，要给用户一句能理解的话；同时把触顶记成一个指标，触顶率上升通常意味着某个工具的描述或返回值该改了，而不是把上限调大。
  6. 可以预期的追问：上限设多少？没有普适值。聊天类任务 5 到 10 步通常够，需要多轮检索的任务可以更高。正确做法是看线上的步数分布，取 p99 再留一点余量，而不是拍脑袋——上限设得越死，你的系统就越靠近固定流程那一端，越不像一个 Agent。

  Key points

  • The root cause is usually uninformative tool results or error text, so fix that layer before adding guards
  • Three complementary hard limits: max steps, a token and cost budget, and a per-step wall-clock timeout
  • Add a semantic guard: identical tool plus identical arguments twice in a row means it is spinning — cut it and tell the model
  • Give the user an honest message when the cap is hit, and track the cap-hit rate as a signal that a tool needs fixing
  • Size the cap from the production step distribution, not intuition; a tighter cap makes the system a workflow rather than an agent

  答题要点

  • 根因通常是工具返回值或错误文案没信息量，模型无法前进只能反复重试，先把这层写好
  • 三条硬护栏互补：最大步数、token 与成本预算、单步墙上时钟超时，只有步数上限并不够
  • 语义护栏：同一工具加同一份参数连续重复调用即判定原地打转，截断并把这个事实回传给模型
  • 触顶要给用户一句交代，不能静默返回空；同时把触顶率当指标，上升说明工具该改而不是把上限调大
  • 上限值按线上步数分布取 p99 加余量；上限越死越接近固定流程，越不像 Agent

D4 Model Integration and System Prompts: a Multi-Provider Abstraction With Fallback, Overriding the Default Persona (dg P03/P04/M04)

• Why do production agents usually integrate more than one model provider?为什么生产级 Agent 通常要接入多个模型 provider？

  Common in ChinaCommon overseasBasic#model-routing#reliability

  How to reason about it · think before answering

  1. First decide whether this is an availability question or an architecture question; answering only 'so it doesn't go down' reads as inexperienced.
  2. Follow the causal chain: the model API is an external dependency, dependencies have failure rates, your ceiling is capped by theirs, so you either accept the cap or add redundancy.
  3. Quantify it: 99.5% monthly availability is about 3.6 hours of downtime; three independently failing providers push that to seconds. Orders of magnitude beat adjectives.
  4. The second reason shows engineering maturity: model pricing and capability shift monthly, and high switching cost means you stay on the expensive slow one out of inertia — coupling really costs you future optionality.
  5. Say the premise out loud before they ask: that order of magnitude assumes the three providers fail independently. If all three are model ids behind one aggregator gateway on a single key — which is what most first versions look like — the gateway going down takes all three with it, the redundancy is fake, and the aggregator has become the new single point of failure. Real independence means direct endpoints at different vendors, with separate credentials and billing. Naming this yourself signals operational experience far more than reciting 0.005 cubed.
  6. Expect the follow-up: isn't this more expensive? No — the happy path calls one provider; what costs money is fallback firing often, which is a signal to investigate the primary, not to remove redundancy.

  分析过程 · 先想清楚再作答

  1. 先判断这题问的是「可用性」还是「架构」。只答「防止挂掉」拿不到分，因为面试官想看的是你有没有真的算过账、踩过坑。
  2. 从一条因果链推：模型 API 是外部依赖 → 外部依赖必然有故障率 → 你的可用性上限被它锁死 → 所以要么接受这个上限，要么加冗余。
  3. 把可用性说成数字才有说服力：单家 99.5% 意味着每月约 3.6 小时不可用；三家独立故障时理论不可用时间降到秒级。数量级差异比形容词有力得多。
  4. 第二个理由往往被忽略，但更能体现工程视角：模型的价格和能力每月都在变，接入成本高会让你因为「改起来麻烦」而一直用贵的慢的那个——高耦合真正的代价是剥夺未来的选择权。
  5. 这里有个必须自己先说破的前提：那个数量级是拿「三家故障互不相关」算出来的。如果三家其实都走同一个聚合网关、共用同一把 key（很多人的第一版就是这样），网关一挂三家一起挂，冗余是假的，聚合网关反而成了新的单点。真正的独立要落到不同厂商的直连端点、各自的凭证和计费上。主动点破这一条，比背出 0.005 的三次方更能体现你真的部署过。
  6. 可以预期的追问：多接几家不是更贵吗？答案是不会——正常路径只调一家，多的只是配置和一层抽象；真正贵的是 fallback 被频繁触发，那说明你该查主 provider 而不是砍掉冗余。

  Key points

  • The model API is an external dependency; outages, rate limits and model deprecations are monthly realities
  • 99.5% monthly availability is roughly 3.6 hours down; multi-provider redundancy cuts that by orders of magnitude
  • Pricing and capability shift constantly, so an abstraction layer turns model swaps into config changes
  • The happy path still calls one provider — redundancy costs an abstraction, not a multiplied bill

  答题要点

  • 模型 API 是外部依赖，厂商故障、限流、模型下线都是每月都会遇到的日常，不是小概率事件
  • 单家 99.5% 可用性等于每月约 3.6 小时不可用；多家冗余能把理论不可用时间降低几个数量级
  • 价格与能力每月都在变，统一抽象层让换模型变成改配置，保住了未来做选择的自由
  • 正常路径只调一家，冗余的成本是一层抽象而不是多倍账单
• What trade-offs shape a model fallback strategy?设计模型 fallback 策略时要权衡哪些因素？

  Common in ChinaCommon overseasIntermediate#model-routing#reliability#cost

  How to reason about it · think before answering

  1. The word 'trade-offs' is the hinge: they are not asking for a for-loop, they want to know you understand fallback has costs.
  2. First key judgment: not every error deserves a fallback, and the test is not the leading digit of the status code but whether another provider could plausibly succeed. A 400 (malformed body) or 403 (blocked by safety policy) fails everywhere, so retrying repeats your own bug at double the cost and latency; a 408, 429 or 5xx is theirs and usually succeeds elsewhere. The two that people get wrong are 402 (out of credit) and 404 (model retired or renamed): both are 4xx, both look like your fault, and both are fixed by switching. A 401 depends on how credentials are managed — one shared gateway key fails everywhere, but per-provider keys mean a revoked key on A is survivable on B. Classification comes before retry.
  3. Cost: switching means paying for the same prompt twice, up to 3x across a three-provider chain. Volunteering this separates people who shipped from people who only read about it.
  4. Latency: serial fallback accumulates timeouts. Three providers at 15s each means a 45s wait — worse than failing fast. Timeouts must be per-provider with an overall budget.
  5. Thundering herd is the most common follow-up: when the primary rate-limits, shifting all traffic at once can take down the backup too. Hence circuit breaking — drop a provider after N consecutive failures, then probe with a trickle.
  6. Expect: how long do you drop it for? Exponential backoff with a half-open probe — the same pattern as database connection pool breakers.

  分析过程 · 先想清楚再作答

  1. 题眼在「权衡」两个字——面试官不要你背一个 for 循环，他要看你知不知道 fallback 是有代价的。
  2. 先拆出第一个关键判断：不是所有错误都该 fallback，而分类的依据不是状态码的首位数字，是「换一家有没有可能变好」。400 请求体不合法、403 被安全策略拦截，换谁都一样，重试只是把同一个 bug 再犯一遍、白花两倍的钱和时间；408 超时、429 限流、5xx 服务端故障是对方的问题，换一家大概率能成。最容易答错的是 402 余额不足和 404 模型被下线或改名——它们同属 4xx、长得像「你的问题」，其实换一家完全可能成功；401 则要看凭证怎么管，三家共用一把网关 key 时换了也没用，各有各的 key 时 A 被吊销切到 B 完全能救。分类是 fallback 的第一步，不是重试。
  3. 再说成本：切换意味着同一段 prompt 你付了两次钱，三家链路最坏是三倍成本。这条一定要主动说出来，它区分了「写过」和「上过线」。
  4. 然后是延迟：串行 fallback 的总耗时是各家超时值的累加。如果每家给 15 秒、三家串下来用户要等 45 秒，那还不如早点失败。所以超时值必须按 provider 分别设，且要设总预算上限。
  5. 最后是雪崩，这是最容易被追问的点：主 provider 限流时你把全部流量瞬间压到备用上，很可能把备用也压垮。所以要加熔断——连续失败 N 次就暂时摘掉该 provider，过一段时间放少量流量试探。
  6. 可以预期的追问：怎么知道该摘多久？答案是指数退避 + 半开状态试探，和数据库连接池的熔断是同一套思路。

  Key points

  • Classify before retrying, judging by whether another provider could plausibly succeed rather than the leading digit: 400/403 must not fail over; 408/429/5xx should; so should 402 (out of credit) and 404 (model retired); 401 depends on whether the providers share one key
  • Cost: every fallback re-pays for the same prompt, so worst-case cost scales with chain length
  • Latency: serial fallback sums the timeouts, so set per-provider timeouts plus an overall budget
  • Thundering herd: shifting full traffic to the backup can topple it too — use circuit breaking with exponential backoff and half-open probes

  答题要点

  • 先分类再重试，判据是「换一家有没有可能变好」而不是状态码首位：400/403 不该切，408/429/5xx 该切，402 余额不足和 404 模型下线同样该切，401 取决于三家是否共用同一把凭证
  • 成本：每次 fallback 都要重付一遍 prompt 的钱，链路越长最坏成本越高
  • 延迟：串行 fallback 的耗时是各超时值累加，必须按 provider 分设超时并设总预算
  • 雪崩防护：主 provider 故障时全量流量压向备用会把备用也压垮，需要熔断 + 指数退避 + 半开试探

D7 Packaging It as a Service: Fastify + SSE + Docker (dg P07); Week One Retrospective

• For a long-lived SSE service in production, what problems do heartbeats, disconnect handling and graceful shutdown each solve?一个 SSE 长连接服务上线，心跳、连接断开处理和优雅退出分别在解决什么问题？

  Common in ChinaCommon overseasDeep dive#sse#reliability#deployment

  How to reason about it · think before answering

  1. The discriminator is that the three have completely different failure symptoms. Someone who can describe each symptom has shipped one; 'they all improve stability' is a non-answer.
  2. Heartbeats prevent middleboxes from killing you. Load balancers and gateways commonly close idle connections after 60 to 120 seconds, and agents are full of silent gaps while the model reasons, calls a tool or waits on a slow API. The symptom is a stream that dies halfway for no visible reason and never reproduces against a local server. Implement it as an SSE comment line, which clients silently ignore, so no client change is needed.
  3. Disconnect handling is about money. When a user closes the tab the server does not stop on its own: the model keeps generating and tokens keep billing with nobody receiving. It is the most expensive oversight in streaming services, and staging never reveals it because nobody closes tabs mid-run. Watch for the response closing, distinguish a premature close from a normal finish, and abort the upstream request.
  4. One detail must be right or it exposes you immediately: in Node listen on the response object's close, not the request's. The request emits close once its body has been read, so using it as a disconnect signal misfires on every normal request and you see streams stopping after one or two chunks.
  5. Graceful shutdown is about deploys cutting live requests. On SIGTERM the process should stop accepting new connections, give in-flight streams a short window, then exit; otherwise users watch a reply stop mid-sentence. This assumes the signal actually reaches the process — if the container's PID 1 is a package manager, SIGTERM never arrives and the runtime kills you on timeout.
  6. Expect: how long is the window? Shorter than the orchestrator's termination grace period (10s by default in Docker, 30s in Kubernetes), or you get SIGKILLed anyway; and refuse new connections immediately so the load balancer drains traffic away.

  分析过程 · 先想清楚再作答

  1. 这题的区分度在于三件事各自的失败现象完全不同，能分别说出现象的人一定真上过线。答成「都是为了稳定性」等于没答。
  2. 心跳解决的是「被中间设施误杀」。负载均衡和网关普遍有空闲超时，常见 60 到 120 秒，一段时间没有字节流动就关连接；而 Agent 天生有大量静默期——模型在思考、在调工具、在等慢接口。现象是连接莫名其妙断在一半，且本地直连时完全复现不了。实现上用 SSE 的注释行（冒号开头）做心跳，客户端会安静忽略，不用改客户端代码。
  3. 连接断开处理解决的是「花钱」。用户关掉页面之后服务端不会自动停，模型继续生成、token 继续计费，只是没人接收。这是流式服务里最贵的疏忽，而且测试环境暴露不出来，因为没人会中途关页面。做法是监听响应对象的关闭事件，判定是被掐断而不是正常收尾，就把上游请求一起中止。
  4. 这里有个必须说对的细节，说错会当场暴露没写过：Node 里要监听的是响应对象的 close，不是 request 的——request 的 close 在请求体读完时就触发，拿它当断线信号会把每一条正常请求都误判成客户端跑了，现象是每次只推出一两个片段就停。
  5. 优雅退出解决的是「发布时切断在途请求」。容器收到 SIGTERM 后应当先停止接受新连接，给在途的流一点收尾时间再退出，否则用户看到的是回复说了一半突然没了。前提是信号真的能传到进程——CMD 写成包管理器的话 PID 1 不是 node，SIGTERM 传不到，只能等超时被强杀。
  6. 可以预期的追问：收尾时间给多久？答案是要小于编排系统的终止宽限期（Docker 默认十秒、K8s 默认三十秒），超过就会被 SIGKILL，等于白设计；同时新连接要立刻拒绝，让负载均衡把流量挪走。

  Key points

  • Heartbeats defeat idle timeouts in middleboxes, since agent silence often exceeds a gateway's 60 to 120 seconds; SSE comment lines do it transparently
  • Disconnect handling stops waste: after a user closes the tab, an unaware server keeps burning tokens, and staging never shows it
  • In Node listen on the response's close, not the request's — the latter fires when the body is read and misclassifies normal requests as disconnects
  • Graceful shutdown stops deploys from cutting live streams: on SIGTERM refuse new connections and drain, within the orchestrator's grace period
  • It only works if the signal reaches the process, so PID 1 must be node itself rather than a package manager

  答题要点

  • 心跳防的是中间设施的空闲超时，Agent 的静默期常常超过网关的 60 到 120 秒，用 SSE 注释行实现，客户端无感
  • 断开处理防的是浪费：用户关页面后服务端不停就是纯烧 token，测试环境暴露不出来
  • Node 里要监听响应对象的 close 而不是 request 的——后者在请求体读完时就触发，会把正常请求误判成断线
  • 优雅退出防的是发布切断在途流：SIGTERM 后先停收新连接、给在途流收尾时间，收尾窗口要小于编排系统的终止宽限期
  • 前提是信号能传到进程：容器的 PID 1 必须是 node 本身，不能是包管理器

D8 Why Split Gateway and Worker; Postgres Table Design (sessions/runs/messages) + Drizzle

• With at-least-once delivery, how do you guarantee a redelivered message does not create two runs?消息总线是至少一次投递，同一条消息被重复投递时，怎么保证不会产生两条 run？

  Common in ChinaCommon overseasDeep dive#idempotency#database#reliability

  How to reason about it · think before answering

  1. This question is about which layer idempotency lives in. Anyone who answers 'check whether it exists, then insert' has usually just failed it — that is exactly the answer being screened out.
  2. State the premise: duplicates are not accidents. The bus is at-least-once, clients retry on timeout, users double-click. The same message arriving twice is certain, so the goal is not to prevent duplicates but to make duplicates produce the same result.
  3. Then derive the key: idempotency needs a key derived from request content. A random UUID differs every time and buys nothing; hash the session id, the client message id and the message body together, falling back to content plus a coarse time bucket when the client has no id.
  4. Land it in storage: put a unique constraint on that column in the runs table, write the insert as on-conflict-do-nothing, and when it returns zero rows read back the existing run and return the same run id. Two requests, one run, one id.
  5. Explain why check-then-insert fails, which is the whole point: two gateway instances can query, both see nothing, and both insert. The window between the two statements cannot be closed in application code, it is too narrow to reproduce under load tests, and it leaks a few bad rows every day in production. The database's unique constraint has to be the final arbiter; the application-level check only saves a wasted insert.
  6. Expect the follow-up: what about duplicate execution on the consumer side? The unique constraint gives you one run, but a worker can still receive it twice, so status changes need conditional updates (move to running only if the current status is pending) plus an explicit transition whitelist that blocks a finished run from being pushed back to running and overwriting a reply the user already saw.

  分析过程 · 先想清楚再作答

  1. 这题在考幂等的落点在哪一层。凡是答「在代码里先查一下有没有，没有再插入」的，基本当场结束——因为那正是这题想筛掉的答案。
  2. 先把前提摊开：重复不是意外。总线是至少一次语义、客户端会超时重发、用户会手抖双击，同一句话到达两次是必然事件。所以设计目标不是「避免重复到达」，而是「重复到达时结果相同」。
  3. 然后给推导：幂等需要一个由请求内容决定的键。随机 UUID 每次都不同，等于没有幂等；正确取法是把会话 id、客户端消息 id、消息内容拼起来做哈希，客户端没有消息 id 时退用内容加一个粗粒度时间窗。
  4. 结论落在存储层：在 runs 表的这一列上加唯一约束，插入写成「冲突就什么都不做」，返回零行时回查那条已有的 run，把同一个 runId 返回给用户。两次请求、一条 run、一个 runId。
  5. 解释为什么「先查后插」不行，这是本题的分水岭：两个 Gateway 实例可以同时查、同时发现没有、同时插入，这两步之间有一个应用层拦不住的时间窗；它窄到压测复现不出来，上线后每天漏几条。**幂等的最终裁判必须是数据库的唯一约束**，应用层的判断只是为了少一次插入尝试。
  6. 可以预期的追问：那消费侧的重复执行呢？答：唯一约束保证了只有一条 run，但 Worker 可能重复拿到同一条 run，所以状态迁移也要带条件更新（只有当前状态是 pending 时才能改成 running），并且用一个显式的迁移白名单挡住「已完成的 run 被推回运行中」这种会覆盖用户已收到回复的情况。

  Key points

  • Redelivery is certain, so the goal is identical outcomes on duplicates, not preventing duplicates
  • The idempotency key must be derived from request content — session id plus client message id plus body, hashed; a random UUID buys nothing
  • Put a unique constraint on that column, insert with on-conflict-do-nothing, and read back the existing run when zero rows return
  • Check-then-insert races under concurrency; the window between the statements cannot be closed in application code, so the unique constraint must be the final arbiter
  • On the consumer side add conditional status updates and a transition whitelist so a finished run is never re-run or overwritten

  答题要点

  • 重复投递是必然事件，设计目标是「重复到达时结果相同」，不是「避免重复」
  • 幂等键必须由请求内容决定：会话 id 加客户端消息 id 加内容做哈希，随机 UUID 等于没有幂等
  • 在 runs 的幂等键列上建唯一约束，插入用「冲突就什么都不做」，零行时回查已有 run 返回同一个 runId
  • 先查后插在并发下必然出双份，两条语句之间的时间窗应用层拦不住，幂等的最终裁判是数据库唯一约束
  • 消费侧还要用条件更新加状态迁移白名单，避免同一条 run 被重复执行或把已完成的回复覆盖掉

D9 A Redis Streams Message Bus: XADD/XREADGROUP/XACK/XAUTOCLAIM, Consumer Groups, Poison Messages

• What is at-least-once delivery, and given that messages get redelivered, how do you actually make the business side idempotent?什么是 at-least-once？既然消息会被重复投递，业务上到底要怎么保证幂等？

  Common in ChinaCommon overseasDeep dive#message-bus#idempotency#reliability

  How to reason about it · think before answering

  1. This is the question that gets probed hardest. Most candidates say 'at-least-once, so make the business idempotent' and stop — but the follow-up is exactly what matters: which line of code enforces it.
  2. Explain why the duplicate cannot be removed: committing the business write and acking are two writes to two systems (say Postgres and Redis), so there is always a crash window between finishing the work and XACK. The window can shrink but not disappear, which is why exactly-once is not something the bus gives you.
  3. That yields a sentence worth saying out loud: exactly-once is an effect produced by consumer-side idempotency, not a capability provided by the broker. Kafka transactions achieve it inside a read-Kafka-write-Kafka loop, but the moment the sink is a database or third-party API you are back to at-least-once.
  4. Now name the concrete guards and what each one blocks. First, a unique constraint on runs.idempotency_key with insert ... on conflict do nothing, which blocks duplicate submissions: on conflict the gateway returns the existing run id and never publishes a second bus message. Second, unique(run_id, seq) on the messages table, also on conflict do nothing, which blocks duplicate execution: even if two consumers finish the same run simultaneously the user sees one reply. A cheap short-circuit can sit in between — read the run first and just re-ack if it is already done — but that saves money; correctness comes from the two constraints.
  5. Then the step people get wrong: deriving the key. It must be reproducible from the same intent. Generating a fresh uuid on every retry is the classic mistake, because every retry becomes a new intent and the constraint never fires. The client should mint the key once and reuse it across retries; a server-side fallback can hash session id plus message body plus a second-resolution timestamp.
  6. Expect: what about irreversible side effects such as issuing a refund? Push the idempotency key into the external call (most payment gateways accept an idempotency key header), and record an 'initiated' row locally before calling so the same key deduplicates. For APIs with no such support, fall back to a local state machine plus reconciliation, and say plainly that you would move such operations off the automatic retry path.

  分析过程 · 先想清楚再作答

  1. 这题是本章最容易被追到底的一道。绝大多数人能说出「至少一次，所以业务要幂等」，然后就没有下文了——面试官等的恰恰是下文：幂等具体落在哪一行代码上。答不出具体落点，前半句就是背的。
  2. 先解释为什么消费不掉这个重复：写业务和销号是两个系统的两次写（比如 Postgres 加 Redis），处理完成到 XACK 之间必然存在一个可以崩溃的窗口，崩在那里消息就会被重投。这个窗口只能变小，不能消失，所以 exactly-once 不是总线给你的语义。
  3. 由此得到一句可以直接说出口的结论：exactly-once 是消费端幂等做出来的**效果**，不是中间件提供的**能力**。Kafka 的事务能在「读 Kafka 写 Kafka」的闭环里做到，一旦下游是数据库或第三方 API 就又退回至少一次。
  4. 然后给具体落点，两道闸门要分清各自挡什么：第一道是 runs 表 idempotency_key 上的唯一约束，配 insert on conflict do nothing，挡的是**客户端重复提交**——冲突时接入层直接返回已有的 runId，连总线都不投第二遍；第二道是 messages 表的 unique(run_id, seq)，同样 on conflict do nothing，挡的是**同一条总线消息被执行两遍**，就算两个消费者真的同时跑完，用户也只会看到一条回复。中间还可以加一道便宜的短路：捞到消息先看 run 是不是已经 done，是就直接补一个 XACK 走人——但那是省钱的优化，正确性靠的是那两个唯一约束。
  5. 接着讲最容易做错的一步：幂等键怎么取。它必须能从「同一个意图」稳定推出来。客户端每次重试都新生成一个 uuid 是最常见的错法，那每次都是新意图，唯一约束一次都命中不了，闸门形同虚设。正确做法是客户端生成一次、重试复用同一个值，服务端兜底可以用「会话 id 加消息内容哈希加秒级时间戳」。
  6. 可以预期的追问：不可逆的副作用怎么办，比如发一次退款？答：把外部调用也变成带幂等键的（大多数支付网关都支持 idempotency key 头），并且先在本地库里落一条「已发起」记录再调用，用同一个键去重；实在不支持的接口就只能靠本地状态机加人工对账，这时要主动说出「这类操作我会把它挪出重试路径」。

  Key points

  • At-least-once means a message is processed one or more times, because the business commit and the XACK are two writes to two systems with an unavoidable crash window
  • Exactly-once is an effect of consumer-side idempotency, not a broker feature; any database or third-party sink puts you back at at-least-once
  • Guard one: a unique constraint on runs.idempotency_key with on conflict do nothing blocks duplicate submissions and skips publishing a second bus message
  • Guard two: unique(run_id, seq) on messages with on conflict do nothing blocks duplicate execution, so the user sees exactly one reply
  • The idempotency key must be derivable from the same intent and reused across retries; minting a new uuid per retry defeats the whole mechanism
  • For irreversible side effects, pass the idempotency key through to the external API and record an initiated row locally before calling

  答题要点

  • at-least-once：消息至少被处理一次、可能多次，因为业务提交和 XACK 是两个系统的两次写，中间的崩溃窗口消不掉
  • exactly-once 是消费端幂等做出来的效果，不是中间件的能力；下游只要是数据库或第三方 API 就退回至少一次
  • 闸门一：runs.idempotency_key 唯一约束 + on conflict do nothing，挡客户端重复提交，冲突时不再投递总线消息
  • 闸门二：messages 表 unique(run_id, seq) + on conflict do nothing，挡同一条消息被执行两遍，用户只会看到一条回复
  • 幂等键必须从同一个意图稳定推导，客户端重试要复用同一个值；每次重试新生成 uuid 等于没有幂等
  • 不可逆副作用要把幂等键透传给外部接口，并先落一条本地记录再调用
• What do you do with a message that keeps failing? Design a poison-message isolation mechanism.一条消息反复处理失败怎么办？请设计一个毒消息隔离机制。

  Common in ChinaCommon overseasIntermediate#message-bus#error-handling#reliability

  How to reason about it · think before answering

  1. This question probes whether you have ever watched one bad message stall an entire stream. The test is simple: does your answer contain a concrete threshold and a concrete place where isolation happens? If not, you are talking theory.
  2. Describe the failure mode first: under at-least-once you do not ack on failure, so the message stays pending and gets redelivered. A message that fails for everyone therefore loops forever — delivered, failed, idle timeout, claimed, failed — never recovering while continuously consuming worker capacity.
  3. Then give the mechanism, three actions and all of them required. One, use the delivery count the pending list already tracks rather than building a counter table. Two, past the threshold (three deliveries in this course) move the message to a dead-letter stream carrying the original id, delivery count and failure reason. Three, XACK the original stream and mark the run failed with the error recorded. Moving without acking leaves it pending for another takeover; acking without moving makes both the message and its reason disappear, leaving the user stuck on 'thinking'.
  4. Justify the threshold: one delivery kills messages that a single network blip would have let through; ten wastes ten executions of money and time on a message that can never succeed. Three deliveries, spaced by the idle threshold, survives almost all transient faults.
  5. Volunteer a limitation: Redis Streams has no native exponential backoff — redelivery timing is governed by the idle threshold. Backoff requires republishing the message with a next-eligible timestamp, which means building a delay queue yourself. Naming this shows you know where Streams ends.
  6. Expect: is creating the dead-letter stream the end of it? No. Its depth must be alerted on, since going from zero to non-zero usually means a class of input your code cannot handle — a real bug, not bad luck. Keep a replay path too: republish the stored fields back to the original stream, and because the idempotency key is preserved, replay cannot cause duplicate execution. Teams that build a dead-letter stream and never open it have simply muted their failures.

  分析过程 · 先想清楚再作答

  1. 这题在考你有没有踩过「一条坏消息拖垮整条流」。判断标准很简单：你的回答里有没有出现一个具体的阈值和一个具体的落地位置，没有就是在讲概念。
  2. 先把故障模式说清楚：按 at-least-once 的规矩，失败就不 ack、留在 pending 等重投，于是一条无论谁来都会失败的消息进入死循环——投递、失败、闲置超时、被接手、再失败。它自己永远好不了，还持续占用消费者的处理能力。
  3. 然后给机制，三个动作缺一不可：一、判定依据用 pending 清单自己记的投递次数，不要另建计数表；二、超过阈值（本课固定 3 次）就把消息搬到一条死信流，字段里带上原始消息 id、投递次数和失败原因；三、对原流 XACK，同时把这次执行标成失败并写入错误原因。只搬不 ack，它还躺在 pending 里等着被接手；只 ack 不搬，消息和失败原因一起消失，用户永远停在「正在思考」。
  4. 阈值的取值要给出权衡：定 1 会让一次网络抖动就把本来能成功的消息判死；定 10 会在一条必死的消息上浪费十次执行的钱和时间。3 次配合每次之间的空闲阈值，足够熬过绝大多数瞬时故障。
  5. 还要主动说出一个缺口：Redis Streams 没有原生的指数退避，重投时机由空闲阈值决定。想要退避就得自己把消息重新投递并带上「下次可执行时间」，那已经是在实现延迟队列了——这一条能体现你知道 Streams 的边界在哪。
  6. 可以预期的追问：死信流建完就完了吗？答：不。死信条数必须接进告警，它从 0 变成非 0 通常意味着有一类输入你的代码处理不了，是真 bug 而不是运气差；还要留一个重放入口——把死信里的字段原样投回原流即可，因为幂等键还在，重放不会产生重复执行。见过团队把死信建起来半年没打开过，那等于把故障静音了。

  Key points

  • Failure mode: under at-least-once you do not ack on failure, so an always-failing message is redelivered forever and keeps consuming worker capacity
  • Use the delivery count already tracked in the pending list rather than a separate counter table
  • Fix the threshold at three deliveries: one kills transient failures, ten wastes ten executions on a message that can never succeed
  • Isolation needs all three actions: move to a dead-letter stream with original id, delivery count and reason; XACK the original stream; mark the run failed with the error stored
  • Redis Streams has no native exponential backoff — redelivery timing follows the idle threshold, so backoff means implementing delayed republishing yourself
  • Alert on dead-letter depth and keep a replay path; the idempotency key survives, so replay cannot duplicate execution

  答题要点

  • 故障模式：at-least-once 下失败不 ack，一条永远失败的消息会无限重投并持续占用消费者
  • 判定依据用 pending 清单里记的投递次数，不需要另建计数表
  • 阈值固定 3 次：定 1 会误杀瞬时故障，定 10 会在必死消息上浪费十次执行成本
  • 隔离动作三件缺一不可：搬到死信流（带原始 id、投递次数、失败原因）、对原流 XACK、把这次执行标成失败并写入原因
  • Redis Streams 没有原生指数退避，重投时机由空闲阈值决定，要退避得自己实现延迟投递
  • 死信流要接告警并留重放入口；幂等键还在，重放不会导致重复执行

D10 Sharding and Leases: Hashing userId → shard, SET NX + TTL + Lua Renewal, Per-User Ordering, Handoff

• What happens when two workers both believe they hold the same shard lease (split brain), and how do you mitigate it?两个 worker 同时认为自己持有同一个 shard 的租约（脑裂）会造成什么后果，怎么规避？

  Common in ChinaCommon overseasDeep dive#split-brain#fencing-token#reliability

  How to reason about it · think before answering

  1. The scoring criterion here is explicit: does your answer contain the sentence 'a Redis lease alone cannot give absolute mutual exclusion'. Anyone who says SET NX plus a TTL makes it safe gets probed until they run out of answers.
  2. Start with how split brain arises, and use the common case: not a crash, but a holder that merely froze for five seconds — a full GC, a noisy neighbour saturating the host CPU, cgroup throttling. It wakes up still believing it holds shard 68, keeps processing the in-flight message and keeps writing, while the lease expired and was taken. Add the second layer: Redis replication is asynchronous, so a failover can lose the last few milliseconds of writes and let two workers both win SET NX.
  3. Then the consequences, expressed in business terms rather than 'inconsistent data': two messages from one user processed concurrently means out-of-order replies, a corrupted context window, and unique(run_id, seq) violations that silently drop a message. Worst is reordered or duplicated side effects — swap 'cancel the order' with 'move the delivery date' and you cancel an order the user wanted to keep.
  4. The key shift: since you cannot rule out that timeline on the Redis side, the goal is not to prevent split brain but to make the second writer's writes fail — push conflict detection and rejection down to the layer that actually causes side effects.
  5. Give three mitigations by value. First, a self-kill rule in the worker: after two consecutive renewal failures, or when the last success is older than two thirds of the TTL, stop processing and clear the held set — cheapest, and it bounds the 'I think I still hold it' window to two renewal periods. Second, fencing tokens: take a monotonically increasing number (Redis INCR) when acquiring, store it in the lease value, attach it to every side-effecting operation, and have the downstream accept only numbers not lower than the highest it has seen — in a database that is one conditional update. The revived predecessor carries a stale number and is rejected. Third, re-validate the lease immediately before each write inside the same script or transaction, which shrinks the window without closing it.
  6. Expect the follow-up: where does fencing break down? It needs downstream cooperation. Databases do conditional updates, but a third-party endpoint (SMS, payments) will not compare your token, so you fall back to idempotency keys that make duplicate execution harmless rather than impossible. True mutual exclusion means moving to a consensus-backed system such as etcd or ZooKeeper session leases, paying in write latency and operational complexity.

  分析过程 · 先想清楚再作答

  1. 这题的判分点非常明确：答案里有没有出现「单靠 Redis 租约做不到绝对互斥」。说「用了 SET NX 加 TTL 就安全了」的人，会被追问到答不上来。
  2. 先讲脑裂是怎么发生的，而且要举那个最常见的场景——不是进程崩溃，是持有者只卡了 5 秒：一次 full GC、宿主机 CPU 被邻居打满、容器被 cgroup 限流。它醒过来时内存里还写着「我持有 shard 68」，继续处理手上那条消息、继续写库，而 Redis 里的租约早已到期并被别人抢走。再补一层：Redis 主从复制是异步的，切主时可能丢掉最后几毫秒的写入，于是两个 worker 都能 SET NX 成功。
  3. 然后讲后果，而且要落到业务上而不是停在「数据不一致」：同一个用户的两条消息被两个进程并发处理，回复乱序、上下文错乱、messages 表的 unique(run_id, seq) 撞约束导致落库失败；最严重的是有副作用的工具被重排或重复执行——「取消订单」和「改配送日期」顺序反了，结果是取消了一个用户本来想留下的订单。
  4. 关键的认知转折：既然无法在 Redis 一侧排除这条时间线，正确的思路就不是「让脑裂不发生」，而是「让第二个人的写入落不了地」——把冲突的检测与拒绝推到真正产生副作用的那一层。
  5. 三条手段按性价比给出。一是 worker 自己的自杀规则：连续两次续约失败、或距上次成功续约超过 TTL 的三分之二，立刻停止处理并清空持有集合——最便宜，把「我以为我还持有」的窗口从无限压到两个续约周期。二是 fencing token：抢租约时从一个单调递增计数器取号（Redis 的 INCR）写进租约值，之后所有有副作用的操作都带上它，下游只接受不比见过的最大号小的写入，落到数据库上就是一句条件更新；醒过来的前任拿的是旧号，写入直接被拒。三是每次写之前重新校验租约，并把校验与写入放进同一段脚本或同一个事务——这只缩小窗口，不消除。
  6. 可预期的追问：fencing 的局限在哪？答「它需要下游配合」。数据库能做条件更新所以好使，但下游是第三方接口（发短信、扣款）时你没法让对方帮你比号，这时只能退回幂等键，把重复执行变成无害，而不是让它不发生。真要绝对互斥就得换到有共识协议的系统（etcd、ZooKeeper 的会话租约），代价是写入延迟和运维复杂度。

  Key points

  • A Redis lease alone cannot guarantee mutual exclusion: a frozen holder that revives, and asynchronous replication losing writes on failover, are both unavoidable
  • State consequences in business terms: out-of-order replies, corrupted context, unique-constraint violations dropping messages, and reordered or duplicated side effects
  • The goal is to make the second writer's writes fail — push conflict detection to the side-effecting layer instead of hoping split brain never happens
  • Three mitigations: a worker self-kill rule on repeated renewal failure, fencing tokens enforced as conditional updates, and re-validating the lease immediately before writing
  • Fencing needs downstream cooperation; against third-party endpoints fall back to idempotency keys, and true mutual exclusion means a consensus system like etcd or ZooKeeper

  答题要点

  • 单靠 Redis 租约做不到绝对互斥：持有者被冻结再醒来、以及主从异步复制丢写，这两条时间线排除不掉
  • 后果要落到业务：同用户回复乱序、上下文错乱、唯一约束冲突丢消息，最严重是有副作用的工具被重排或重复执行
  • 思路是「让第二个人的写入落不了地」，把冲突检测推到产生副作用的那一层，而不是指望脑裂不发生
  • 三条手段：worker 自杀规则（续约连续失败就放手）、fencing token（写入时带单调号做条件更新）、写前重新校验租约
  • fencing 需要下游配合；下游是第三方接口时只能退回幂等键，要绝对互斥就得换 etcd / ZooKeeper 这类有共识协议的系统

D14 Deployment and Operations: Multi-Worker Compose, Heartbeats, Health Checks, Graceful Shutdown, Dev/Prod Isolation; Week Two Retrospective

• What is graceful shutdown, and why is killing a process outright risky? Walk through the steps.什么是优雅停机？为什么直接 kill 进程有风险？请说出具体步骤。

  Common in ChinaCommon overseasIntermediate#deployment#reliability#operations

  How to reason about it · think before answering

  1. This question tests whether you have actually shipped a release. Reciting finish in-flight work before exiting is just the definition; the interviewer wants the cost, the steps, and the ordering.
  2. Make the cost concrete. Deploys, scale-downs, host maintenance and spot reclamation all send SIGTERM, wait a grace period, then SIGKILL. SIGKILL cannot be trapped, and landing it on a worker mid-agent-loop means: the run is stuck in running forever while the user watches a spinner; you already paid for the model call but never persisted the reply; the unacked message waits for the idle threshold before anyone claims it. One deploy cuts off dozens of conversations — that is the everyday cost.
  3. Then give three steps and stress that the order is fixed. One, stop accepting work: flip a flag so the consume loop stops reading from the stream (messages already fetched but not started stay in pending for someone else, which is faster than forcing a whole batch through). Two, wait for the in-flight execution, but with a ceiling. Three, proactively release leases, deregister from the heartbeat dashboard, and exit.
  4. The ceiling in step two earns points: a hung model call means you wait forever, and the grace period will SIGKILL you anyway. Better to concede and exit — the unacked message is still pending and someone will redo it. This course uses 20 seconds, derived from the upper bound of a normal execution plus margin.
  5. Step three also earns points: leases normally change hands via TTL expiry, but that path exists for sudden death. On a planned shutdown you know you are leaving, so releasing proactively lets the successor take over on its next scan instead of waiting out a full TTL. The release must be conditional — delete only the badge that still bears your name, or you will tear down the badge of whoever just claimed it after your lease expired.
  6. Finish with two companions; miss either and the rest is wasted. The configured grace period must exceed the wait ceiling in code (code waits 20s while compose defaults to 10s, so SIGKILL lands at second 10 and your three steps only half-run). And the signal must actually reach your process (if the entrypoint is a package manager, PID 1 is the package manager, SIGTERM may never arrive, and your shutdown code never runs once).

  分析过程 · 先想清楚再作答

  1. 这题考的是「你有没有真的发过版」。答「等任务跑完再退出」只是定义，面试官要的是代价、步骤和顺序。
  2. 先把代价说具体。发版、缩容、机器维护、抢占式实例回收都会先发 SIGTERM、等宽限期、超时 SIGKILL。SIGKILL 拦不住，落到正在跑 Agent 循环的 Worker 身上：这次的 run 永远停在 running，用户界面一直转圈；模型调用的钱已经付了，回复却没落库；没确认的消息要等空闲阈值到了才被别人接手，用户白等一轮。一次发版掐断几十次对话，这就是日常代价。
  3. 然后给三步，强调顺序不能变：第一步拒新——把开关拨过去，消费循环下一轮不再从流里取消息（已经读到手上还没开始的那几条，留在 pending 里由别人接手，比硬扛完一整批更快）；第二步等手头这次执行跑完，但要有上限；第三步主动交还租约、从心跳面板注销，然后退出。
  4. 第二步的上限是加分点：一次卡死的模型调用会让你永远等不到，而宽限期一到照样 SIGKILL。与其被动挨刀，不如自己认输退出——没确认的消息还在 pending 里，别人会接手重做。本课取 20 秒，取法是「一次正常执行的耗时上限」再留余量。
  5. 第三步也是加分点：租约本来靠 TTL 到期自然易主，但那是为进程猝死准备的。计划内下线你明知道自己要走，主动交还能让接手方下一轮扫描就上岗，而不是白等一个 TTL。交还必须带条件——只删还写着自己名字的那把牌子，否则租约已过期、别人刚抢到时，你就把对方的值班牌撕了。
  6. 最后两件配套的事，漏一件前面全白做：宽限期的配置必须大于代码里的等待上限（代码等 20 秒而 compose 默认只等 10 秒，第 10 秒就 SIGKILL，三步只走到一半）；以及信号得真的传到你的进程（启动命令写成包管理器，PID 1 就是包管理器，SIGTERM 未必传得到，停机代码一次都不会执行）。

  Key points

  • Concrete cost of a hard kill: the run is stuck in running, the user stares at a spinner, the model call is paid for but the reply is unsaved, and the unacked message waits out the idle threshold
  • Three steps in a fixed order: refuse new work, wait for in-flight work with a ceiling, then release leases and deregister before exiting
  • The wait needs a ceiling (20s here): a hung model call never returns and the grace period kills you anyway, so concede — the message is still pending for someone else
  • Releasing leases proactively lets the successor start on its next scan instead of waiting a full TTL; the release must be conditional on still owning it
  • The configured grace period must exceed the in-code wait ceiling, or the three steps only half-run (stop_grace_period / terminationGracePeriodSeconds)
  • Make sure the signal reaches your process: exec the business process directly rather than letting a package manager be PID 1

  答题要点

  • 直接 kill 的具体代价：run 永远停在 running、用户界面一直转圈、模型的钱已付但回复没落库、没确认的消息要等空闲阈值才被接手
  • 三步且顺序不能变：拒绝新任务 → 等手头的跑完（有上限）→ 主动交还租约并注销心跳，然后退出
  • 等待必须有上限（本课 20 秒）：卡死的模型调用会让你永远等不到，宽限期一到照样被 SIGKILL，不如自己认输，消息还在 pending 里
  • 主动交还租约让接手方下一轮就上岗，而不是白等一个 TTL；交还必须条件化，只删还写着自己名字的那把
  • 宽限期配置必须大于代码里的等待上限，否则三步只执行到一半（compose 的 stop_grace_period / K8s 的 terminationGracePeriodSeconds）
  • 信号要真传到进程：用 exec 形式直接起业务进程，别让包管理器当 PID 1
• During a rolling deploy, how do you keep in-flight tasks from being interrupted?滚动发布时，如何避免正在处理的任务被打断？

  Common in ChinaCommon overseasIntermediate#deployment#reliability#operations

  How to reason about it · think before answering

  1. This is the applied version of the previous question, and the difference is that it demands the orchestrator's side too — describing only the in-process steps answers half of it.
  2. The full skeleton is both sides cooperating: the orchestrator first removes traffic (turns readiness red so the load balancer stops sending new requests), then sends SIGTERM, then waits out the grace period; the process uses that window to finish in-flight work, hand back ownership, and exit cleanly. That sentence is the trunk; everything else is detail.
  3. Then distinguish the two kinds of replica, which is where the points are. A gateway has inbound connections, so draining traffic means something for it. A worker has no inbound connections at all — it pulls work from the bus, so draining for it means stop fetching new messages, which is step one of graceful shutdown. The same word is two different mechanisms on the two replica types, and saying so shows you understand pull versus push.
  4. Next, batching and ordering: replace only a subset at a time (manual batches in compose, maxUnavailable / maxSurge in Kubernetes) so enough replicas are always alive to absorb traffic. This is where the version field in the heartbeat payload pays off — you can see how many old and new replicas remain instead of deploying blind.
  5. Also mention state compatibility: during a rolling deploy old and new code run simultaneously, so schema migrations must be backward compatible (add a nullable column, dual-write, drop the old column last) and message formats cannot change in one shot. Many candidates miss this layer — however gracefully processes stop, two versions that cannot read the same data will still cause an incident.
  6. Expect: what if a single execution legitimately takes five minutes and the grace period cannot wait that long? The answer is not to stretch the grace period to five minutes but to make the task interruptible and resumable — break long work into steps that checkpoint progress (the run state machine from D11 plus at-least-once with idempotency from D9 give you exactly this), so the next replica continues the interrupted step.

  分析过程 · 先想清楚再作答

  1. 这题是上一题的应用题，区别在于它要求你把编排系统那一侧也讲进来——只讲进程内的三步只答了一半。
  2. 完整骨架是两侧配合：编排系统先摘流量（把就绪探针转红，让负载均衡不再把新请求打过来）、再发 SIGTERM、然后等宽限期；进程在这段时间里把手头的活做完、交还所有权、干净退出。这一句话就是答案的主干，剩下都是细节。
  3. 然后区分两类副本，这是拿分点。Gateway 有入站连接，摘流量对它有意义；Worker 没有任何入站连接，它是自己去总线取活的，所谓「摘流量」对它就是「自己不再取新消息」——也就是停机三步的第一步。**同一个词在两类副本上是两种机制**，能说清这一点说明你理解拉与推的差别。
  4. 接着讲批次与顺序：一次只换一部分副本（compose 里手动分批，K8s 里靠 maxUnavailable / maxSurge），保证任何时刻都有足够的存活副本接得住流量。心跳面板上的版本号字段这时派上用场——你能看到新旧两批各剩几个，而不是盲发。
  5. 还要提一句状态兼容：滚动发布期间新旧代码同时在线，所以数据库迁移必须向后兼容（先加可空列、再双写、最后才删旧列），消息格式也不能一次性改。这是很多人漏掉的一层——进程停得再优雅，新旧版本读不了同一份数据照样出事故。
  6. 可以预期的追问：如果一次执行本来就要跑 5 分钟，宽限期不可能等那么久怎么办？答案不是把宽限期拉到 5 分钟，而是让任务可中断可重入——把长任务切成可保存进度的小步（D11 的 run 状态机和 D9 的 at-least-once 加幂等正好提供了这个基础），被打断的那一步由下一个副本接着做。

  Key points

  • The full skeleton is both sides: orchestrator drains traffic, sends SIGTERM, waits the grace period; the process finishes in-flight work, hands back ownership, exits cleanly
  • Draining means two different things for gateways and workers: readiness turning red versus the worker itself stopping its fetch from the bus
  • Replace in batches (maxUnavailable / maxSurge or manual) so enough replicas stay alive; the version field in heartbeats shows how many old and new remain
  • Old and new code run concurrently, so migrations must be backward compatible (nullable column, dual-write, drop last) and message formats cannot change in one step
  • Long tasks are not solved by a longer grace period but by being interruptible and resumable — checkpointed steps that the next replica can continue

  答题要点

  • 完整骨架是两侧配合：编排系统先摘流量、再发 SIGTERM、等宽限期；进程在这段时间做完手头的活、交还所有权、干净退出
  • Gateway 和 Worker 的「摘流量」是两种机制：前者靠就绪探针转红让负载均衡停止转发，后者靠自己不再从总线取新消息
  • 分批替换（maxUnavailable / maxSurge 或手动分批），保证任何时刻有足够存活副本；心跳里的版本号让你看到新旧两批各剩几个
  • 新旧代码同时在线，所以数据库迁移必须向后兼容（加可空列 → 双写 → 最后删旧列），消息格式不能一次性改
  • 长任务不该靠拉长宽限期解决，而要做成可中断可重入：切成能保存进度的小步，被打断的那步由下一个副本接着做

D16 Dynamic Routing With a Supervisor: Structured-Output Routing, Override, routingReason

• Why use structured output rather than natural language for routing? What exactly goes wrong with free text?为什么要让模型输出 structured output 而不是自然语言来做路由？自然语言到底差在哪？

  Common in ChinaCommon overseasIntermediate#structured-output#routing#reliability

  How to reason about it · think before answering

  1. The trap is answering structured output is cleaner and easier to parse. Those are adjectives, not reasons. The interviewer wants a concrete failure you have actually debugged.
  2. Lead with the sharpest point: free-text routing fails silently. The model replies I think the order desk should look at this — it judged correctly, but it spoke prose, not an id. Your regex misses, you fall through to the default, and the log shows only smalltalk. A correct model with a broken parser looks exactly like a wrong model, so you spend two days tuning a prompt that was never the problem.
  3. Then list three holes and map each to what structured output fixes: wording drifts across versions so regexes never catch up; there is no confidence signal, so you cannot tell certainty from guessing; and an invented route name only explodes at runtime, whereas an enum is a gate that exists before the request is even sent.
  4. Explain the mechanism rather than stopping at zod is nicer: send the schema in the request (response_format with a json_schema), so decoding is constrained by the enum, then validate the response with the same declaration. One declaration used twice means request and validation cannot drift apart.
  5. The counterintuitive point that separates candidates: structured output does not remove the need to validate. Not every gateway or model enforces the schema strictly, and a fallback model may not at all. Your parse function should return something-to-be-validated, not an already-typed decision.
  6. Expect: what if the model does not support json_schema? Fall back to few-shot plus a strict prompt plus your own validation. The real gate was never the model's discipline; it is your parsing layer.

  分析过程 · 先想清楚再作答

  1. 这题最容易答成「结构化更规范、更好解析」——这是形容词，不是理由。面试官想听的是一个具体的失败场景，最好是你真的调过的那种。
  2. 把最锋利的一刀先亮出来：**自然语言路由的失败是静默的**。模型回「我觉得这个可以让查订单的同事看一下」，它其实判对了，但说的是人话不是 id，正则匹配不上就落进兜底，日志里只留下一个 smalltalk。模型是对的、解析是错的，而它和「模型判错了」在日志里长得一模一样。你会去调提示词，调两天才发现问题在那三行正则。
  3. 然后给三个漏洞，一条一条对上结构化输出解决了什么：输出会漂移（今天回「订单查询」明天回「查订单」，正则永远追不上，模型小版本升级你就掉准确率）；没有置信度（自然语言里没有「我有多大把握」这个信息，你没法区分它很确定还是在猜）；拼错或自造的路由名要到运行时才炸（枚举是一道编译期就存在的闸门）。
  4. 接着说清机制，别停在「用 zod 更规范」：把 schema 发进请求（response_format 里的 json_schema），模型的解码过程被枚举约束；回来之后**用同一份声明再校验一遍**。一份声明两用，请求与校验不会漂移。
  5. 关键的反直觉点，答到这里就拉开差距了：**结构化输出不等于不用校验**。不是所有网关、所有模型都严格执行 schema，降级到备用模型时更说不准。所以解析函数的返回类型应该是「一段待校验的东西」，而不是「已经是 RouteDecision」。
  6. 可以预期的追问：那不支持 json_schema 的模型怎么办？答案是退回「few-shot 加严格提示词加自己校验」，闸门仍然在你的枚举校验那一步——真正兜底的从来不是模型的自觉，是你的解析层。

  Key points

  • Free-text routing fails silently: a correct judgement in prose misses your regex and falls through, looking identical to a wrong judgement in the logs
  • Three holes: wording drifts, there is no confidence signal, and invented route names only fail at runtime
  • One declaration used twice: the schema constrains decoding in the request and validates the response, so the two cannot drift
  • An enum is a gate that exists before the call, turning a misspelled route from an incident into a parse failure
  • Structured output does not remove validation — gateways and fallback models may not enforce the schema, so parsing must return an unvalidated value

  答题要点

  • 自然语言路由的失败是静默的：模型判对了但说的是人话，正则匹配不上就落兜底，和判错在日志里完全一样
  • 三个漏洞：措辞会漂移（正则追不上）、没有置信度（分不清确定与猜）、自造的路由名要到运行时才炸
  • 机制是一份声明两用：schema 随请求发出去约束解码，回来后用同一份声明校验，请求与校验不会漂移
  • 枚举是编译期就存在的闸门，把「拼错的路由名」从线上事故降级成一次解析失败
  • 结构化输出不等于不用校验：网关和降级模型未必严格执行 schema，解析函数的返回类型应该是「待校验」而不是「已经是」
• How should the system handle an uncertain or wrong routing decision, and how do you pick the threshold?路由不确定或者路由错误时，系统应该怎么兜底？阈值该怎么定？

  Common in ChinaCommon overseasDeep dive#routing#fallback#reliability

  How to reason about it · think before answering

  1. The hinge is that uncertain and wrong are two different failures. Most candidates answer retry or escalate to a human, collapsing both into one. The discriminator is stating a value judgement before giving a policy.
  2. The claim first: routing to the wrong sub-agent is far worse than failing to route. A failure announces itself and lets you ask a clarifying question. A wrong route does not — the receiving agent has no idea it got the wrong job and will produce a confident, well-formatted, wrong answer that the user will act on. A confident wrong answer costs a hundred times more than I did not catch that.
  3. Then give a concrete policy with real numbers: if the model's confidence is below 0.6, or the route name is not in the allowed list, fall back to the small-talk agent and stamp the reason with a fallback prefix plus a cause code (low confidence, unknown route, invalid shape). The fallback agent's job is to ask for the one missing detail rather than guess — falling back means handing the uncertainty back to the user.
  4. The threshold question is the real test, so do not recite a number: it depends on which error is more expensive. In customer support one extra question costs mild annoyance while a misroute can become a wrong refund promise, so stay conservative. For an internal tool the extra question is the bigger cost, so lower it. Then give a method: sweep thresholds over a golden set, plot misroute rate against clarification rate, and pick the knee.
  5. Name the trap: the confidence number is self-reported and is not a probability. Nine tenths does not mean nine in ten are right. It is a usable ranking signal within one model and one prompt — good as a gate, useless for expected-value math. Real accuracy comes from offline evaluation.
  6. Expect: does falling back just hide the problem? Not if you record cause codes. Group a week of fallbacks by cause and you can see exactly which intent the routing prompt fails to describe. The fallback stops the bleeding; the cause code is what fixes it.

  分析过程 · 先想清楚再作答

  1. 题眼在「不确定」和「错误」是两件事。多数人只答重试或人工接管，那是把两个问题揉成一个。区分度在于你能不能先给出一条价值判断，再给策略。
  2. 先立论：**路由到错的子 Agent，比路由失败糟糕得多**。失败你至少知道自己失败了，可以追问一句；错了，接手的子 Agent 完全不知道自己接错了活，会用笃定的语气给出一个格式完整的错误答案，用户不会怀疑，会照着去操作。一个自信的错误答案比一句「我没听清」贵一百倍。
  3. 再给可执行的策略，数字要具体：模型给的置信度低于 0.6，或者路由名不在合法名单里，一律落到兜底的 smalltalk，并在 routingReason 里打上 fallback 前缀加原因码（低置信度、未知路由、结构非法各一种）。兜底那位的人设是「信息不足先追问一句缺的关键信息，不要猜」——兜底的本质是把不确定性还给用户。
  4. 阈值怎么定这一问是重点，别背数字：**取决于两类错误哪一类更贵**。客服场景里多问一句只是用户小小的不耐烦，派错可能变成一条错误的退款承诺，所以宁可保守取 0.6；内部工具型 Agent 里多问一句反而更烦人，阈值就该放低。再补一句可落地的定法：拿标准样本集扫一遍，画出不同阈值下的误派率与追问率，选拐点。
  5. 必须点破的一个坑：**置信度是模型自己报的，它不是概率**。模型说 0.9 不代表有九成对。它只是同一模型、同一提示词下相对可用的排序信号，只能当闸门用，不能拿去算期望值。真正的准确率要靠离线评估去量。
  6. 可以预期的追问：兜底会不会把问题掩盖掉？答案是不会，前提是你记了原因码——把一周内落进兜底的请求按原因分组，能直接看出分诊提示词缺了哪一类描述。兜底是止血，原因码才是治本的输入。

  Key points

  • Separate the two: a failed route can ask a clarifying question, a wrong route produces a confident wrong answer, and the second is far costlier
  • Policy: confidence below 0.6 or a route outside the allowed list falls back to small talk, stamped with a fallback prefix and a cause code
  • Falling back is not picking someone at random — the fallback agent asks for the missing detail instead of guessing
  • The threshold depends on which error costs more; sweep it over a golden set and pick the knee between misroutes and clarifications
  • Self-reported confidence is not a probability — use it as a gate only, and measure real accuracy offline
  • Record cause codes on every fallback; grouping them shows which intent the routing prompt fails to describe

  答题要点

  • 先分清两件事：路由失败可以追问，路由错误会让子 Agent 自信地给出错误答案，后者贵得多
  • 策略：置信度低于 0.6 或路由名不在名单里，一律落兜底的 smalltalk，并在 routingReason 打上 fallback 前缀加原因码
  • 兜底不是随便找个人接，而是把不确定性还给用户——兜底那位应当追问缺失的关键信息而不是猜
  • 阈值取决于两类错误哪一类更贵：客服场景多问一句便宜、派错很贵，所以保守；定法是拿标准样本集扫阈值找拐点
  • 置信度是模型自报的，不是概率，只能当闸门用；真正的准确率要靠离线评估量
  • 落兜底时记原因码，按原因分组就能看出分诊提示词缺了哪一类描述

D17 Planner-Executor-Critic Plus a Shared Workspace: Workspace State, toolBudget, Parallel Fan-Out, a Review Loop

• Why give each subtask a tool-call budget, and what do you do when it runs out?为什么要给每个子任务设 toolBudget 这样的预算？超了预算之后你会怎么处理？

  Common in ChinaCommon overseasIntermediate#cost-control#reliability#agent-design

  How to reason about it · think before answering

  1. The hinge is the second half. Everyone can say it controls cost; what separates people is what happens when the budget runs out. Answering throw an exception usually means you have never shipped a user-facing agent.
  2. Make the why concrete: a stuck subtask rarely errors — it queries, dislikes the result, and queries again. The model never gets tired; it will spend whatever you allow. A per-conversation cap is the outer gate, a per-subtask budget is the inner one, and the finer grain tells you which piece went out of control instead of only that the conversation was expensive.
  3. Add the design point people miss: the budget must be per subtask, not per execution. With a review loop, retries have to draw on the same budget, or two rejections triple the real allowance and the gate is meaningless.
  4. The conclusion is the exhaustion path: degrade — return what you already have with a flag — rather than throw. Explain why: throwing upgrades this piece is half done into the whole request failed. The user waited several seconds and gets an error page, when in reality only one of three pieces is missing. Two and a half answers plus a clear note beats an error page every time.
  5. Say something about the flag too: it turns degradation into an observable, countable fact instead of a log line. The layer above decides whether to escalate to a human, and monitoring plots a degradation rate — two systems with the same average score but 30 percent versus 3 percent degradation are not the same system.
  6. Expect: how big should the budget be? Derive it from how many tool calls the task normally needs plus margin, not a round number pulled from the air. And pair it with a second dimension — wall-clock or tokens — because one very slow tool call can ruin a request while counting as a single call.

  分析过程 · 先想清楚再作答

  1. 这题的题眼在后半句。前半句几乎人人会答「防止成本失控」，真正拉开差距的是超限之后的动作——答「抛异常」的人基本没做过面向用户的 Agent。
  2. 先把「为什么」说具体。子任务卡住的典型形态不是报错，而是反复查、反复不满意、再查——模型不会喊累，它会把额度花光为止。整轮对话的成本封顶是外层的闸，子任务预算是内层的闸；粒度细到单件事的好处是超支时你能精确指出是哪一件失控了，而不是只看到这次对话贵了。
  3. 再点一个容易被忽略的设计点：预算必须是子任务级的，不是单次执行级的。有评审回路时，被打回重做也得计费，否则打回两次实际额度就翻三倍，这道闸等于没设。
  4. 结论是超限的处理：降级返回已有结果并打上标记，不抛错。理由要说透——抛错等于把「这件事只做了一半」升级成「整个请求失败」，用户等了几秒最后看到一句服务异常，可他其实只是没拿到三件事里的一件。给出两件半的答案并说明哪半件没做成，永远比一个错误页有用。
  5. 降级标记本身也要说：它让降级变成可观测、可统计的事实，而不是日志里的一句话。上层据此决定要不要转人工，监控据此画降级率——两个平均分一样的系统，降级率百分之三十和百分之三完全不是一回事。
  6. 可以预期的追问：预算该设多少？答案是从「这件事正常需要几次工具调用」反推再留一点余量，不是拍脑袋取整数；同时要有第二个维度的闸（挂钟时间或 token 数），因为一次超长的工具调用同样能拖垮请求，而它只算一次。

  Key points

  • A stuck subtask loops rather than errors, and the model will spend whatever you allow; a conversation cap is the outer gate, a subtask budget the inner one that localises the blowup
  • The budget must be per subtask, not per execution, or two review rejections triple the real allowance
  • On exhaustion, degrade and flag rather than throw — throwing upgrades half done into whole request failed and discards what was already retrieved
  • The degradation flag makes the degradation rate a real metric for escalation and evaluation
  • Size the budget from the task's normal tool-call count plus margin, and pair it with a wall-clock or token gate

  答题要点

  • 子任务卡住的典型形态是反复查而不是报错，模型会把额度花光为止；整轮封顶是外层闸，子任务预算是内层闸，细粒度让你能定位到是哪一件失控
  • 预算必须是子任务级而不是单次执行级，否则被评审打回两次实际额度就翻三倍
  • 超限必须降级返回已有结果并标记，不能抛错——抛错把「做了一半」升级成「整个请求失败」，用户连已经查到的部分都拿不到
  • 降级标记让降级率变成可统计指标，上层据此决定转人工，评估据此区分两个平均分相同的系统
  • 预算大小从这件事正常需要几次工具调用反推并留余量，同时配一个时间或 token 维度的闸
• How do you keep a Critic review loop from spinning forever, and what else needs guarding besides a retry cap?Critic 的评审回路怎么防止陷入死循环？除了次数上限还有什么要防的？

  Common in ChinaCommon overseasIntermediate#reflection#loop-guard#reliability

  How to reason about it · think before answering

  1. Asking what else besides a cap tells you the interviewer already expects the cap. What is really being tested is whether you have run this loop for real. The cap earns baseline credit; naming the other two failure modes is what passes.
  2. Failure one is infinite rejection: every revision draws a new complaint and nothing converges. The cap exists to guarantee termination, not to save money. Two rejections and three executions is a reasonable default, because an effective fix usually lands on the second attempt — if the third still fails, the rubric itself is the problem.
  3. Failure two is a rejection with no actionable content. If the reviewer only says not good enough, the executor has nothing to act on and resubmits the same thing, burning the full cap. Rejections must carry a specific reason, and that reason must be written back into the subtask goal. Missing the refund conclusion, please add it is actionable; poor quality is not.
  4. Failure three is the dangerous one people rarely mention: when reviewer and executor share a model and a prompt, the reviewer tends to approve its own output. A single model has consistent preferences about what a good answer looks like, so pass rates go implausibly high and the review step becomes theatre. Mitigations by value: give the reviewer an objective, checkable rubric; use a different model even a cheaper one; score item by item rather than emitting one verdict.
  5. Also distinguish the framework's safety net from your business cap: orchestration frameworks usually ship a recursion limit, but that is a last-resort fuse — it is graph-wide so you cannot tell which loop ran away, and it throws, which means you lose the partial results you were supposed to degrade to.
  6. Expect: what do you return once the cap is used up? Return what you have, flag it as degraded, and carry the last review comment out with it so the layer above can decide whether to escalate. The loop's value is not only fixing things — it is stating precisely what could not be fixed.

  分析过程 · 先想清楚再作答

  1. 问「除了次数上限还有什么」，说明面试官已经预设你会答上限，真正在考的是你有没有真的跑过这条回路。只答上限的人拿基础分，能说出另外两种失效方式的才算过。
  2. 第一种就是无限打回：每改一版评审者挑一个新毛病，永远收敛不了。上限的作用不是省钱，是**保证流程一定会结束**。本课取最多打回 2 次、共 3 次执行，这个量级的取法是「一次有效的修改通常在第二次就完成，第三次还不行说明判据本身有问题」。
  3. 第二种是打回不说人话：评审者只回一句「不合格」，执行者拿不到可执行信息，第二稿原样再交一遍，于是必然打满上限、白烧三倍的钱。所以打回必须带具体理由，而且理由要回写进子任务的目标里带给执行者——「缺了退款结论，请补上」才是可执行的，「质量不佳」不是。
  4. 第三种最危险也最少被提到：评审者和执行者用同一个模型、同一套提示词时，它倾向于认可自己的输出。同一个模型对「什么算好答案」的偏好是一致的，让它复核自己刚写的东西，通过率会高得离谱，这道工序等于没有。缓解手段按性价比排：给评审者一份可核对的客观验收要求；换一个不同的模型来评审，哪怕更便宜；把评审做成逐条打分而不是一句结论。
  5. 还要点一句框架的兜底与业务上限的区别：编排框架通常自带一个递归步数上限，但那是最后一道保险丝，不能当业务上限用——它是全图的，你不知道是哪条回路失控；而且它触发时抛异常，你连已有结果都拿不到，正好违背「降级返回」的原则。
  6. 可以预期的追问：上限用完了返回什么？答：返回已有结果并标记降级，同时把最后一次的评审意见一起带出去，让上层能判断要不要转人工——这条回路的价值不只是修好，还包括「修不好时说清楚差在哪」。

  Key points

  • A retry cap exists to guarantee termination, not to save money — two rejections, three executions total
  • Rejections must carry specific, actionable reasons written back into the subtask goal; not good enough guarantees an identical resubmission and a maxed-out cap
  • The most dangerous failure is a reviewer sharing model and prompt with the executor: it approves its own output, pass rates inflate, and the step becomes theatre
  • Mitigate with an objective checkable rubric, a different model for review, and item-by-item scoring instead of a single verdict
  • The framework's recursion limit is a fuse, not a business cap: it is graph-wide and it throws, so you lose the partial results you meant to degrade to
  • When the cap is spent, return what you have with a degraded flag plus the last review comment so the layer above can escalate

  答题要点

  • 次数上限的作用是保证流程一定会结束，不是省钱；本课取最多打回 2 次、共 3 次执行
  • 打回必须带具体、可执行的理由并回写进子任务目标，只说「不合格」会让执行者原样重交、必然打满上限
  • 最危险的是评审者与执行者同模型同提示词，它倾向于认可自己的输出，通过率虚高、这道工序等于没有
  • 缓解手段：给客观可核对的验收要求、换一个模型来评审、逐条打分而不是一句结论
  • 框架自带的递归上限只是保险丝，不能当业务上限：它是全图的、触发时抛异常，连已有结果都拿不到
  • 上限用完要返回已有结果加降级标记，并把最后一次评审意见带出去，供上层决定是否转人工

D18 History Fidelity and Summarization, Multimodal Placeholders, Checkpointer Persistence

• When summarizing a long conversation, how do you keep the critical information from being lost — and what is different about this in a multi-agent system?长对话做摘要时，怎么保证关键信息不丢？在多 Agent 场景下这件事有什么特别的？

  Common in ChinaCommon overseasIntermediate#context-compression#multi-agent#reliability

  How to reason about it · think before answering

  1. The hinge is the second half. Answering only keep user constraints and the last few turns is the standard single-agent answer — passable, not memorable. Asking what is different in multi-agent is asking whether you have actually hit this in a collaboration graph.
  2. Get the single-agent half solid first: trigger on thresholds, never a timer. This course uses more than 20 messages or an estimated 8000 tokens, counting one character as one token — deliberately high, because underestimating means the threshold never fires. Keep the last 6 messages verbatim. Align the cut to a turn boundary: cutting between a tool call and its result produces a dangling message and most vendors return 400.
  3. Then name the real difference: in a single agent a summary loses detail; in a multi-agent graph a summary loses the criteria. A critic decides whether output passes by telling apart what is being reviewed from what the requirement was. A smooth narrative summary that flattens speakers reads fine and is useless to the critic.
  4. So multi-agent summarization has one extra hard requirement: every compressed message must leave behind two coordinates — its index and its speaker. The implementation is one line: build the transcript with numbered, role-prefixed entries before handing it to the model.
  5. Add the boundary that shows you have shipped this: summarize natural-language history only, never structured fields. Compressing the shared workspace into a sentence kills every lookup by task id and every comparison against an acceptance requirement, and structured data does not come back. Attachments are even more off-limits — they hold a reference, not content, so summarizing one orphans the underlying object.
  6. Expect the follow-up: which model writes the summary, and what if it fails? A cheaper small model is fine since the job is condensation, not reasoning. On failure the correct behaviour is to skip this round of compression, keep running, and alert — not to fail the whole execution. Setting the threshold at seventy or eighty percent exists precisely to leave that rescue room.

  分析过程 · 先想清楚再作答

  1. 题眼在后半句。只答「保留用户约束、保留最近几轮」是单 Agent 的标准答案，能过但不出彩；面试官问「多 Agent 有什么特别的」，是在看你有没有真的在协作图里踩过这个坑。
  2. 先把单 Agent 那半答扎实：触发用阈值不用定时器，本课口径是消息超过 20 条或估算超过 8000 token（一个字符算一个 token，故意高估，低估会让阈值永远触发不了）；保留最近 6 条原文不动；切口必须对齐到一轮的开头，切在工具调用与工具结果之间会让下一次请求出现悬空消息，多数厂商直接返回 400。
  3. 然后给出多 Agent 那半的关键差别：单 Agent 里摘要丢的是**细节**，多 Agent 里摘要丢的是**判据**。评审者判一份产出合不合格，靠的是分清「这句是待验收的产出、那句是验收要求」；一段把发言人抹平的流水摘要读起来通顺，但评审拿它做不了任何判断。
  4. 所以多 Agent 的摘要有一条额外硬要求：每条被压掉的消息，在摘要里都要留下「第几条 + 谁说的」这两个坐标。实现只有一行——把转录写成带序号和角色前缀的形式再交给模型。
  5. 再补一条边界，这条最能显出你写过：**摘要只对自然语言历史动手，不碰任何结构化字段**。把共享工作区压成一句话，「按 id 找到某条子任务、比对验收要求」就整个失效了，结构化数据压成自然语言就再也回不去。附件字段更是碰不得——它存的是引用不是内容，摘要掉等于把那个对象变成孤儿。
  6. 可以预期的追问：摘要用哪个模型、失败了怎么办？答可以用更便宜的小模型（它只做归纳不做推理），失败时的正确行为是**跳过这一轮压缩继续跑**并告警，而不是让整次执行失败——阈值定在七八成就是为了留出这次抢救余量。

  Key points

  • Trigger on thresholds, not timers: more than 20 messages or an estimated 8000 tokens, counting one character as one token to stay conservative
  • Keep the last 6 messages verbatim and align the cut to a turn boundary, or you ship a dangling tool call and the request 400s
  • The multi-agent difference: a summary loses criteria, not just detail — the critic needs to know who said what and at which step
  • So every compressed message keeps its index and speaker in the summary; the implementation is a numbered, role-prefixed transcript
  • Summarize natural-language history only — never the shared workspace or other structured fields, and never the attachment references
  • A cheaper small model is fine for summarizing; if the call fails, skip compression for this round and alert rather than failing the run

  答题要点

  • 触发用阈值不用定时器：超过 20 条或估算超过 8000 token，token 按一字符一 token 保守高估
  • 保留最近 6 条原文不动，切口必须对齐到一轮开头，否则会出现有调用没结果的悬空消息、请求直接 400
  • 多 Agent 的差别：摘要丢的不是细节而是判据，评审者靠「谁在第几步说的」区分产出与验收要求
  • 所以每条被压掉的消息都要在摘要里留下条号与发言人，实现就是把转录写成带序号和角色的形式
  • 只压自然语言历史，不碰共享工作区这类结构化字段，更不能碰存引用的附件字段
  • 摘要可用更便宜的小模型；摘要调用失败时跳过这一轮压缩并告警，不要让整次执行失败
• What do you need to watch out for when replaying execution from a checkpoint? Give failure modes you would actually hit.从 checkpoint 恢复执行（replay）需要注意什么？说几个真实会踩的坑。

  Common in ChinaCommon overseasDeep dive#checkpointing#replay#reliability

  How to reason about it · think before answering

  1. The easy failure is answering just load it and keep going. The discriminator is recognising that almost every replay bug is silent — no exception, clean logs, plausible output, and you only notice when you diff the data. Saying that up front wins half the question.
  2. Give a chain first: a checkpoint stores the state shape as the code of that moment understood it, and replay pushes it back into today's code. So every failure comes from a mismatch across those two ends — the shape of the data, the entry point of execution, and things that should never have been replayed at all.
  3. Trap one: feeding the input again on resume. Resume takes no input; the state is already in the checkpoint. Passing the original message once more makes the framework treat it as a fresh update stacked on the interrupt point, and the history quietly doubles. Nothing throws.
  4. Trap two: forking without a checkpoint id. With only the thread id you get that thread's latest state, so start over from step 2 silently becomes append after the last step. Again nothing throws; you only see it by diffing the task list.
  5. Trap three: version drift. Rename a field or add a required one and every old checkpoint stops matching the new code. A missing field reads as undefined, which renders as the literal string undefined in user-facing text and as NaN in arithmetic — a tool-budget ceiling compared against NaN is always false, so the budget silently stops existing on resumed threads. Migrate on read, and keep the migration to defaults and renames only: it must never fail.
  6. Trap four: replayable data that should not be replayed. A one-off human override written into graph state gets checkpointed and re-applied on every resume. The test: does this describe how this run executes, or what this conversation is? The former belongs in runtime config, only the latter in state.
  7. Expect the follow-up: are pending parallel tasks preserved? Yes — a checkpoint holds not just the state snapshot but the steps not yet run, arguments included, so the planner does not re-run. But they live in a framework-internal channel, so a hand-rolled store that persists state and forgets that half will resume into a graph that looks finished while no work was ever dispatched.

  分析过程 · 先想清楚再作答

  1. 这题最容易答成「读出来接着跑就行」。区分度在于你能不能说出**这些坑几乎全是静默的**——不抛异常、日志干净、结果看起来也对，只有对比数据时才发现不对。能说出这一点，答案就已经赢了一半。
  2. 先给一条推导链：检查点里存的是「当时那个版本的代码眼里的状态形状」，恢复就是把它塞回今天这个版本的代码里。所以所有坑都来自**两端不一致**：数据的形状、执行的入口、和那些不该被重放的东西。
  3. 坑一，恢复时又把输入喂了一遍。恢复的入口是不带输入地调用，状态已经在检查点里；带着原来那句话再调一次，框架会把它当成一次新的状态更新叠在中断点上，历史变成两份。它不报错。
  4. 坑二，分叉忘了带检查点 id。只给会话 id 拿到的是这条线最新的状态，于是「从第 2 步重来」变成了「在最后一步后面接着写」。同样不报错，只有对比子任务列表才看得出来。
  5. 坑三，版本兼容。改一个字段名、加一个必填字段，库里的老检查点就和新代码对不上；而缺字段读出来是 undefined，拼进文案就是字符串「undefined」，参与算术就是 NaN——比如工具预算的上限判断，一旦变成 NaN 比较，恒为假，预算上限在恢复出来的那条线上彻底失效。正确做法是在读的那一侧迁移，迁移函数只补默认值和改名、不做业务判断，绝不能失败。
  6. 坑四，不该被重放的东西进了状态。一次性的人工干预（比如人工改派）如果写进图状态，就会被检查点持久化并在每次恢复时重放一遍。判断口径：这条信息说的是「这一次执行怎么跑」还是「这个会话是什么」，前者进运行时配置，后者才进状态。
  7. 可以预期的追问：待执行的并行子任务存不存？答存——检查点里除了状态快照还有一份「还没跑的那几步，连参数一起」，所以恢复不用重跑规划节点；但它存在框架的内部通道里，自研存储层只实现「存状态」而漏掉这一半，恢复出来的图会看起来跑完了、其实一件活都没派出去。

  Key points

  • Lead with the pattern: replay bugs are almost all silent — no exception, clean logs, plausible output
  • Resume takes no input; passing one appends another update at the interrupt point and doubles the history
  • Forking requires the checkpoint id — thread id alone lands on the latest state, turning start over from step 2 into append after the end
  • Version drift: missing fields read as undefined or NaN, so comparisons like a tool-budget ceiling become permanently false. Migrate on read, restricted to defaults and renames, and never let it fail
  • Keep one-off human overrides out of graph state or they get persisted and re-applied on every resume — how this run executes belongs in config, what this conversation is belongs in state
  • Pending parallel tasks are stored with their arguments, so the planner does not re-run; a hand-rolled store that skips that half resumes into a graph that dispatches nothing

  答题要点

  • 先点破共性：replay 的坑几乎全是静默的，不报错、日志干净、结果看着也对
  • 恢复不要带输入，带了就是在中断点上又追加一次，历史变成两份
  • 分叉必须带检查点 id，只给会话 id 会落在最新状态上，「从第 2 步重来」变成「接着往后写」
  • 版本兼容：缺字段读出来是 undefined 或 NaN，会让预算上限之类的比较恒为假；在读的那一侧迁移，迁移只补默认值和改名且不能失败
  • 一次性的人工干预不要进图状态，否则会被持久化并在每次恢复时重放；「这次怎么跑」进配置，「这个会话是什么」才进状态
  • 待执行的并行子任务连参数一起存在检查点里，所以恢复不重跑规划；自研存储层漏掉这一半，恢复出来的图会一件活都不派

D21 Evaluation and Observability: a Golden Set, LLM-as-Judge, Tracing, a Failure-Rate/Cost Dashboard; Pi vs. LangGraph Summary; Week Three Retrospective

• What makes LLM-as-judge unreliable, and what do you do about it?用大模型给大模型的输出打分（LLM-as-judge），有哪些不可靠的地方？怎么办？

  Common in ChinaCommon overseasDeep dive#evaluation#llm-as-judge#reliability

  How to reason about it · think before answering

  1. This screens for whether you have actually used it. People who have can name specific failure shapes with magnitudes; people who have not just say it might be inaccurate.
  2. First, self-preference: when the judge and the evaluated agent share a model, it favours its own output — the same model has a consistent notion of what a good answer looks like, so asking it to review what it just wrote gets an approving verdict. Measured: on the same batch of deliberately degraded outputs, a same-model judge gave 14/15 while a different model gave 12/15, and the extra passes were exactly the borderline cases worth catching. This is not confined to judges — every model-grading-model position has it, and a Critic node is the same problem.
  3. Second, length bias: judges reward longer answers. Measured: padding a correct 33-character reply with 141 characters of irrelevant pleasantries moved an impression-based rubric from 2 to 4 without changing a word of substance.
  4. Third, rubric drift: scores shift wholesale when the judge prompt is tweaked. The same output scored 2 under one rubric and 5 under another. Hence the hard rule: scores are comparable only within one judge prompt, and cross-version comparison is meaningless.
  5. Match each remedy to its failure rather than saying run it a few more times. Freeze and version the judge prompt — every score record carries its rubric version and judge model, which are its coordinates, and a dashboard that finds two rubrics mixed should refuse to aggregate rather than emit a meaningless average. Default to a different model as judge, as a default and not an option. Keep a small human-labelled calibration set and re-run it whenever the rubric changes, comparing verdicts (pass or fail) rather than score deltas — one point of drift is fine, a flipped verdict is an incident.
  6. And one deeper fix: replace impressionistic criteria with a checkable list, which also dissolves length bias — counting items off a list gives padding nothing to earn. Measured, that padded reply scored 5 both before and after under the checklist rubric.
  7. Expect: is a judge cheaper than humans? The judge's cost is the same order as the system being evaluated, so what a full evaluation run costs decides whether you run it per commit or nightly. Human cost is not money but latency — it cannot give you feedback at the speed of one prompt edit, which is why humans belong on the calibration set only.

  分析过程 · 先想清楚再作答

  1. 这题筛的是「你是真用过，还是听说过」。用过的人能报出具体的失效形态和量级，没用过的人只会说「可能不准」。
  2. 第一种，**同源偏差**：judge 和被评估的 Agent 用同一个模型时，它偏向认可自己的输出——同一个模型对「什么算好答案」的偏好是一致的，让它复核自己刚写的东西，它当然觉得没问题。实测数量级：同一批被改坏的产出，同源 judge 给 14/15，换个模型只给 12/15，被多放过去的正是最该抓的边缘产出。这个坑不止在 judge，**凡是「模型评模型」的位置都有**，Critic 节点是同一个问题。
  3. 第二种，**长度偏好**：judge 倾向给篇幅大的答案更高分。实测：一条 33 字的正确回复灌上 141 字无关客套话，凭印象打分的提示词就从 2 分涨到 4 分，内容一个字没变。
  4. 第三种，**评分提示词漂移**：judge 的评分随提示词微调整体移动。同一份产出，两套评分提示词一套给 2 分一套给 5 分。所以有条硬纪律——**分数只在同一套 judge 提示词内部可比**，跨版本比较是没有意义的。
  5. 解药要一一对应，别笼统说「多测几次」：固定 judge 提示词并版本化（每条评分记录带上 rubric 版本与 judge 模型，那是它的坐标；面板发现混了两套口径应当直接拒绝聚合，而不是算出一个没含义的平均分）；默认用不同的模型当 judge，而且这该是默认值不是可选项；留一小批人工标注做校准集，每次改评分提示词拿它对一遍，**比的是结论（过或不过）而不是分数差**——差 1 分无所谓，结论翻了就是事故。
  6. 还有一条更根本的：**把评分标准从主观印象换成可核对的清单**，它同时解掉长度偏好——照清单逐条数，灌水加不了分。实测那条灌水回复在清单口径下前后都是 5 分，纹丝不动。
  7. 可以预期的追问：judge 便宜还是人工便宜？答：judge 的成本和被评估的系统本身一个量级，所以「跑一次全量评估多少钱」是你决定每次提交都跑还是每天跑一次的依据；而人工的成本不在钱在延迟——它给不了你改一次提示词就想看一次结果的反馈速度，所以人工只该用在校准集上。

  Key points

  • Self-preference: a same-model judge inflates scores (14/15 vs 12/15 cross-model), and it applies to every model-grading-model spot including Critic
  • Length bias: 141 characters of padding moved an impression score from 2 to 4 with no substantive change
  • Rubric drift: the same output scored 2 and 5 under two rubrics, so scores compare only within one judge prompt
  • Remedies map one-to-one: version the rubric and store it alongside each record, refuse to aggregate mixed rubrics, default to a different judge model
  • Keep a human-labelled calibration set and compare verdicts, not score deltas — a point of drift is fine, a flipped verdict is an incident
  • The deeper fix is a checkable list instead of impressions, which also removes length bias (the padded reply scored 5 both ways)

  答题要点

  • 同源偏差：judge 与被评估 Agent 同模型会虚高（实测 14/15 vs 异源 12/15），且凡「模型评模型」的位置都有，Critic 同理
  • 长度偏好：灌水 141 字能让印象分从 2 涨到 4，内容一字未变
  • 评分提示词漂移：同一产出两套 rubric 一个 2 分一个 5 分，所以分数只在同一套提示词内部可比
  • 解药一一对应：rubric 版本化并随记录存坐标、面板发现混口径直接拒绝聚合、默认换模型当 judge
  • 留人工标注校准集，比结论（过/不过）而不是比分数差——差 1 分无所谓，结论翻了是事故
  • 更根本的是把主观印象换成可核对的清单，同时解掉长度偏好（清单口径下灌水前后都是 5 分）

MCP in 7 Days: Wire Tools Into Any Agent

D5 Writing an MCP Client: Discovering and Calling Tools Inside Your Own Agent Loop, Multi-Server Aggregation and Name Collisions

• One of your MCP servers times out in production. How should your agent loop react?线上一个 MCP 服务端超时了。你的 Agent 循环应该怎么反应？

  Common in ChinaCommon overseasIntermediate#client#reliability

  How to reason about it · think before answering

  1. This probes engineering instinct: can you separate 'one dependency is down' from 'this turn fails'. Answering 'retry three times' just moves the problem, and the interviewer will ask what the user is staring at meanwhile.
  2. Split by phase first. A timeout happens at two very different moments: discovery (server/discover or tools/list) and invocation (tools/call). The correct reaction differs, and blurring them shows you have not built this.
  3. Discovery: wrap each server in its own try/catch, record the failure with its reason in a down list, and continue to the next server. The tool table loses a few entries but the loop still starts. Record the reason, not a boolean, because afterwards you must be able to say what is missing and why.
  4. Invocation: translate the failure into a tool result with isError true and feed it back to the model rather than throwing. MCP already uses isError for 'the tool failed but the protocol succeeded', so the model can switch tools or arguments; throwing kills the turn and leaves the user with no explanation.
  5. Then three supporting points. Every request needs a timeout, because on stdio a silent server is silent forever. Idempotency decides whether a retry is safe, and the idempotentHint annotation is a hint, not a guarantee, so writes need a client-side dedup key. And outages must be visible, surfaced in the UI or in the system prompt, or the model will behave as if the capability never existed and confidently report nothing found.
  6. Conclusion: the worst outcome of one server timing out should be a few missing tools plus an explicit note, never a failed turn.
  7. Likely follow-ups: should you add a circuit breaker? Yes, after consecutive failures mark the server unusable for a while so you stop paying a timeout every turn, with recovery by health check or reconnect on the next session. And how do you set the timeout? Per tool rather than per server, since a thirty-second analysis tool and a cache lookup should not share a threshold.

  分析过程 · 先想清楚再作答

  1. 这题看的是工程直觉：能不能把「一个依赖挂了」和「这一轮对话失败」分开。答「重试三次」是把问题往后推了一步，面试官会立刻追问重试期间用户在等什么。
  2. 先分阶段。超时发生在两个完全不同的时刻：发现阶段（server/discover 或 tools/list）和调用阶段（tools/call）。两个阶段的正确反应不一样，混着答就会露怯。
  3. 发现阶段：逐个服务端 try/catch，失败的记进一张掉线表并继续下一个。整张工具表少几个工具，但循环照常起得来。记的必须是原因而不是一个布尔值，因为事后你要能回答少了什么、为什么少。
  4. 调用阶段：把失败翻译成一条 isError 为真的工具结果喂回模型，不要抛。理由是 MCP 本来就用 isError 表达「工具执行失败但协议是成功的」，模型看得见这句话就有机会换个工具或换个参数；抛出去只会把整轮对话打断，而且用户什么解释都得不到。
  5. 接着补三件配套的事。一是**每条请求都必须有超时**，stdio 上服务端不回你就永远不回；二是**幂等性决定能不能重试**，工具注解里的 idempotentHint 是提示不是保证，写操作的重试要靠客户端自己的去重键；三是**掉线要让用户看得见**，把掉线的服务端标在界面上或写进系统提示，否则模型会表现得像那个能力从来不存在，一本正经地说查不到。
  6. 结论：一个服务端超时，最坏的后果应该是少几个工具加一条明确的说明，而不是这一轮对话失败。
  7. 可预期的追问：要不要熔断？连续失败到阈值就把这个服务端标记为不可用一段时间，避免每一轮都白等一次超时；恢复用探活或下一次会话重连。再追问会问到超时值怎么定——按工具而不是按服务端定，一个跑三十秒的分析工具和一个查缓存的工具不该共用一个阈值。

  Key points

  • Split by phase: per-server try/catch during discovery with a recorded reason, and never throw during invocation
  • Translate call failures into isError tool results so the model can switch tools or arguments
  • Every request needs a timeout; retry safety depends on idempotency, and the annotation is a hint, not a guarantee
  • Outages must be visible to user and model, otherwise silent degradation makes the model deny the capability ever existed

  答题要点

  • 分阶段：发现阶段逐个服务端 try/catch 记进掉线表并继续，调用阶段一律不抛
  • 调用失败翻译成 isError 为真的工具结果喂回模型，让它换工具或换参数
  • 每条请求必须设超时；能不能重试取决于幂等性，注解只是提示不是保证
  • 掉线必须对用户和模型可见，否则会变成静默降级，模型会假装那个能力不存在

RAG in 14 Days: From Retrieval to Trustworthy Answers

D12 Agentic RAG: Turning Retrieval Into a Tool So the Model Decides Whether to Search, How Many Times, and Whether to Start Over

• Self-reflective retrieval rewrites the query and retries. How do you guarantee it terminates instead of spinning on the same query forever?自反思式检索会反复改写查询重试。你怎么保证它一定会停下来，而不是在同一个查询上原地打转？

  Common in ChinaCommon overseasIntermediate#agentic-rag#self-reflection#reliability

  How to reason about it · think before answering

  1. This checks whether you have actually run such a loop. 'Set a max iteration count' is half an answer: it stops one failure mode and lets two others through.
  2. Split runaway behaviour into three shapes and give each its own brake. Progress that never completes is capped by max rounds. Per-round budgets that pass individually but blow up in aggregate need a cumulative token budget - four rounds of 600 tokens each never trips a per-round check yet quadruples what reaches the model. Spinning in place needs duplicate-query detection.
  3. Two implementation details prove you have written it: the duplicate check belongs before the retrieval call, otherwise you pay for a call to learn you are looping; and queries must be normalized to a set of terms, or 'failover approval' and 'approval failover' count as two distinct queries and the loop keeps turning.
  4. Say what happens after it stops: stop reasons must be recorded as distinct categories - satisfied, gave up, hit round cap, hit token budget, duplicate query. Collapsing them into 'loop finished' hides how often the system simply surrendered.
  5. An easy miss: installing a brake is not testing it. If the default token budget sits far above real usage it never fires, which is the same as not having one. Every brake needs a case that trips it.
  6. Expected follow-up: what if the model says 'not enough' when it actually is? Make the assessment structured - which elements are covered, which are missing - and treat an empty missing list as sufficient, so the decision is auditable rather than a bare boolean.

  分析过程 · 先想清楚再作答

  1. 这题在考「有没有真让循环跑过」。只答「设一个最大轮数」的能拿一半分，因为最大轮数只拦住了一类失控，剩下两类照样漏出去。
  2. 怎么拆：把失控分成三种形态，每种配一道闸。一是「每轮都在推进但永远推进不完」，用最大轮数拦；二是「每轮都不超标但累计爆掉」，用累计 token 预算拦——四轮各读 600 token 没有一轮超标，可送进模型的材料已经是单轮的四倍；三是「原地打转」，用重复查询检测拦。
  3. 重复查询检测有两个实现细节，答出来就说明真写过：一是要放在检索之前，否则要白花一次调用才发现自己在转圈；二是判重要对查询做归一化，只看词的集合，否则「主备切换 审批」和「审批 主备切换」会被当成两个不同的查询，圈照转不误。
  4. 还要说清停下来之后怎么办：停止原因必须分类记录，「查够了」「主动认输」「撞到轮数」「撞到预算」「原地打转」是五种不同的结局。把它们混成一个「循环结束」，你就永远看不见系统在多大比例的问题上其实是放弃了。
  5. 一个容易被忽略的点：闸门装了不等于验过。默认预算如果比实际用量高一大截，跑多少遍都踩不响它，等于没装。每一道闸都要构造一个用例把它踩响，这是验收的一部分。
  6. 可预期的追问是「模型自己说不够，但其实已经够了怎么办」。答案是自评要给结构化输出（覆盖了哪些要素、缺哪些），缺失项为空却仍判不够时按「够了」处理——让判断可审计，而不是信一个布尔值。

  Key points

  • Three brakes, none optional: max rounds, cumulative token budget, duplicate-query detection.
  • The cumulative budget catches rounds that each pass but blow up together - the round cap cannot see that.
  • Check for duplicates before retrieving, and normalize the query to a term set before comparing.
  • Record stop reasons as distinct categories rather than one 'finished' bucket.
  • Every brake needs a case that actually trips it; an untested brake is no brake.
  • Have the assessor emit covered and missing elements so 'not enough' is auditable.

  答题要点

  • 三道闸缺一不可：最大轮数、累计 token 预算、重复查询检测。
  • 累计预算拦的是「每轮都不超但加起来爆掉」，轮数闸看不见这件事。
  • 重复查询检测要放在检索之前，且查询要归一化成词的集合再判重。
  • 停止原因分类记录：查够了、主动认输、撞轮数、撞预算、原地打转是五种结局。
  • 每一道闸都要构造用例踩响，装了没验过等于没装。
  • 自评输出结构化的覆盖与缺失项，让「不够」这个判断可审计。