Interview Bank

From Frontend Engineer to Agent Engineer in 30 Days

D15 A Tour of Multi-Agent Patterns (Router/Supervisor, Planner-Executor, Critic, Swarm, Blackboard) and When Not to Use Them; Getting Started With LangGraph

• When should you not introduce a multi-agent system? Give operational criteria, not it depends.什么情况下不应该引入多 Agent 系统？请给出可操作的判据，而不是「视情况而定」。

  Common in ChinaCommon overseasDeep dive#multi-agent#architecture#trade-offs

  How to reason about it · think before answering

  1. This is the highest-signal question in the set because it is asked in reverse. Most candidates keep selling how powerful multi-agent is, while the interviewer is looking for someone who will say no — on a real team, blocking one unnecessary architecture upgrade is worth more than implementing three patterns.
  2. Lead with the default: do not split. Then give three criteria, any one of which justifies splitting — the system prompt contains mutually exclusive behavioural requirements (strictly enforce refund rules while also warmly retaining the customer; these are not hard to write, they are impossible to optimise together); the tool count exceeds what the model picks reliably (roughly eight as a rule of thumb, and the first response to crossing it is consolidating tools, not splitting agents); or one step needs its own failure and retry semantics. None of the three, and a single agent with a few tools is enough.
  3. Then name the most common bad split: treating a prompt problem as an architecture problem. Quality is poor, so we split into three agents — but nine times out of ten poor quality comes from vague prompts, tool descriptions that interfere with each other, or irrelevant history in the context. All three survive the split and are now harder to find. Splitting fixes conflicting responsibilities, not weak capability.
  4. Add two scenarios that clearly should not split: latency-sensitive interactions, where each extra hop is another model round trip and voice or realtime completion becomes unusable; and read-only lookup flows, where a support assistant with three or four tools gains no accuracy from splitting and simply triples the bill.
  5. Then offer an executable verification path, which earns points: keep the single-agent version as a baseline for any split and A/B both against the same golden set, comparing accuracy, per-conversation cost and latency together. An architecture upgrade with no baseline is a refactor with no evidence.
  6. Expect: what if your manager insists on multi-agent? Frame it as a reversible experiment — make the one cut you are most confident in (usually the conflicting-rules criterion), keep the baseline, and bring data in two weeks. That answer shows technical judgement and a way to disagree without stonewalling.

  分析过程 · 先想清楚再作答

  1. 这是本组最有区分度的题，因为它反着问。绝大多数候选人会顺着「多 Agent 很强大」讲下去，而面试官问这题正是想找那个会说不的人——**在真实团队里，拦住一次不必要的架构升级，价值高于实现三个模式**。
  2. 先给结论式的默认值：默认答案是不拆。然后给三条判据，命中任意一条才拆——一是单个 Agent 的系统提示词里出现了互斥的行为要求（既要严格核对退款规则又要热情挽留，这两条不是难写，是不可能同时最优）；二是工具数量超过模型能稳定选对的规模（经验线大约八个，超线的第一反应是合并工具而不是拆 Agent）；三是某一步需要独立的失败与重试语义。三条都不命中，单 Agent 加几个工具就够。
  3. 接着点名最常见的错拆：把提示词问题当成架构问题。「回答质量不好，所以拆成三个 Agent」——质量差有九成来自提示词含糊、工具描述互相干扰、上下文塞了无关历史，这三样拆完一样存在，只是分散到三个地方更难查。**拆 Agent 解决的是职责冲突，不是能力不足。**
  4. 再补两类明确不该拆的场景：一是低延迟要求的场景，多一跳就多一次模型往返，对语音或实时补全这类交互直接不可用；二是只读的简单查询链路，三五个工具的客服助手拆了只是把一次调用变成三次，准确率不会涨、账单会涨。
  5. 然后给一条可执行的验证路径，这是加分项：任何拆分都先留住单 Agent 版本当对照基线，用同一批标准样本集跑 A/B，同时比准确率、每次对话成本和延迟。**拿不出对照基线的架构升级，等于没有证据的重构。**
  6. 可以预期的追问：那如果老板就是要求上多 Agent 呢？答：那就把它当成一个可回退的实验来做——先按判据拆最有把握的那一刀（通常是互斥规则那一条），保留基线，两周后拿数据说话。这个回答同时展示了技术判断和沟通方式，比硬顶或硬上都好。

  Key points

  • Default to not splitting; split only if one of three criteria holds: mutually exclusive prompt requirements, tool count past the roughly-eight warning line, or a step needing its own failure and retry semantics
  • Too many tools should first trigger tool consolidation and tighter descriptions; splitting agents is the second response
  • The most common bad split is treating a prompt problem as an architecture problem — vague prompts, interfering tool descriptions and irrelevant history all survive the split
  • Clear do-not-split cases: latency-sensitive interactions where every hop adds a model round trip, and read-only lookup flows where accuracy does not move but the bill does
  • Always keep the single-agent version as a baseline and compare accuracy, cost and latency on the same golden set
  • Say the costs out loud: latency multiplies with steps, spend roughly triples, and debugging now spans routing plus state merging

  答题要点

  • 默认答案是不拆；三条判据命中任意一条才拆：提示词有互斥要求、工具超过约八个的告警线、某一步需要独立的失败与重试语义
  • 工具太多的第一反应是合并工具与收敛描述，拆 Agent 是第二反应
  • 最常见的错拆是把提示词问题当架构问题——质量差多半来自提示词含糊、工具描述干扰、上下文塞了无关历史，拆完这三样照旧存在
  • 明确不该拆：低延迟交互（每多一跳就多一次模型往返）、只读的简单查询链路（准确率不涨、账单涨）
  • 任何拆分都要留单 Agent 版本当对照基线，用同一批标准样本集比准确率、成本和延迟
  • 代价要说出口：延迟按步数乘倍数、成本约三倍、调试要同时排查路由与状态合并

D16 Dynamic Routing With a Supervisor: Structured-Output Routing, Override, routingReason

• How should the system handle an uncertain or wrong routing decision, and how do you pick the threshold?路由不确定或者路由错误时，系统应该怎么兜底？阈值该怎么定？

  Common in ChinaCommon overseasDeep dive#routing#fallback#reliability

  How to reason about it · think before answering

  1. The hinge is that uncertain and wrong are two different failures. Most candidates answer retry or escalate to a human, collapsing both into one. The discriminator is stating a value judgement before giving a policy.
  2. The claim first: routing to the wrong sub-agent is far worse than failing to route. A failure announces itself and lets you ask a clarifying question. A wrong route does not — the receiving agent has no idea it got the wrong job and will produce a confident, well-formatted, wrong answer that the user will act on. A confident wrong answer costs a hundred times more than I did not catch that.
  3. Then give a concrete policy with real numbers: if the model's confidence is below 0.6, or the route name is not in the allowed list, fall back to the small-talk agent and stamp the reason with a fallback prefix plus a cause code (low confidence, unknown route, invalid shape). The fallback agent's job is to ask for the one missing detail rather than guess — falling back means handing the uncertainty back to the user.
  4. The threshold question is the real test, so do not recite a number: it depends on which error is more expensive. In customer support one extra question costs mild annoyance while a misroute can become a wrong refund promise, so stay conservative. For an internal tool the extra question is the bigger cost, so lower it. Then give a method: sweep thresholds over a golden set, plot misroute rate against clarification rate, and pick the knee.
  5. Name the trap: the confidence number is self-reported and is not a probability. Nine tenths does not mean nine in ten are right. It is a usable ranking signal within one model and one prompt — good as a gate, useless for expected-value math. Real accuracy comes from offline evaluation.
  6. Expect: does falling back just hide the problem? Not if you record cause codes. Group a week of fallbacks by cause and you can see exactly which intent the routing prompt fails to describe. The fallback stops the bleeding; the cause code is what fixes it.

  分析过程 · 先想清楚再作答

  1. 题眼在「不确定」和「错误」是两件事。多数人只答重试或人工接管，那是把两个问题揉成一个。区分度在于你能不能先给出一条价值判断，再给策略。
  2. 先立论：**路由到错的子 Agent，比路由失败糟糕得多**。失败你至少知道自己失败了，可以追问一句；错了，接手的子 Agent 完全不知道自己接错了活，会用笃定的语气给出一个格式完整的错误答案，用户不会怀疑，会照着去操作。一个自信的错误答案比一句「我没听清」贵一百倍。
  3. 再给可执行的策略，数字要具体：模型给的置信度低于 0.6，或者路由名不在合法名单里，一律落到兜底的 smalltalk，并在 routingReason 里打上 fallback 前缀加原因码（低置信度、未知路由、结构非法各一种）。兜底那位的人设是「信息不足先追问一句缺的关键信息，不要猜」——兜底的本质是把不确定性还给用户。
  4. 阈值怎么定这一问是重点，别背数字：**取决于两类错误哪一类更贵**。客服场景里多问一句只是用户小小的不耐烦，派错可能变成一条错误的退款承诺，所以宁可保守取 0.6；内部工具型 Agent 里多问一句反而更烦人，阈值就该放低。再补一句可落地的定法：拿标准样本集扫一遍，画出不同阈值下的误派率与追问率，选拐点。
  5. 必须点破的一个坑：**置信度是模型自己报的，它不是概率**。模型说 0.9 不代表有九成对。它只是同一模型、同一提示词下相对可用的排序信号，只能当闸门用，不能拿去算期望值。真正的准确率要靠离线评估去量。
  6. 可以预期的追问：兜底会不会把问题掩盖掉？答案是不会，前提是你记了原因码——把一周内落进兜底的请求按原因分组，能直接看出分诊提示词缺了哪一类描述。兜底是止血，原因码才是治本的输入。

  Key points

  • Separate the two: a failed route can ask a clarifying question, a wrong route produces a confident wrong answer, and the second is far costlier
  • Policy: confidence below 0.6 or a route outside the allowed list falls back to small talk, stamped with a fallback prefix and a cause code
  • Falling back is not picking someone at random — the fallback agent asks for the missing detail instead of guessing
  • The threshold depends on which error costs more; sweep it over a golden set and pick the knee between misroutes and clarifications
  • Self-reported confidence is not a probability — use it as a gate only, and measure real accuracy offline
  • Record cause codes on every fallback; grouping them shows which intent the routing prompt fails to describe

  答题要点

  • 先分清两件事：路由失败可以追问，路由错误会让子 Agent 自信地给出错误答案，后者贵得多
  • 策略：置信度低于 0.6 或路由名不在名单里，一律落兜底的 smalltalk，并在 routingReason 打上 fallback 前缀加原因码
  • 兜底不是随便找个人接，而是把不确定性还给用户——兜底那位应当追问缺失的关键信息而不是猜
  • 阈值取决于两类错误哪一类更贵：客服场景多问一句便宜、派错很贵，所以保守；定法是拿标准样本集扫阈值找拐点
  • 置信度是模型自报的，不是概率，只能当闸门用；真正的准确率要靠离线评估量
  • 落兜底时记原因码，按原因分组就能看出分诊提示词缺了哪一类描述

D17 Planner-Executor-Critic Plus a Shared Workspace: Workspace State, toolBudget, Parallel Fan-Out, a Review Loop

• When several subtasks run in parallel and all write the same shared state, how do you design it so they do not clobber each other?多个子任务并行执行、都要写同一份共享状态时，怎么设计才不会互相覆盖？

  Common in ChinaCommon overseasDeep dive#multi-agent#state-management#concurrency

  How to reason about it · think before answering

  1. This one separates people fast, because most candidates answer locks or immutable data structures — instincts carried over from threads. A graph runtime has no concurrent memory writes at all: updates are collected and merged. Answering in the wrong frame is worse than answering incompletely.
  2. Get the mechanism right first: parallel nodes each return a delta, the runtime groups all deltas from the same step by field, then calls that field's reducer to compute the new value. So the question is not how to lock, it is whether that field's reducer is correct.
  3. Then give a reusable chain: how many writers touch this field in one step, and do they write the same record? One writer — last-write-wins is fine. Several writers on different records — appending to a list is fine. Several writers on the same record — upsert by key. Several writers on different fields of the same record — merge per field. Four cases, four reducers, and the chain transfers to any framework.
  4. Land on the common mistake: implementing update this record as append a new version with the same id. The symptom is not an error — the same id exists twice and which one comes first depends on who finished first, so any lookup by id may return the stale version. Clean logs, occasionally wrong results.
  5. Add the trade-off: you can leave the reducer alone and dedupe by id at every read instead. But there are three or four read sites, and missing one is an intermittent stale read; a reducer is written once and every read is clean afterwards. Solve it once on the field, or N times at the read sites.
  6. Expect: does nondeterministic ordering matter? Ideally the reducer is order-insensitive (commutative); if it is not, you must guarantee one writer per record. Upsert-by-id is the latter — it is last-write-wins and is safe only because each record has exactly one executor per round.

  分析过程 · 先想清楚再作答

  1. 这题的区分度极高，因为大多数人会答成「加锁」或者「用不可变数据结构」——都是从多线程经验迁移过来的答案，但图的执行模型里根本没有并发写内存这回事，写入是被收集起来统一合并的。答错方向比答不全更致命。
  2. 先把机制说对：并行节点各自返回一份增量，框架把同一轮里所有增量按字段收集，再逐字段调用这个字段的合并规则（reducer）算出新值。所以问题不是「怎么加锁」，而是**这个字段的合并规则写得对不对**。
  3. 然后给一条可复用的判断链：先问这个字段同一轮会被几个人写；再问他们写的是不是同一条记录。只有一个写者，默认的后写覆盖就够；多个写者写不同记录，数组追加就够；多个写者写同一条记录的同一份数据，要按主键原地更新；多个写者写同一条记录的不同字段，要做字段级合并。四种情况四种 reducer，这条链能直接迁移到任何框架。
  4. 结论落在最容易踩的那一格：把「更新一条记录」写成「往数组里追加一条同 id 的新版本」。它的症状不是报错，是同一个 id 在状态里有两份、而且哪份在前取决于谁先跑完——下游任何按 id 查的地方都可能拿到过期版本，日志干净、结果偶尔错。
  5. 补一句权衡：也可以不动 reducer，改成每处读状态前先按 id 去重。但读取点有三四处，漏一处就是一个偶发脏读；reducer 只写一次，之后所有读取点自动干净。在字段上解决一次，还是在每个读取点解决 N 次，这是同一个问题的两种成本。
  6. 可以预期的追问：那顺序不确定要不要紧？答：合并规则最好对顺序不敏感（可交换），做不到就必须保证每条记录只有一个写者。本课的按 id 原地更新属于后者——它是最后写入者获胜，靠「一轮里一条记录只有一个执行者」这个前提才安全。

  Key points

  • A graph runtime has no concurrent memory writes: nodes return deltas, the runtime groups them per field and calls that field's reducer — so the answer is a correct reducer, not a lock
  • Decision chain: how many writers per step, and same record or not — overwrite, append, upsert by key, or per-field merge
  • The classic bug is implementing update as append-a-new-version-with-the-same-id: two entries per id, order depends on who finished first, lookups return stale data, and nothing ever errors
  • The alternative is deduping at every read site, but there are several and missing one gives an intermittent stale read; a reducer is written once
  • Prefer an order-insensitive reducer; if it is not, guarantee exactly one writer per record per step

  答题要点

  • 图的执行模型里没有并发写内存：节点各返回增量，框架按字段收集后调用该字段的 reducer 合并，所以问题是 reducer 写得对不对，不是加不加锁
  • 判断链：同一轮几个写者、写的是不是同一条记录——单写者用覆盖、多写者写不同记录用追加、多写者写同一条记录用按主键原地更新、写同一条记录的不同字段要字段级合并
  • 最常见的错是把「更新」写成「追加同 id 的新版本」，症状是同 id 两份、顺序取决于谁先跑完、按 id 查会拿到过期版本，而且全程不报错
  • 另一条路是每处读取前手动去重，但读取点有好几处，漏一处就是偶发脏读；reducer 只写一次就一劳永逸
  • 合并规则最好对顺序不敏感；做不到就必须保证一轮里一条记录只有一个写者

D18 History Fidelity and Summarization, Multimodal Placeholders, Checkpointer Persistence

• What do you need to watch out for when replaying execution from a checkpoint? Give failure modes you would actually hit.从 checkpoint 恢复执行（replay）需要注意什么？说几个真实会踩的坑。

  Common in ChinaCommon overseasDeep dive#checkpointing#replay#reliability

  How to reason about it · think before answering

  1. The easy failure is answering just load it and keep going. The discriminator is recognising that almost every replay bug is silent — no exception, clean logs, plausible output, and you only notice when you diff the data. Saying that up front wins half the question.
  2. Give a chain first: a checkpoint stores the state shape as the code of that moment understood it, and replay pushes it back into today's code. So every failure comes from a mismatch across those two ends — the shape of the data, the entry point of execution, and things that should never have been replayed at all.
  3. Trap one: feeding the input again on resume. Resume takes no input; the state is already in the checkpoint. Passing the original message once more makes the framework treat it as a fresh update stacked on the interrupt point, and the history quietly doubles. Nothing throws.
  4. Trap two: forking without a checkpoint id. With only the thread id you get that thread's latest state, so start over from step 2 silently becomes append after the last step. Again nothing throws; you only see it by diffing the task list.
  5. Trap three: version drift. Rename a field or add a required one and every old checkpoint stops matching the new code. A missing field reads as undefined, which renders as the literal string undefined in user-facing text and as NaN in arithmetic — a tool-budget ceiling compared against NaN is always false, so the budget silently stops existing on resumed threads. Migrate on read, and keep the migration to defaults and renames only: it must never fail.
  6. Trap four: replayable data that should not be replayed. A one-off human override written into graph state gets checkpointed and re-applied on every resume. The test: does this describe how this run executes, or what this conversation is? The former belongs in runtime config, only the latter in state.
  7. Expect the follow-up: are pending parallel tasks preserved? Yes — a checkpoint holds not just the state snapshot but the steps not yet run, arguments included, so the planner does not re-run. But they live in a framework-internal channel, so a hand-rolled store that persists state and forgets that half will resume into a graph that looks finished while no work was ever dispatched.

  分析过程 · 先想清楚再作答

  1. 这题最容易答成「读出来接着跑就行」。区分度在于你能不能说出**这些坑几乎全是静默的**——不抛异常、日志干净、结果看起来也对，只有对比数据时才发现不对。能说出这一点，答案就已经赢了一半。
  2. 先给一条推导链：检查点里存的是「当时那个版本的代码眼里的状态形状」，恢复就是把它塞回今天这个版本的代码里。所以所有坑都来自**两端不一致**：数据的形状、执行的入口、和那些不该被重放的东西。
  3. 坑一，恢复时又把输入喂了一遍。恢复的入口是不带输入地调用，状态已经在检查点里；带着原来那句话再调一次，框架会把它当成一次新的状态更新叠在中断点上，历史变成两份。它不报错。
  4. 坑二，分叉忘了带检查点 id。只给会话 id 拿到的是这条线最新的状态，于是「从第 2 步重来」变成了「在最后一步后面接着写」。同样不报错，只有对比子任务列表才看得出来。
  5. 坑三，版本兼容。改一个字段名、加一个必填字段，库里的老检查点就和新代码对不上；而缺字段读出来是 undefined，拼进文案就是字符串「undefined」，参与算术就是 NaN——比如工具预算的上限判断，一旦变成 NaN 比较，恒为假，预算上限在恢复出来的那条线上彻底失效。正确做法是在读的那一侧迁移，迁移函数只补默认值和改名、不做业务判断，绝不能失败。
  6. 坑四，不该被重放的东西进了状态。一次性的人工干预（比如人工改派）如果写进图状态，就会被检查点持久化并在每次恢复时重放一遍。判断口径：这条信息说的是「这一次执行怎么跑」还是「这个会话是什么」，前者进运行时配置，后者才进状态。
  7. 可以预期的追问：待执行的并行子任务存不存？答存——检查点里除了状态快照还有一份「还没跑的那几步，连参数一起」，所以恢复不用重跑规划节点；但它存在框架的内部通道里，自研存储层只实现「存状态」而漏掉这一半，恢复出来的图会看起来跑完了、其实一件活都没派出去。

  Key points

  • Lead with the pattern: replay bugs are almost all silent — no exception, clean logs, plausible output
  • Resume takes no input; passing one appends another update at the interrupt point and doubles the history
  • Forking requires the checkpoint id — thread id alone lands on the latest state, turning start over from step 2 into append after the end
  • Version drift: missing fields read as undefined or NaN, so comparisons like a tool-budget ceiling become permanently false. Migrate on read, restricted to defaults and renames, and never let it fail
  • Keep one-off human overrides out of graph state or they get persisted and re-applied on every resume — how this run executes belongs in config, what this conversation is belongs in state
  • Pending parallel tasks are stored with their arguments, so the planner does not re-run; a hand-rolled store that skips that half resumes into a graph that dispatches nothing

  答题要点

  • 先点破共性：replay 的坑几乎全是静默的，不报错、日志干净、结果看着也对
  • 恢复不要带输入，带了就是在中断点上又追加一次，历史变成两份
  • 分叉必须带检查点 id，只给会话 id 会落在最新状态上，「从第 2 步重来」变成「接着往后写」
  • 版本兼容：缺字段读出来是 undefined 或 NaN，会让预算上限之类的比较恒为假；在读的那一侧迁移，迁移只补默认值和改名且不能失败
  • 一次性的人工干预不要进图状态，否则会被持久化并在每次恢复时重放；「这次怎么跑」进配置，「这个会话是什么」才进状态
  • 待执行的并行子任务连参数一起存在检查点里，所以恢复不重跑规划；自研存储层漏掉这一半，恢复出来的图会一件活都不派

D19 Cross-Service Agent Integration: Minting a User-Level JWT, JWKS Signature Verification, the inject/memory/usage Interfaces, Idempotent externalId

• You are designing the API surface an Agent platform exposes to other services. How do you draw the responsibility boundaries?设计一组给外部服务调用的 Agent 平台接口，你会怎么划分职责边界？

  Common in ChinaCommon overseasDeep dive#api-design#security#architecture

  How to reason about it · think before answering

  1. This is an open design question testing whether you have a reusable criterion. Candidates who start listing endpoints run out of material under follow-ups; candidates who give the criterion first turn follow-ups into extra points.
  2. Offer the criterion: draw boundaries by who owns the data, not by who calls it. Sessions, run records, memories and the cost ledger belong to the platform, so the platform exposes exactly three things — write one event in (inject), read and write memory, and read usage. Orchestration belongs to the caller, so the platform should not offer run this graph for me; that pulls someone else's responsibility inside your walls and freezes both sides.
  3. Second criterion, the security invariant that runs through the whole course: identity comes from the token, never from the request body. No endpoint accepts a userId; the server always reads sub. Break this once and the authorization model collapses — a usage endpoint that accepts a userId query parameter lets any valid token enumerate everyone's spend. The same principle appeared on the memory search tool: the model gets no identity parameter, the server fills it in.
  4. Third, return the minimum necessary. Usage returns aggregates, not line items, because line items carry run ids and model choices — that hands over your internal strategy. Memory supports a query with a result limit rather than dump everything this user ever said; once that exists, some caller in a hurry will make it the default.
  5. Fourth, every write endpoint must be safely replayable: an externalId, a uniqueness constraint underneath, and 200 on a repeat. Cross-service calls will be duplicated; this is not optional.
  6. Expect: what dimension do you rate-limit on? Per user, not per caller — limiting per caller lets one user's runaway retries consume everyone's budget. Also guard against loops: tag injected messages with their source, or two services can pull each other into an infinite cycle and the bill is the only thing that tells you.

  分析过程 · 先想清楚再作答

  1. 这是开放题，考的是你有没有一条能反复用的划分依据。上来就罗列接口清单的人会被追问到没词；先给依据再给清单的人，追问反而是加分机会。
  2. 给一条判据：**按「谁拥有这份数据」划，不按「谁调用它」划。** 会话、执行记录、记忆、成本台账都属于平台，所以平台开的三个口子恰好是「写一条进来（inject）」「读写记忆（memory）」「查账（usage）」；编排逻辑属于对方，平台就不该提供「帮我跑一遍这个图」的接口——那是把对方的职责搬到自己身上，将来两边都改不动。
  3. 第二条判据是**贯穿全课的安全不变量：身份只能来自令牌，不能来自请求体**。所有接口都不接受 userId 参数，服务端一律从令牌的 sub 取。这条一旦破例，权限模型就整个塌了：查成本的接口如果接受 userId 查询参数，任何一张有效令牌都能遍历所有人的消费金额。同一条原则在 D12 的记忆检索工具上也出现过——不给模型身份参数，服务端自己填。
  4. 第三条是**返回粒度要按最小必要给**。usage 只返回汇总不返回明细，因为明细里带着执行 id 和模型选型，等于把平台的内部策略一并交出去；memory 要支持按 query 检索并限制条数，不提供「把这个人的所有记忆倒出来」的接口——一旦提供，它迟早会被某个图省事的调用方用成默认写法。
  5. 第四条是**每个写接口都要能被安全重放**：带 externalId、唯一约束兜底、重复返回 200。跨服务调用一定会重复，这不是要不要做的问题。
  6. 可以预期的追问：那限流按什么维度做？答「每用户，不是每调用方」——按调用方限流的话，一个用户的异常重试会把所有人的额度吃光；另外写接口要防回环，注入的消息要打来源标记，否则两个服务能把彼此拉进无限循环，账单是唯一会提醒你的东西。

  Key points

  • Draw boundaries by data ownership, not by caller: sessions, memory and the ledger belong to the platform, orchestration belongs to the caller
  • Three endpoints for three kinds of ownership — inject, memory, usage — and no run this graph for me endpoint that crosses the line
  • No endpoint accepts a userId; identity always comes from the token's sub, and one exception collapses the model
  • Return the minimum necessary: usage gives aggregates only, memory takes a query with a limit instead of dumping everything
  • Every write endpoint carries an externalId backed by a uniqueness constraint and answers 200 on repeats
  • Rate-limit per user rather than per caller, and tag injected messages with their source so two services cannot loop forever

  答题要点

  • 按「谁拥有这份数据」划边界，不按「谁调用」划：会话、记忆、台账属于平台，编排属于对方
  • 三个口子对应三种所有权：inject 写入、memory 读写、usage 查账；不提供「帮我跑图」这种越界接口
  • 所有接口都不接受 userId 参数，身份一律从令牌 sub 取——这条破例一次权限模型就塌了
  • 返回粒度按最小必要：usage 只给汇总不给明细，memory 按 query 限条数而不是全量倒出
  • 每个写接口都带 externalId 并由唯一约束兜底，重复返回 200
  • 限流按每用户而不是每调用方；注入的消息要打来源标记防止两个服务互相回环

D20 Scheduled Jobs and Proactive Outreach: Time Zones, Quiet Hours, Daily Caps, a Notification Provider Abstraction

• Building a scheduled push service for users worldwide, what timezone pitfalls would you hit, and how do you handle the DST switchover day?做一个面向全球用户的定时推送服务，时区上你会踩到哪些坑？夏令时切换那天怎么处理？

  Common in ChinaCommon overseasDeep dive#timezone#scheduling#correctness

  How to reason about it · think before answering

  1. All the signal in this question lives in the DST half. Store UTC, render local is the passing grade; giving a verifiable ruling for the switchover day is what separates knowing the pitfall from having fixed it.
  2. Nail the two basics first: always store UTC (an absolute instant) and convert to the user's zone before any judgement (a wall-clock time). The self-check is one sentence — can this column plus the user's stored timezone uniquely reconstruct the absolute instant? A local time string cannot.
  3. The second pitfall is the timezone field itself: store the IANA identifier (Asia/Shanghai), never a UTC offset. Offsets shift twice a year under DST; the identifier is the rule and the offset is only what that rule evaluated to on one particular day, so it is stale the moment you persist it.
  4. The third is the two anomalies on switchover day: spring-forward makes some local time simply not exist (02:30 on 2026-03-08 in New York), and fall-back makes some local time occur twice (01:30 on 2026-11-01). If your schedule point lands in either window, send at 8am local has no unique answer. State the ruling explicitly rather than leaving it to whatever the library picks: shift a nonexistent time forward past the transition (02:30 becomes 03:30), and take the first occurrence when it happens twice — which is exactly what java.time's ZonedDateTime.of does, so you can assert on it in tests.
  5. The fourth is the one people miss: when a user travels across zones, which timezone counts. Answer: the explicit field on the user profile, never silent drift from device reports; a device report should only prompt the user to confirm a change. Go one level deeper if you can — when the zone jumps more than three hours within 24 hours, treat that day's quiet hours as the union of the old and new zones and stay silent if either is quiet. Being conservative costs a few hours of delay; being aggressive costs a 3am buzz.
  6. Expect: why are these bugs so hard to catch? Because your laptop, CI and production are often all in one zone, frequently UTC, so forgot to convert stays green everywhere. Give the fix: pin the test users to three distinct zones, none equal to the server's, and every server-timezone dependency turns red immediately.

  分析过程 · 先想清楚再作答

  1. 这题的区分度全在夏令时那半句。只答「存 UTC、展示转本地」是及格线，能不能给出夏令时那天的**可验证裁定**决定了你是「知道有坑」还是「填过坑」。
  2. 先把基础两条说死：时间一律存 UTC（存的是绝对时刻），判断前先转成用户本地时区（判断的是墙上时间）。判据是一句可自查的话——只靠这一列加上用户档案里的时区，能不能唯一还原出那个绝对时刻。存本地时间字符串答不上来。
  3. 第二个坑是时区字段本身：必须存 IANA 标识（Asia/Shanghai）而不是 UTC 偏移量。偏移量一年会随夏令时变两次，标识是规则、偏移量只是规则在某一天算出来的结果，存结果的那一刻它就过期了。
  4. 第三个坑是夏令时那天的两种反常：春季前跳会让某个本地时间**根本不存在**（纽约 2026-03-08 的 02:30），秋季回拨会让某个本地时间**出现两次**（2026-11-01 的 01:30）。只要你的调度点落在这两个窗口里，「每天早上 8 点发」就解释不出唯一答案。裁定要显式给出而不是交给库随便选：不存在就顺延到过渡之后（02:30 变 03:30），出现两次就取第一次——这也正是 java.time 的 ZonedDateTime.of 的默认行为，可以直接写成断言测试。
  5. 第四个坑最容易被漏：用户跨时区旅行时，他的时区以哪一次为准。答案是以用户档案里那个显式字段为准、绝不跟着设备静默漂移；设备上报只用来询问是否切换。更细一层可以补：时区在 24 小时内跳变超过 3 小时时，当天的安静时段按新旧两个时区的并集处理，任何一边在安静就不发——保守的代价是晚几小时收到，激进的代价是在人家凌晨三点响一声。
  6. 可以预期的追问：这种 bug 为什么很难被测出来？因为本机、CI、生产常常都在同一个时区甚至都在 UTC，「忘了转时区」在所有测试里都是绿的。给出判据：把测试用户的时区故意设成三个互不相同、且都不等于服务器时区的值，任何依赖服务器时区的判断当场变红。

  Key points

  • Store UTC everywhere, convert to the user's zone before judging; the check is whether column plus zone reconstructs the instant
  • Persist IANA identifiers, not UTC offsets — offsets change twice a year and are stale on write
  • Two DST anomalies: a local time that does not exist (spring forward) and one that occurs twice (fall back)
  • Make the ruling explicit and assertable: shift nonexistent times past the transition, take the first of a duplicated pair (matching java.time)
  • For travellers, trust the explicit profile field, not device drift; on jumps over three hours, treat quiet hours as the union of both zones
  • Pin test users to three zones different from the server's, or the missing conversion stays green in every test

  答题要点

  • 存储一律 UTC，判断前转用户本地时区；自查判据是这一列加时区能否唯一还原绝对时刻
  • 时区存 IANA 标识而不是 UTC 偏移量——偏移量随夏令时一年变两次，存下来就过期
  • 夏令时两种反常：本地时间不存在（春季前跳）、本地时间出现两次（秋季回拨）
  • 裁定要显式且可断言：不存在就顺延到过渡之后，出现两次取第一次（与 java.time 默认一致）
  • 跨时区旅行以用户档案里的显式字段为准，不跟设备漂；跳变超过 3 小时时按新旧时区的并集判安静
  • 测试里把用户时区设成三个不同于服务器的值，否则「忘了转时区」在所有测试里都是绿的

D21 Evaluation and Observability: a Golden Set, LLM-as-Judge, Tracing, a Failure-Rate/Cost Dashboard; Pi vs. LangGraph Summary; Week Three Retrospective

• What makes LLM-as-judge unreliable, and what do you do about it?用大模型给大模型的输出打分（LLM-as-judge），有哪些不可靠的地方？怎么办？

  Common in ChinaCommon overseasDeep dive#evaluation#llm-as-judge#reliability

  How to reason about it · think before answering

  1. This screens for whether you have actually used it. People who have can name specific failure shapes with magnitudes; people who have not just say it might be inaccurate.
  2. First, self-preference: when the judge and the evaluated agent share a model, it favours its own output — the same model has a consistent notion of what a good answer looks like, so asking it to review what it just wrote gets an approving verdict. Measured: on the same batch of deliberately degraded outputs, a same-model judge gave 14/15 while a different model gave 12/15, and the extra passes were exactly the borderline cases worth catching. This is not confined to judges — every model-grading-model position has it, and a Critic node is the same problem.
  3. Second, length bias: judges reward longer answers. Measured: padding a correct 33-character reply with 141 characters of irrelevant pleasantries moved an impression-based rubric from 2 to 4 without changing a word of substance.
  4. Third, rubric drift: scores shift wholesale when the judge prompt is tweaked. The same output scored 2 under one rubric and 5 under another. Hence the hard rule: scores are comparable only within one judge prompt, and cross-version comparison is meaningless.
  5. Match each remedy to its failure rather than saying run it a few more times. Freeze and version the judge prompt — every score record carries its rubric version and judge model, which are its coordinates, and a dashboard that finds two rubrics mixed should refuse to aggregate rather than emit a meaningless average. Default to a different model as judge, as a default and not an option. Keep a small human-labelled calibration set and re-run it whenever the rubric changes, comparing verdicts (pass or fail) rather than score deltas — one point of drift is fine, a flipped verdict is an incident.
  6. And one deeper fix: replace impressionistic criteria with a checkable list, which also dissolves length bias — counting items off a list gives padding nothing to earn. Measured, that padded reply scored 5 both before and after under the checklist rubric.
  7. Expect: is a judge cheaper than humans? The judge's cost is the same order as the system being evaluated, so what a full evaluation run costs decides whether you run it per commit or nightly. Human cost is not money but latency — it cannot give you feedback at the speed of one prompt edit, which is why humans belong on the calibration set only.

  分析过程 · 先想清楚再作答

  1. 这题筛的是「你是真用过，还是听说过」。用过的人能报出具体的失效形态和量级，没用过的人只会说「可能不准」。
  2. 第一种，**同源偏差**：judge 和被评估的 Agent 用同一个模型时，它偏向认可自己的输出——同一个模型对「什么算好答案」的偏好是一致的，让它复核自己刚写的东西，它当然觉得没问题。实测数量级：同一批被改坏的产出，同源 judge 给 14/15，换个模型只给 12/15，被多放过去的正是最该抓的边缘产出。这个坑不止在 judge，**凡是「模型评模型」的位置都有**，Critic 节点是同一个问题。
  3. 第二种，**长度偏好**：judge 倾向给篇幅大的答案更高分。实测：一条 33 字的正确回复灌上 141 字无关客套话，凭印象打分的提示词就从 2 分涨到 4 分，内容一个字没变。
  4. 第三种，**评分提示词漂移**：judge 的评分随提示词微调整体移动。同一份产出，两套评分提示词一套给 2 分一套给 5 分。所以有条硬纪律——**分数只在同一套 judge 提示词内部可比**，跨版本比较是没有意义的。
  5. 解药要一一对应，别笼统说「多测几次」：固定 judge 提示词并版本化（每条评分记录带上 rubric 版本与 judge 模型，那是它的坐标；面板发现混了两套口径应当直接拒绝聚合，而不是算出一个没含义的平均分）；默认用不同的模型当 judge，而且这该是默认值不是可选项；留一小批人工标注做校准集，每次改评分提示词拿它对一遍，**比的是结论（过或不过）而不是分数差**——差 1 分无所谓，结论翻了就是事故。
  6. 还有一条更根本的：**把评分标准从主观印象换成可核对的清单**，它同时解掉长度偏好——照清单逐条数，灌水加不了分。实测那条灌水回复在清单口径下前后都是 5 分，纹丝不动。
  7. 可以预期的追问：judge 便宜还是人工便宜？答：judge 的成本和被评估的系统本身一个量级，所以「跑一次全量评估多少钱」是你决定每次提交都跑还是每天跑一次的依据；而人工的成本不在钱在延迟——它给不了你改一次提示词就想看一次结果的反馈速度，所以人工只该用在校准集上。

  Key points

  • Self-preference: a same-model judge inflates scores (14/15 vs 12/15 cross-model), and it applies to every model-grading-model spot including Critic
  • Length bias: 141 characters of padding moved an impression score from 2 to 4 with no substantive change
  • Rubric drift: the same output scored 2 and 5 under two rubrics, so scores compare only within one judge prompt
  • Remedies map one-to-one: version the rubric and store it alongside each record, refuse to aggregate mixed rubrics, default to a different judge model
  • Keep a human-labelled calibration set and compare verdicts, not score deltas — a point of drift is fine, a flipped verdict is an incident
  • The deeper fix is a checkable list instead of impressions, which also removes length bias (the padded reply scored 5 both ways)

  答题要点

  • 同源偏差：judge 与被评估 Agent 同模型会虚高（实测 14/15 vs 异源 12/15），且凡「模型评模型」的位置都有，Critic 同理
  • 长度偏好：灌水 141 字能让印象分从 2 涨到 4，内容一字未变
  • 评分提示词漂移：同一产出两套 rubric 一个 2 分一个 5 分，所以分数只在同一套提示词内部可比
  • 解药一一对应：rubric 版本化并随记录存坐标、面板发现混口径直接拒绝聚合、默认换模型当 judge
  • 留人工标注校准集，比结论（过/不过）而不是比分数差——差 1 分无所谓，结论翻了是事故
  • 更根本的是把主观印象换成可核对的清单，同时解掉长度偏好（清单口径下灌水前后都是 5 分）
• System design: a multi-agent support platform is live, the team edits prompts several times a week, nobody can say whether quality is improving, and cost is only known as a month-end total. Design its evaluation and observability system.系统设计：一个多 Agent 客服平台已经上线，团队每周改几次提示词，但没人说得清质量是变好还是变差，成本也只有一个月底的总数。请为它设计一套评估与可观测体系。

  Common in ChinaCommon overseasDeep dive#system-design#evaluation#observability#cost

  How to reason about it · think before answering

  1. Do not draw an architecture diagram yet. The trap is that this sounds like build monitoring, so many candidates open with Prometheus and Grafana — that answers infrastructure, not this question. Spend three to five minutes on four things: how often prompts change and how they ship (weekly cadence, canary, rollback); how problems surface today (user complaints, or someone happening to notice); what history exists (how long conversations are retained, whether they can be replayed); and who consumes this (engineers debugging, or an executive watching spend). All four materially change the design, so asking them scores.
  2. Then the trunk, in one sentence: one dataset, two readings. Instrument once, as spans; read across for a single request's call tree (debugging) and stack them for a dashboard (trends and cost). This is the foundation — two data sources will eventually disagree and then nobody trusts either. Many candidates fork here into a monitoring system and an evaluation system, which is the source of every later problem.
  3. Then three layers. Layer one, offline regression: a small stable golden set (15 to 50), covering three things — every route exercised, one item per failure mode (low-confidence fallback, tool budget exhaustion, downstream outage), and the cases behind real past incidents. Each item declares its expected route and a checklist of required facts. The maintenance rule is add, never edit: changing an expectation voids all historical scores. Score with an LLM-as-judge using a different model, and version the rubric, storing that version on every record. This layer runs in CI on every prompt change and emits a number comparable to last time.
  4. Layer two, online observability: every request writes a span tree recording the routing rationale (a model decision, lost forever if not captured), per-node tokens and latency, and degradation and fallback events. The dashboard answers four questions: how much is wrong, where it is slow, what it cost, and which role spent it — that last one is multi-agent specific and the most actionable.
  5. Layer three, online sampled evaluation: fifteen offline cases cannot cover the real traffic distribution, so sample a fraction of live requests (say 1%) through the same judge to get a true quality curve. This layer bridges the other two: offline tells you whether you broke something known, online tells you what real users encountered.
  6. Bring numbers on cost, which is what separates levels. A multi-agent request can produce five to ten model calls, so per-call price is an order of magnitude below the real unit cost and you must price per request. Give the arithmetic: 10k DAU at three sessions each and five calls per session is 150k calls a day; at 2000 input and 500 output tokens, $0.15 and $0.60 per million, that is roughly $90 a day. That number implies two things: per-node attribution shows where to optimise, and evaluation's own cost must be tracked separately, since judge calls are the same order as the system itself and decide whether you evaluate per commit or nightly.
  7. Close on adoption, which many candidates omit: wire evaluation into the release process (block a deploy when pass rate drops below threshold), keep the rubric and golden set in the repository under code review, and pair every mechanism with a failure mode — judges favour same-family models, golden sets get gamed (someone tunes prompts to make it green, and at that moment it is worthless), sampling misses the long tail. A proposal with no stated failure modes reads as book knowledge.
  8. Expect, by frequency: which model judges (one tier above the system under test, and necessarily a different family); where the golden set comes from (start with human-labelled production samples, then append every incident); what happens when this system itself misbehaves (the dashboard refuses to aggregate mixed rubric versions rather than emitting a meaningless average); and how long to build (layer two in a week, layer one in two, layer three in a month since it depends on both).

  分析过程 · 先想清楚再作答

  1. 先别画架构图。这道题的陷阱是它听起来像「搭一套监控」，于是很多人上来就报 Prometheus 加 Grafana——那答的是基础设施，不是这道题。花三到五分钟问清四件事：一是**改提示词的频率和发布方式**（每周几次、有没有灰度、能不能回滚）；二是**现在出问题是怎么发现的**（用户投诉？还是有人偶然看到？）；三是**有没有历史数据**（线上对话存了多久、能不能回放）；四是**谁来看这套东西**（工程师排障，还是老板看成本）。这四个答案会实质改变设计，问它们本身就是分数。
  2. 然后给主干，一句话定形状：**一份数据、两种读法。** 埋点只做一套（span），横着读是一次请求的调用树（排障用），竖着堆是面板（趋势和成本用）。**这条是地基**——两套数据来源迟早对不上，然后没有人相信任何一个。很多候选人在这里就分叉成「监控系统」和「评估系统」两套，那是后面所有麻烦的源头。
  3. 接着按三层展开。**第一层，离线回归**：建一个小而稳的 golden set（15 到 50 条），三层覆盖——每条路由都有人走、每种失败模式各一条（置信度不足落兜底、工具预算耗尽降级、下游挂掉）、以及历史上真出过事故的那几条。每条写清期望路由和必备信息清单。维护规矩是**只增不改**：改一条期望，历史分数全部作废。用 LLM-as-judge 对照清单打分，**judge 换一个模型、rubric 版本化并随每条记录存下来**。这一层挂在 CI 上，每次改提示词跑一遍，产出一个能和上次比的数字。
  4. **第二层，在线观测**：每次请求落一棵 span 树，必须记路由理由（模型做的决策，当时不记就永远丢了）、每个节点的 token 与耗时、以及降级和兜底事件。面板回答四个问题：错了多少、慢在哪、花了多少、**钱花在哪个角色身上**。最后一个是多 Agent 特有的，也最有用。
  5. **第三层，在线采样评估**：离线的 15 条覆盖不了真实流量分布，所以按比例采样线上请求（比如 1%）跑同一套 judge，得到一条真实质量曲线。**这一层是前两层的桥**：离线告诉你有没有改坏已知的东西，在线告诉你真实用户遇到了什么。
  6. 成本这块要给数字感，这是区分层级的地方。**多 Agent 一次用户请求可能产生 5 到 10 次模型调用**，所以「每次调用多少钱」比真实单价小一个数量级，**必须按请求算钱**。给个算式：日活一万、人均三次会话、每次 5 次调用就是 15 万次调用；按输入 2000 输出 500 token、$0.15/$0.60 每百万算，一天约 90 美元。这个数立刻推出两件事：按节点分摊能定位省钱的地方，以及**评估本身的成本要单独记**——judge 调用和被评估系统一个量级，它决定你每次提交都跑还是每天跑一次。
  7. 最后收在「怎么让它真的被用起来」，这是很多人漏的一层：把评估结果接进发布流程（通过率跌破阈值就挡住发布）、把 rubric 和 golden set 放进代码仓库走 code review、以及**给每个机制配一句失效模式**——judge 会偏向同源模型、golden set 会被针对性优化（有人为了让它绿而调提示词，那一刻它就失去了意义）、采样会漏掉长尾。说不出失效模式的方案，面试官会认为你只是读过。
  8. 可以预期的追问，按频率排：judge 用什么模型（比被评估的强一档，且必须异源）；golden set 从哪来（先从线上捞一批人工标注，再逐次把事故补进去）；这套东西自己出问题怎么办（面板发现 rubric 混版直接拒绝聚合，而不是给一个没含义的平均分）；多久能上线（第二层一周、第一层两周、第三层一个月，因为它依赖前两层）。

  Key points

  • Spend three to five minutes clarifying four things: prompt change cadence and release process, how problems surface today, what replayable history exists, and who the audience is
  • The trunk is one dataset, two readings: instrument once as spans, read across for a call tree and stack for a dashboard; two sources will disagree
  • Layer one, offline regression: a small stable golden set covering every route, every failure mode and past incidents, add-never-edit, wired into CI
  • Layer two, online observability: span trees recording routing rationale, per-node tokens and latency, degradation events; the dashboard answers wrong/slow/cost/which-role
  • Layer three, sampled online evaluation through the same judge, covering the real distribution the offline set cannot
  • Price per request, not per call: five to ten calls per request, with arithmetic showing ~$90/day at 10k DAU; track evaluation's own cost separately
  • Close on adoption: block releases when pass rate drops, keep rubric and golden set in the repo under review
  • Pair every mechanism with a failure mode: judge self-preference, golden set gaming, sampling missing the tail — omitting these reads as book knowledge

  答题要点

  • 先用三到五分钟问清四件事：改提示词的频率与发布方式、现在问题怎么被发现、有无历史数据可回放、这套东西给谁看
  • 主干是「一份数据、两种读法」：埋点只做一套 span，横着读是调用树、竖着堆是面板；两套数据源迟早对不上
  • 第一层离线回归：小而稳的 golden set，三层覆盖（每条路由、每种失败模式、历史事故），只增不改，挂 CI
  • 第二层在线观测：span 树记路由理由、每节点 token 与耗时、降级兜底事件；面板回答错了多少/慢在哪/花了多少/钱花在哪个角色
  • 第三层在线采样评估：按比例采样线上请求跑同一套 judge，补上离线覆盖不到的真实分布
  • 成本必须按请求算而非按调用：一次请求 5 到 10 次调用，给出日活一万约 90 美元一天的算式；评估自身成本单独记
  • 收在落地：通过率跌破阈值挡发布、rubric 与 golden set 进仓库走 review
  • 每个机制配失效模式：judge 偏向同源、golden set 会被针对性优化、采样漏长尾——说不出失效模式等于只是读过