面试题库

30 天从前端工程师到 Agent 工程师

D8 为什么 Gateway/Worker 分离；Postgres 表设计（sessions/runs/messages）+ Drizzle

• 消息总线是至少一次投递，同一条消息被重复投递时，怎么保证不会产生两条 run？With at-least-once delivery, how do you guarantee a redelivered message does not create two runs?

  国内高频海外高频深入#idempotency#database#reliability

  分析过程 · 先想清楚再作答

  1. 这题在考幂等的落点在哪一层。凡是答「在代码里先查一下有没有，没有再插入」的，基本当场结束——因为那正是这题想筛掉的答案。
  2. 先把前提摊开：重复不是意外。总线是至少一次语义、客户端会超时重发、用户会手抖双击，同一句话到达两次是必然事件。所以设计目标不是「避免重复到达」，而是「重复到达时结果相同」。
  3. 然后给推导：幂等需要一个由请求内容决定的键。随机 UUID 每次都不同，等于没有幂等；正确取法是把会话 id、客户端消息 id、消息内容拼起来做哈希，客户端没有消息 id 时退用内容加一个粗粒度时间窗。
  4. 结论落在存储层：在 runs 表的这一列上加唯一约束，插入写成「冲突就什么都不做」，返回零行时回查那条已有的 run，把同一个 runId 返回给用户。两次请求、一条 run、一个 runId。
  5. 解释为什么「先查后插」不行，这是本题的分水岭：两个 Gateway 实例可以同时查、同时发现没有、同时插入，这两步之间有一个应用层拦不住的时间窗；它窄到压测复现不出来，上线后每天漏几条。**幂等的最终裁判必须是数据库的唯一约束**，应用层的判断只是为了少一次插入尝试。
  6. 可以预期的追问：那消费侧的重复执行呢？答：唯一约束保证了只有一条 run，但 Worker 可能重复拿到同一条 run，所以状态迁移也要带条件更新（只有当前状态是 pending 时才能改成 running），并且用一个显式的迁移白名单挡住「已完成的 run 被推回运行中」这种会覆盖用户已收到回复的情况。

  How to reason about it · think before answering

  1. This question is about which layer idempotency lives in. Anyone who answers 'check whether it exists, then insert' has usually just failed it — that is exactly the answer being screened out.
  2. State the premise: duplicates are not accidents. The bus is at-least-once, clients retry on timeout, users double-click. The same message arriving twice is certain, so the goal is not to prevent duplicates but to make duplicates produce the same result.
  3. Then derive the key: idempotency needs a key derived from request content. A random UUID differs every time and buys nothing; hash the session id, the client message id and the message body together, falling back to content plus a coarse time bucket when the client has no id.
  4. Land it in storage: put a unique constraint on that column in the runs table, write the insert as on-conflict-do-nothing, and when it returns zero rows read back the existing run and return the same run id. Two requests, one run, one id.
  5. Explain why check-then-insert fails, which is the whole point: two gateway instances can query, both see nothing, and both insert. The window between the two statements cannot be closed in application code, it is too narrow to reproduce under load tests, and it leaks a few bad rows every day in production. The database's unique constraint has to be the final arbiter; the application-level check only saves a wasted insert.
  6. Expect the follow-up: what about duplicate execution on the consumer side? The unique constraint gives you one run, but a worker can still receive it twice, so status changes need conditional updates (move to running only if the current status is pending) plus an explicit transition whitelist that blocks a finished run from being pushed back to running and overwriting a reply the user already saw.

  答题要点

  • 重复投递是必然事件，设计目标是「重复到达时结果相同」，不是「避免重复」
  • 幂等键必须由请求内容决定：会话 id 加客户端消息 id 加内容做哈希，随机 UUID 等于没有幂等
  • 在 runs 的幂等键列上建唯一约束，插入用「冲突就什么都不做」，零行时回查已有 run 返回同一个 runId
  • 先查后插在并发下必然出双份，两条语句之间的时间窗应用层拦不住，幂等的最终裁判是数据库唯一约束
  • 消费侧还要用条件更新加状态迁移白名单，避免同一条 run 被重复执行或把已完成的回复覆盖掉

  Key points

  • Redelivery is certain, so the goal is identical outcomes on duplicates, not preventing duplicates
  • The idempotency key must be derived from request content — session id plus client message id plus body, hashed; a random UUID buys nothing
  • Put a unique constraint on that column, insert with on-conflict-do-nothing, and read back the existing run when zero rows return
  • Check-then-insert races under concurrency; the window between the statements cannot be closed in application code, so the unique constraint must be the final arbiter
  • On the consumer side add conditional status updates and a transition whitelist so a finished run is never re-run or overwritten

D9 Redis Streams 消息总线：XADD/XREADGROUP/XACK/XAUTOCLAIM、consumer group、毒消息

• 什么是 at-least-once？既然消息会被重复投递，业务上到底要怎么保证幂等？What is at-least-once delivery, and given that messages get redelivered, how do you actually make the business side idempotent?

  国内高频海外高频深入#message-bus#idempotency#reliability

  分析过程 · 先想清楚再作答

  1. 这题是本章最容易被追到底的一道。绝大多数人能说出「至少一次，所以业务要幂等」，然后就没有下文了——面试官等的恰恰是下文：幂等具体落在哪一行代码上。答不出具体落点，前半句就是背的。
  2. 先解释为什么消费不掉这个重复：写业务和销号是两个系统的两次写（比如 Postgres 加 Redis），处理完成到 XACK 之间必然存在一个可以崩溃的窗口，崩在那里消息就会被重投。这个窗口只能变小，不能消失，所以 exactly-once 不是总线给你的语义。
  3. 由此得到一句可以直接说出口的结论：exactly-once 是消费端幂等做出来的**效果**，不是中间件提供的**能力**。Kafka 的事务能在「读 Kafka 写 Kafka」的闭环里做到，一旦下游是数据库或第三方 API 就又退回至少一次。
  4. 然后给具体落点，两道闸门要分清各自挡什么：第一道是 runs 表 idempotency_key 上的唯一约束，配 insert on conflict do nothing，挡的是**客户端重复提交**——冲突时接入层直接返回已有的 runId，连总线都不投第二遍；第二道是 messages 表的 unique(run_id, seq)，同样 on conflict do nothing，挡的是**同一条总线消息被执行两遍**，就算两个消费者真的同时跑完，用户也只会看到一条回复。中间还可以加一道便宜的短路：捞到消息先看 run 是不是已经 done，是就直接补一个 XACK 走人——但那是省钱的优化，正确性靠的是那两个唯一约束。
  5. 接着讲最容易做错的一步：幂等键怎么取。它必须能从「同一个意图」稳定推出来。客户端每次重试都新生成一个 uuid 是最常见的错法，那每次都是新意图，唯一约束一次都命中不了，闸门形同虚设。正确做法是客户端生成一次、重试复用同一个值，服务端兜底可以用「会话 id 加消息内容哈希加秒级时间戳」。
  6. 可以预期的追问：不可逆的副作用怎么办，比如发一次退款？答：把外部调用也变成带幂等键的（大多数支付网关都支持 idempotency key 头），并且先在本地库里落一条「已发起」记录再调用，用同一个键去重；实在不支持的接口就只能靠本地状态机加人工对账，这时要主动说出「这类操作我会把它挪出重试路径」。

  How to reason about it · think before answering

  1. This is the question that gets probed hardest. Most candidates say 'at-least-once, so make the business idempotent' and stop — but the follow-up is exactly what matters: which line of code enforces it.
  2. Explain why the duplicate cannot be removed: committing the business write and acking are two writes to two systems (say Postgres and Redis), so there is always a crash window between finishing the work and XACK. The window can shrink but not disappear, which is why exactly-once is not something the bus gives you.
  3. That yields a sentence worth saying out loud: exactly-once is an effect produced by consumer-side idempotency, not a capability provided by the broker. Kafka transactions achieve it inside a read-Kafka-write-Kafka loop, but the moment the sink is a database or third-party API you are back to at-least-once.
  4. Now name the concrete guards and what each one blocks. First, a unique constraint on runs.idempotency_key with insert ... on conflict do nothing, which blocks duplicate submissions: on conflict the gateway returns the existing run id and never publishes a second bus message. Second, unique(run_id, seq) on the messages table, also on conflict do nothing, which blocks duplicate execution: even if two consumers finish the same run simultaneously the user sees one reply. A cheap short-circuit can sit in between — read the run first and just re-ack if it is already done — but that saves money; correctness comes from the two constraints.
  5. Then the step people get wrong: deriving the key. It must be reproducible from the same intent. Generating a fresh uuid on every retry is the classic mistake, because every retry becomes a new intent and the constraint never fires. The client should mint the key once and reuse it across retries; a server-side fallback can hash session id plus message body plus a second-resolution timestamp.
  6. Expect: what about irreversible side effects such as issuing a refund? Push the idempotency key into the external call (most payment gateways accept an idempotency key header), and record an 'initiated' row locally before calling so the same key deduplicates. For APIs with no such support, fall back to a local state machine plus reconciliation, and say plainly that you would move such operations off the automatic retry path.

  答题要点

  • at-least-once：消息至少被处理一次、可能多次，因为业务提交和 XACK 是两个系统的两次写，中间的崩溃窗口消不掉
  • exactly-once 是消费端幂等做出来的效果，不是中间件的能力；下游只要是数据库或第三方 API 就退回至少一次
  • 闸门一：runs.idempotency_key 唯一约束 + on conflict do nothing，挡客户端重复提交，冲突时不再投递总线消息
  • 闸门二：messages 表 unique(run_id, seq) + on conflict do nothing，挡同一条消息被执行两遍，用户只会看到一条回复
  • 幂等键必须从同一个意图稳定推导，客户端重试要复用同一个值；每次重试新生成 uuid 等于没有幂等
  • 不可逆副作用要把幂等键透传给外部接口，并先落一条本地记录再调用

  Key points

  • At-least-once means a message is processed one or more times, because the business commit and the XACK are two writes to two systems with an unavoidable crash window
  • Exactly-once is an effect of consumer-side idempotency, not a broker feature; any database or third-party sink puts you back at at-least-once
  • Guard one: a unique constraint on runs.idempotency_key with on conflict do nothing blocks duplicate submissions and skips publishing a second bus message
  • Guard two: unique(run_id, seq) on messages with on conflict do nothing blocks duplicate execution, so the user sees exactly one reply
  • The idempotency key must be derivable from the same intent and reused across retries; minting a new uuid per retry defeats the whole mechanism
  • For irreversible side effects, pass the idempotency key through to the external API and record an initiated row locally before calling
• 一条消息反复处理失败怎么办？请设计一个毒消息隔离机制。What do you do with a message that keeps failing? Design a poison-message isolation mechanism.

  国内高频海外高频进阶#message-bus#error-handling#reliability

  分析过程 · 先想清楚再作答

  1. 这题在考你有没有踩过「一条坏消息拖垮整条流」。判断标准很简单：你的回答里有没有出现一个具体的阈值和一个具体的落地位置，没有就是在讲概念。
  2. 先把故障模式说清楚：按 at-least-once 的规矩，失败就不 ack、留在 pending 等重投，于是一条无论谁来都会失败的消息进入死循环——投递、失败、闲置超时、被接手、再失败。它自己永远好不了，还持续占用消费者的处理能力。
  3. 然后给机制，三个动作缺一不可：一、判定依据用 pending 清单自己记的投递次数，不要另建计数表；二、超过阈值（本课固定 3 次）就把消息搬到一条死信流，字段里带上原始消息 id、投递次数和失败原因；三、对原流 XACK，同时把这次执行标成失败并写入错误原因。只搬不 ack，它还躺在 pending 里等着被接手；只 ack 不搬，消息和失败原因一起消失，用户永远停在「正在思考」。
  4. 阈值的取值要给出权衡：定 1 会让一次网络抖动就把本来能成功的消息判死；定 10 会在一条必死的消息上浪费十次执行的钱和时间。3 次配合每次之间的空闲阈值，足够熬过绝大多数瞬时故障。
  5. 还要主动说出一个缺口：Redis Streams 没有原生的指数退避，重投时机由空闲阈值决定。想要退避就得自己把消息重新投递并带上「下次可执行时间」，那已经是在实现延迟队列了——这一条能体现你知道 Streams 的边界在哪。
  6. 可以预期的追问：死信流建完就完了吗？答：不。死信条数必须接进告警，它从 0 变成非 0 通常意味着有一类输入你的代码处理不了，是真 bug 而不是运气差；还要留一个重放入口——把死信里的字段原样投回原流即可，因为幂等键还在，重放不会产生重复执行。见过团队把死信建起来半年没打开过，那等于把故障静音了。

  How to reason about it · think before answering

  1. This question probes whether you have ever watched one bad message stall an entire stream. The test is simple: does your answer contain a concrete threshold and a concrete place where isolation happens? If not, you are talking theory.
  2. Describe the failure mode first: under at-least-once you do not ack on failure, so the message stays pending and gets redelivered. A message that fails for everyone therefore loops forever — delivered, failed, idle timeout, claimed, failed — never recovering while continuously consuming worker capacity.
  3. Then give the mechanism, three actions and all of them required. One, use the delivery count the pending list already tracks rather than building a counter table. Two, past the threshold (three deliveries in this course) move the message to a dead-letter stream carrying the original id, delivery count and failure reason. Three, XACK the original stream and mark the run failed with the error recorded. Moving without acking leaves it pending for another takeover; acking without moving makes both the message and its reason disappear, leaving the user stuck on 'thinking'.
  4. Justify the threshold: one delivery kills messages that a single network blip would have let through; ten wastes ten executions of money and time on a message that can never succeed. Three deliveries, spaced by the idle threshold, survives almost all transient faults.
  5. Volunteer a limitation: Redis Streams has no native exponential backoff — redelivery timing is governed by the idle threshold. Backoff requires republishing the message with a next-eligible timestamp, which means building a delay queue yourself. Naming this shows you know where Streams ends.
  6. Expect: is creating the dead-letter stream the end of it? No. Its depth must be alerted on, since going from zero to non-zero usually means a class of input your code cannot handle — a real bug, not bad luck. Keep a replay path too: republish the stored fields back to the original stream, and because the idempotency key is preserved, replay cannot cause duplicate execution. Teams that build a dead-letter stream and never open it have simply muted their failures.

  答题要点

  • 故障模式：at-least-once 下失败不 ack，一条永远失败的消息会无限重投并持续占用消费者
  • 判定依据用 pending 清单里记的投递次数，不需要另建计数表
  • 阈值固定 3 次：定 1 会误杀瞬时故障，定 10 会在必死消息上浪费十次执行成本
  • 隔离动作三件缺一不可：搬到死信流（带原始 id、投递次数、失败原因）、对原流 XACK、把这次执行标成失败并写入原因
  • Redis Streams 没有原生指数退避，重投时机由空闲阈值决定，要退避得自己实现延迟投递
  • 死信流要接告警并留重放入口；幂等键还在，重放不会导致重复执行

  Key points

  • Failure mode: under at-least-once you do not ack on failure, so an always-failing message is redelivered forever and keeps consuming worker capacity
  • Use the delivery count already tracked in the pending list rather than a separate counter table
  • Fix the threshold at three deliveries: one kills transient failures, ten wastes ten executions on a message that can never succeed
  • Isolation needs all three actions: move to a dead-letter stream with original id, delivery count and reason; XACK the original stream; mark the run failed with the error stored
  • Redis Streams has no native exponential backoff — redelivery timing follows the idle threshold, so backoff means implementing delayed republishing yourself
  • Alert on dead-letter depth and keep a replay path; the idempotency key survives, so replay cannot duplicate execution

D10 分片与租约：userId 哈希→shard、SET NX + TTL + Lua 续约、同用户顺序、handoff

• 两个 worker 同时认为自己持有同一个 shard 的租约（脑裂）会造成什么后果，怎么规避？What happens when two workers both believe they hold the same shard lease (split brain), and how do you mitigate it?

  国内高频海外高频深入#split-brain#fencing-token#reliability

  分析过程 · 先想清楚再作答

  1. 这题的判分点非常明确：答案里有没有出现「单靠 Redis 租约做不到绝对互斥」。说「用了 SET NX 加 TTL 就安全了」的人，会被追问到答不上来。
  2. 先讲脑裂是怎么发生的，而且要举那个最常见的场景——不是进程崩溃，是持有者只卡了 5 秒：一次 full GC、宿主机 CPU 被邻居打满、容器被 cgroup 限流。它醒过来时内存里还写着「我持有 shard 68」，继续处理手上那条消息、继续写库，而 Redis 里的租约早已到期并被别人抢走。再补一层：Redis 主从复制是异步的，切主时可能丢掉最后几毫秒的写入，于是两个 worker 都能 SET NX 成功。
  3. 然后讲后果，而且要落到业务上而不是停在「数据不一致」：同一个用户的两条消息被两个进程并发处理，回复乱序、上下文错乱、messages 表的 unique(run_id, seq) 撞约束导致落库失败；最严重的是有副作用的工具被重排或重复执行——「取消订单」和「改配送日期」顺序反了，结果是取消了一个用户本来想留下的订单。
  4. 关键的认知转折：既然无法在 Redis 一侧排除这条时间线，正确的思路就不是「让脑裂不发生」，而是「让第二个人的写入落不了地」——把冲突的检测与拒绝推到真正产生副作用的那一层。
  5. 三条手段按性价比给出。一是 worker 自己的自杀规则：连续两次续约失败、或距上次成功续约超过 TTL 的三分之二，立刻停止处理并清空持有集合——最便宜，把「我以为我还持有」的窗口从无限压到两个续约周期。二是 fencing token：抢租约时从一个单调递增计数器取号（Redis 的 INCR）写进租约值，之后所有有副作用的操作都带上它，下游只接受不比见过的最大号小的写入，落到数据库上就是一句条件更新；醒过来的前任拿的是旧号，写入直接被拒。三是每次写之前重新校验租约，并把校验与写入放进同一段脚本或同一个事务——这只缩小窗口，不消除。
  6. 可预期的追问：fencing 的局限在哪？答「它需要下游配合」。数据库能做条件更新所以好使，但下游是第三方接口（发短信、扣款）时你没法让对方帮你比号，这时只能退回幂等键，把重复执行变成无害，而不是让它不发生。真要绝对互斥就得换到有共识协议的系统（etcd、ZooKeeper 的会话租约），代价是写入延迟和运维复杂度。

  How to reason about it · think before answering

  1. The scoring criterion here is explicit: does your answer contain the sentence 'a Redis lease alone cannot give absolute mutual exclusion'. Anyone who says SET NX plus a TTL makes it safe gets probed until they run out of answers.
  2. Start with how split brain arises, and use the common case: not a crash, but a holder that merely froze for five seconds — a full GC, a noisy neighbour saturating the host CPU, cgroup throttling. It wakes up still believing it holds shard 68, keeps processing the in-flight message and keeps writing, while the lease expired and was taken. Add the second layer: Redis replication is asynchronous, so a failover can lose the last few milliseconds of writes and let two workers both win SET NX.
  3. Then the consequences, expressed in business terms rather than 'inconsistent data': two messages from one user processed concurrently means out-of-order replies, a corrupted context window, and unique(run_id, seq) violations that silently drop a message. Worst is reordered or duplicated side effects — swap 'cancel the order' with 'move the delivery date' and you cancel an order the user wanted to keep.
  4. The key shift: since you cannot rule out that timeline on the Redis side, the goal is not to prevent split brain but to make the second writer's writes fail — push conflict detection and rejection down to the layer that actually causes side effects.
  5. Give three mitigations by value. First, a self-kill rule in the worker: after two consecutive renewal failures, or when the last success is older than two thirds of the TTL, stop processing and clear the held set — cheapest, and it bounds the 'I think I still hold it' window to two renewal periods. Second, fencing tokens: take a monotonically increasing number (Redis INCR) when acquiring, store it in the lease value, attach it to every side-effecting operation, and have the downstream accept only numbers not lower than the highest it has seen — in a database that is one conditional update. The revived predecessor carries a stale number and is rejected. Third, re-validate the lease immediately before each write inside the same script or transaction, which shrinks the window without closing it.
  6. Expect the follow-up: where does fencing break down? It needs downstream cooperation. Databases do conditional updates, but a third-party endpoint (SMS, payments) will not compare your token, so you fall back to idempotency keys that make duplicate execution harmless rather than impossible. True mutual exclusion means moving to a consensus-backed system such as etcd or ZooKeeper session leases, paying in write latency and operational complexity.

  答题要点

  • 单靠 Redis 租约做不到绝对互斥：持有者被冻结再醒来、以及主从异步复制丢写，这两条时间线排除不掉
  • 后果要落到业务：同用户回复乱序、上下文错乱、唯一约束冲突丢消息，最严重是有副作用的工具被重排或重复执行
  • 思路是「让第二个人的写入落不了地」，把冲突检测推到产生副作用的那一层，而不是指望脑裂不发生
  • 三条手段：worker 自杀规则（续约连续失败就放手）、fencing token（写入时带单调号做条件更新）、写前重新校验租约
  • fencing 需要下游配合；下游是第三方接口时只能退回幂等键，要绝对互斥就得换 etcd / ZooKeeper 这类有共识协议的系统

  Key points

  • A Redis lease alone cannot guarantee mutual exclusion: a frozen holder that revives, and asynchronous replication losing writes on failover, are both unavoidable
  • State consequences in business terms: out-of-order replies, corrupted context, unique-constraint violations dropping messages, and reordered or duplicated side effects
  • The goal is to make the second writer's writes fail — push conflict detection to the side-effecting layer instead of hoping split brain never happens
  • Three mitigations: a worker self-kill rule on repeated renewal failure, fencing tokens enforced as conditional updates, and re-validating the lease immediately before writing
  • Fencing needs downstream cooperation; against third-party endpoints fall back to idempotency keys, and true mutual exclusion means a consensus system like etcd or ZooKeeper

D14 部署运维：compose 多 worker、心跳、健康检查、优雅停机、dev/prod 隔离；W2 复盘

• 什么是优雅停机？为什么直接 kill 进程有风险？请说出具体步骤。What is graceful shutdown, and why is killing a process outright risky? Walk through the steps.

  国内高频海外高频进阶#deployment#reliability#operations

  分析过程 · 先想清楚再作答

  1. 这题考的是「你有没有真的发过版」。答「等任务跑完再退出」只是定义，面试官要的是代价、步骤和顺序。
  2. 先把代价说具体。发版、缩容、机器维护、抢占式实例回收都会先发 SIGTERM、等宽限期、超时 SIGKILL。SIGKILL 拦不住，落到正在跑 Agent 循环的 Worker 身上：这次的 run 永远停在 running，用户界面一直转圈；模型调用的钱已经付了，回复却没落库；没确认的消息要等空闲阈值到了才被别人接手，用户白等一轮。一次发版掐断几十次对话，这就是日常代价。
  3. 然后给三步，强调顺序不能变：第一步拒新——把开关拨过去，消费循环下一轮不再从流里取消息（已经读到手上还没开始的那几条，留在 pending 里由别人接手，比硬扛完一整批更快）；第二步等手头这次执行跑完，但要有上限；第三步主动交还租约、从心跳面板注销，然后退出。
  4. 第二步的上限是加分点：一次卡死的模型调用会让你永远等不到，而宽限期一到照样 SIGKILL。与其被动挨刀，不如自己认输退出——没确认的消息还在 pending 里，别人会接手重做。本课取 20 秒，取法是「一次正常执行的耗时上限」再留余量。
  5. 第三步也是加分点：租约本来靠 TTL 到期自然易主，但那是为进程猝死准备的。计划内下线你明知道自己要走，主动交还能让接手方下一轮扫描就上岗，而不是白等一个 TTL。交还必须带条件——只删还写着自己名字的那把牌子，否则租约已过期、别人刚抢到时，你就把对方的值班牌撕了。
  6. 最后两件配套的事，漏一件前面全白做：宽限期的配置必须大于代码里的等待上限（代码等 20 秒而 compose 默认只等 10 秒，第 10 秒就 SIGKILL，三步只走到一半）；以及信号得真的传到你的进程（启动命令写成包管理器，PID 1 就是包管理器，SIGTERM 未必传得到，停机代码一次都不会执行）。

  How to reason about it · think before answering

  1. This question tests whether you have actually shipped a release. Reciting finish in-flight work before exiting is just the definition; the interviewer wants the cost, the steps, and the ordering.
  2. Make the cost concrete. Deploys, scale-downs, host maintenance and spot reclamation all send SIGTERM, wait a grace period, then SIGKILL. SIGKILL cannot be trapped, and landing it on a worker mid-agent-loop means: the run is stuck in running forever while the user watches a spinner; you already paid for the model call but never persisted the reply; the unacked message waits for the idle threshold before anyone claims it. One deploy cuts off dozens of conversations — that is the everyday cost.
  3. Then give three steps and stress that the order is fixed. One, stop accepting work: flip a flag so the consume loop stops reading from the stream (messages already fetched but not started stay in pending for someone else, which is faster than forcing a whole batch through). Two, wait for the in-flight execution, but with a ceiling. Three, proactively release leases, deregister from the heartbeat dashboard, and exit.
  4. The ceiling in step two earns points: a hung model call means you wait forever, and the grace period will SIGKILL you anyway. Better to concede and exit — the unacked message is still pending and someone will redo it. This course uses 20 seconds, derived from the upper bound of a normal execution plus margin.
  5. Step three also earns points: leases normally change hands via TTL expiry, but that path exists for sudden death. On a planned shutdown you know you are leaving, so releasing proactively lets the successor take over on its next scan instead of waiting out a full TTL. The release must be conditional — delete only the badge that still bears your name, or you will tear down the badge of whoever just claimed it after your lease expired.
  6. Finish with two companions; miss either and the rest is wasted. The configured grace period must exceed the wait ceiling in code (code waits 20s while compose defaults to 10s, so SIGKILL lands at second 10 and your three steps only half-run). And the signal must actually reach your process (if the entrypoint is a package manager, PID 1 is the package manager, SIGTERM may never arrive, and your shutdown code never runs once).

  答题要点

  • 直接 kill 的具体代价：run 永远停在 running、用户界面一直转圈、模型的钱已付但回复没落库、没确认的消息要等空闲阈值才被接手
  • 三步且顺序不能变：拒绝新任务 → 等手头的跑完（有上限）→ 主动交还租约并注销心跳，然后退出
  • 等待必须有上限（本课 20 秒）：卡死的模型调用会让你永远等不到，宽限期一到照样被 SIGKILL，不如自己认输，消息还在 pending 里
  • 主动交还租约让接手方下一轮就上岗，而不是白等一个 TTL；交还必须条件化，只删还写着自己名字的那把
  • 宽限期配置必须大于代码里的等待上限，否则三步只执行到一半（compose 的 stop_grace_period / K8s 的 terminationGracePeriodSeconds）
  • 信号要真传到进程：用 exec 形式直接起业务进程，别让包管理器当 PID 1

  Key points

  • Concrete cost of a hard kill: the run is stuck in running, the user stares at a spinner, the model call is paid for but the reply is unsaved, and the unacked message waits out the idle threshold
  • Three steps in a fixed order: refuse new work, wait for in-flight work with a ceiling, then release leases and deregister before exiting
  • The wait needs a ceiling (20s here): a hung model call never returns and the grace period kills you anyway, so concede — the message is still pending for someone else
  • Releasing leases proactively lets the successor start on its next scan instead of waiting a full TTL; the release must be conditional on still owning it
  • The configured grace period must exceed the in-code wait ceiling, or the three steps only half-run (stop_grace_period / terminationGracePeriodSeconds)
  • Make sure the signal reaches your process: exec the business process directly rather than letting a package manager be PID 1
• 滚动发布时，如何避免正在处理的任务被打断？During a rolling deploy, how do you keep in-flight tasks from being interrupted?

  国内高频海外高频进阶#deployment#reliability#operations

  分析过程 · 先想清楚再作答

  1. 这题是上一题的应用题，区别在于它要求你把编排系统那一侧也讲进来——只讲进程内的三步只答了一半。
  2. 完整骨架是两侧配合：编排系统先摘流量（把就绪探针转红，让负载均衡不再把新请求打过来）、再发 SIGTERM、然后等宽限期；进程在这段时间里把手头的活做完、交还所有权、干净退出。这一句话就是答案的主干，剩下都是细节。
  3. 然后区分两类副本，这是拿分点。Gateway 有入站连接，摘流量对它有意义；Worker 没有任何入站连接，它是自己去总线取活的，所谓「摘流量」对它就是「自己不再取新消息」——也就是停机三步的第一步。**同一个词在两类副本上是两种机制**，能说清这一点说明你理解拉与推的差别。
  4. 接着讲批次与顺序：一次只换一部分副本（compose 里手动分批，K8s 里靠 maxUnavailable / maxSurge），保证任何时刻都有足够的存活副本接得住流量。心跳面板上的版本号字段这时派上用场——你能看到新旧两批各剩几个，而不是盲发。
  5. 还要提一句状态兼容：滚动发布期间新旧代码同时在线，所以数据库迁移必须向后兼容（先加可空列、再双写、最后才删旧列），消息格式也不能一次性改。这是很多人漏掉的一层——进程停得再优雅，新旧版本读不了同一份数据照样出事故。
  6. 可以预期的追问：如果一次执行本来就要跑 5 分钟，宽限期不可能等那么久怎么办？答案不是把宽限期拉到 5 分钟，而是让任务可中断可重入——把长任务切成可保存进度的小步（D11 的 run 状态机和 D9 的 at-least-once 加幂等正好提供了这个基础），被打断的那一步由下一个副本接着做。

  How to reason about it · think before answering

  1. This is the applied version of the previous question, and the difference is that it demands the orchestrator's side too — describing only the in-process steps answers half of it.
  2. The full skeleton is both sides cooperating: the orchestrator first removes traffic (turns readiness red so the load balancer stops sending new requests), then sends SIGTERM, then waits out the grace period; the process uses that window to finish in-flight work, hand back ownership, and exit cleanly. That sentence is the trunk; everything else is detail.
  3. Then distinguish the two kinds of replica, which is where the points are. A gateway has inbound connections, so draining traffic means something for it. A worker has no inbound connections at all — it pulls work from the bus, so draining for it means stop fetching new messages, which is step one of graceful shutdown. The same word is two different mechanisms on the two replica types, and saying so shows you understand pull versus push.
  4. Next, batching and ordering: replace only a subset at a time (manual batches in compose, maxUnavailable / maxSurge in Kubernetes) so enough replicas are always alive to absorb traffic. This is where the version field in the heartbeat payload pays off — you can see how many old and new replicas remain instead of deploying blind.
  5. Also mention state compatibility: during a rolling deploy old and new code run simultaneously, so schema migrations must be backward compatible (add a nullable column, dual-write, drop the old column last) and message formats cannot change in one shot. Many candidates miss this layer — however gracefully processes stop, two versions that cannot read the same data will still cause an incident.
  6. Expect: what if a single execution legitimately takes five minutes and the grace period cannot wait that long? The answer is not to stretch the grace period to five minutes but to make the task interruptible and resumable — break long work into steps that checkpoint progress (the run state machine from D11 plus at-least-once with idempotency from D9 give you exactly this), so the next replica continues the interrupted step.

  答题要点

  • 完整骨架是两侧配合：编排系统先摘流量、再发 SIGTERM、等宽限期；进程在这段时间做完手头的活、交还所有权、干净退出
  • Gateway 和 Worker 的「摘流量」是两种机制：前者靠就绪探针转红让负载均衡停止转发，后者靠自己不再从总线取新消息
  • 分批替换（maxUnavailable / maxSurge 或手动分批），保证任何时刻有足够存活副本；心跳里的版本号让你看到新旧两批各剩几个
  • 新旧代码同时在线，所以数据库迁移必须向后兼容（加可空列 → 双写 → 最后删旧列），消息格式不能一次性改
  • 长任务不该靠拉长宽限期解决，而要做成可中断可重入：切成能保存进度的小步，被打断的那步由下一个副本接着做

  Key points

  • The full skeleton is both sides: orchestrator drains traffic, sends SIGTERM, waits the grace period; the process finishes in-flight work, hands back ownership, exits cleanly
  • Draining means two different things for gateways and workers: readiness turning red versus the worker itself stopping its fetch from the bus
  • Replace in batches (maxUnavailable / maxSurge or manual) so enough replicas stay alive; the version field in heartbeats shows how many old and new remain
  • Old and new code run concurrently, so migrations must be backward compatible (nullable column, dual-write, drop last) and message formats cannot change in one step
  • Long tasks are not solved by a longer grace period but by being interruptible and resumable — checkpointed steps that the next replica can continue