Interview Bank

From Frontend Engineer to Agent Engineer in 30 Days

D14 Deployment and Operations: Multi-Worker Compose, Heartbeats, Health Checks, Graceful Shutdown, Dev/Prod Isolation; Week Two Retrospective

• What is graceful shutdown, and why is killing a process outright risky? Walk through the steps.什么是优雅停机？为什么直接 kill 进程有风险？请说出具体步骤。

  Common in ChinaCommon overseasIntermediate#deployment#reliability#operations

  How to reason about it · think before answering

  1. This question tests whether you have actually shipped a release. Reciting finish in-flight work before exiting is just the definition; the interviewer wants the cost, the steps, and the ordering.
  2. Make the cost concrete. Deploys, scale-downs, host maintenance and spot reclamation all send SIGTERM, wait a grace period, then SIGKILL. SIGKILL cannot be trapped, and landing it on a worker mid-agent-loop means: the run is stuck in running forever while the user watches a spinner; you already paid for the model call but never persisted the reply; the unacked message waits for the idle threshold before anyone claims it. One deploy cuts off dozens of conversations — that is the everyday cost.
  3. Then give three steps and stress that the order is fixed. One, stop accepting work: flip a flag so the consume loop stops reading from the stream (messages already fetched but not started stay in pending for someone else, which is faster than forcing a whole batch through). Two, wait for the in-flight execution, but with a ceiling. Three, proactively release leases, deregister from the heartbeat dashboard, and exit.
  4. The ceiling in step two earns points: a hung model call means you wait forever, and the grace period will SIGKILL you anyway. Better to concede and exit — the unacked message is still pending and someone will redo it. This course uses 20 seconds, derived from the upper bound of a normal execution plus margin.
  5. Step three also earns points: leases normally change hands via TTL expiry, but that path exists for sudden death. On a planned shutdown you know you are leaving, so releasing proactively lets the successor take over on its next scan instead of waiting out a full TTL. The release must be conditional — delete only the badge that still bears your name, or you will tear down the badge of whoever just claimed it after your lease expired.
  6. Finish with two companions; miss either and the rest is wasted. The configured grace period must exceed the wait ceiling in code (code waits 20s while compose defaults to 10s, so SIGKILL lands at second 10 and your three steps only half-run). And the signal must actually reach your process (if the entrypoint is a package manager, PID 1 is the package manager, SIGTERM may never arrive, and your shutdown code never runs once).

  分析过程 · 先想清楚再作答

  1. 这题考的是「你有没有真的发过版」。答「等任务跑完再退出」只是定义，面试官要的是代价、步骤和顺序。
  2. 先把代价说具体。发版、缩容、机器维护、抢占式实例回收都会先发 SIGTERM、等宽限期、超时 SIGKILL。SIGKILL 拦不住，落到正在跑 Agent 循环的 Worker 身上：这次的 run 永远停在 running，用户界面一直转圈；模型调用的钱已经付了，回复却没落库；没确认的消息要等空闲阈值到了才被别人接手，用户白等一轮。一次发版掐断几十次对话，这就是日常代价。
  3. 然后给三步，强调顺序不能变：第一步拒新——把开关拨过去，消费循环下一轮不再从流里取消息（已经读到手上还没开始的那几条，留在 pending 里由别人接手，比硬扛完一整批更快）；第二步等手头这次执行跑完，但要有上限；第三步主动交还租约、从心跳面板注销，然后退出。
  4. 第二步的上限是加分点：一次卡死的模型调用会让你永远等不到，而宽限期一到照样 SIGKILL。与其被动挨刀，不如自己认输退出——没确认的消息还在 pending 里，别人会接手重做。本课取 20 秒，取法是「一次正常执行的耗时上限」再留余量。
  5. 第三步也是加分点：租约本来靠 TTL 到期自然易主，但那是为进程猝死准备的。计划内下线你明知道自己要走，主动交还能让接手方下一轮扫描就上岗，而不是白等一个 TTL。交还必须带条件——只删还写着自己名字的那把牌子，否则租约已过期、别人刚抢到时，你就把对方的值班牌撕了。
  6. 最后两件配套的事，漏一件前面全白做：宽限期的配置必须大于代码里的等待上限（代码等 20 秒而 compose 默认只等 10 秒，第 10 秒就 SIGKILL，三步只走到一半）；以及信号得真的传到你的进程（启动命令写成包管理器，PID 1 就是包管理器，SIGTERM 未必传得到，停机代码一次都不会执行）。

  Key points

  • Concrete cost of a hard kill: the run is stuck in running, the user stares at a spinner, the model call is paid for but the reply is unsaved, and the unacked message waits out the idle threshold
  • Three steps in a fixed order: refuse new work, wait for in-flight work with a ceiling, then release leases and deregister before exiting
  • The wait needs a ceiling (20s here): a hung model call never returns and the grace period kills you anyway, so concede — the message is still pending for someone else
  • Releasing leases proactively lets the successor start on its next scan instead of waiting a full TTL; the release must be conditional on still owning it
  • The configured grace period must exceed the in-code wait ceiling, or the three steps only half-run (stop_grace_period / terminationGracePeriodSeconds)
  • Make sure the signal reaches your process: exec the business process directly rather than letting a package manager be PID 1

  答题要点

  • 直接 kill 的具体代价：run 永远停在 running、用户界面一直转圈、模型的钱已付但回复没落库、没确认的消息要等空闲阈值才被接手
  • 三步且顺序不能变：拒绝新任务 → 等手头的跑完（有上限）→ 主动交还租约并注销心跳，然后退出
  • 等待必须有上限（本课 20 秒）：卡死的模型调用会让你永远等不到，宽限期一到照样被 SIGKILL，不如自己认输，消息还在 pending 里
  • 主动交还租约让接手方下一轮就上岗，而不是白等一个 TTL；交还必须条件化，只删还写着自己名字的那把
  • 宽限期配置必须大于代码里的等待上限，否则三步只执行到一半（compose 的 stop_grace_period / K8s 的 terminationGracePeriodSeconds）
  • 信号要真传到进程：用 exec 形式直接起业务进程，别让包管理器当 PID 1
• During a rolling deploy, how do you keep in-flight tasks from being interrupted?滚动发布时，如何避免正在处理的任务被打断？

  Common in ChinaCommon overseasIntermediate#deployment#reliability#operations

  How to reason about it · think before answering

  1. This is the applied version of the previous question, and the difference is that it demands the orchestrator's side too — describing only the in-process steps answers half of it.
  2. The full skeleton is both sides cooperating: the orchestrator first removes traffic (turns readiness red so the load balancer stops sending new requests), then sends SIGTERM, then waits out the grace period; the process uses that window to finish in-flight work, hand back ownership, and exit cleanly. That sentence is the trunk; everything else is detail.
  3. Then distinguish the two kinds of replica, which is where the points are. A gateway has inbound connections, so draining traffic means something for it. A worker has no inbound connections at all — it pulls work from the bus, so draining for it means stop fetching new messages, which is step one of graceful shutdown. The same word is two different mechanisms on the two replica types, and saying so shows you understand pull versus push.
  4. Next, batching and ordering: replace only a subset at a time (manual batches in compose, maxUnavailable / maxSurge in Kubernetes) so enough replicas are always alive to absorb traffic. This is where the version field in the heartbeat payload pays off — you can see how many old and new replicas remain instead of deploying blind.
  5. Also mention state compatibility: during a rolling deploy old and new code run simultaneously, so schema migrations must be backward compatible (add a nullable column, dual-write, drop the old column last) and message formats cannot change in one shot. Many candidates miss this layer — however gracefully processes stop, two versions that cannot read the same data will still cause an incident.
  6. Expect: what if a single execution legitimately takes five minutes and the grace period cannot wait that long? The answer is not to stretch the grace period to five minutes but to make the task interruptible and resumable — break long work into steps that checkpoint progress (the run state machine from D11 plus at-least-once with idempotency from D9 give you exactly this), so the next replica continues the interrupted step.

  分析过程 · 先想清楚再作答

  1. 这题是上一题的应用题，区别在于它要求你把编排系统那一侧也讲进来——只讲进程内的三步只答了一半。
  2. 完整骨架是两侧配合：编排系统先摘流量（把就绪探针转红，让负载均衡不再把新请求打过来）、再发 SIGTERM、然后等宽限期；进程在这段时间里把手头的活做完、交还所有权、干净退出。这一句话就是答案的主干，剩下都是细节。
  3. 然后区分两类副本，这是拿分点。Gateway 有入站连接，摘流量对它有意义；Worker 没有任何入站连接，它是自己去总线取活的，所谓「摘流量」对它就是「自己不再取新消息」——也就是停机三步的第一步。**同一个词在两类副本上是两种机制**，能说清这一点说明你理解拉与推的差别。
  4. 接着讲批次与顺序：一次只换一部分副本（compose 里手动分批，K8s 里靠 maxUnavailable / maxSurge），保证任何时刻都有足够的存活副本接得住流量。心跳面板上的版本号字段这时派上用场——你能看到新旧两批各剩几个，而不是盲发。
  5. 还要提一句状态兼容：滚动发布期间新旧代码同时在线，所以数据库迁移必须向后兼容（先加可空列、再双写、最后才删旧列），消息格式也不能一次性改。这是很多人漏掉的一层——进程停得再优雅，新旧版本读不了同一份数据照样出事故。
  6. 可以预期的追问：如果一次执行本来就要跑 5 分钟，宽限期不可能等那么久怎么办？答案不是把宽限期拉到 5 分钟，而是让任务可中断可重入——把长任务切成可保存进度的小步（D11 的 run 状态机和 D9 的 at-least-once 加幂等正好提供了这个基础），被打断的那一步由下一个副本接着做。

  Key points

  • The full skeleton is both sides: orchestrator drains traffic, sends SIGTERM, waits the grace period; the process finishes in-flight work, hands back ownership, exits cleanly
  • Draining means two different things for gateways and workers: readiness turning red versus the worker itself stopping its fetch from the bus
  • Replace in batches (maxUnavailable / maxSurge or manual) so enough replicas stay alive; the version field in heartbeats shows how many old and new remain
  • Old and new code run concurrently, so migrations must be backward compatible (nullable column, dual-write, drop last) and message formats cannot change in one step
  • Long tasks are not solved by a longer grace period but by being interruptible and resumable — checkpointed steps that the next replica can continue

  答题要点

  • 完整骨架是两侧配合：编排系统先摘流量、再发 SIGTERM、等宽限期；进程在这段时间做完手头的活、交还所有权、干净退出
  • Gateway 和 Worker 的「摘流量」是两种机制：前者靠就绪探针转红让负载均衡停止转发，后者靠自己不再从总线取新消息
  • 分批替换（maxUnavailable / maxSurge 或手动分批），保证任何时刻有足够存活副本；心跳里的版本号让你看到新旧两批各剩几个
  • 新旧代码同时在线，所以数据库迁移必须向后兼容（加可空列 → 双写 → 最后删旧列），消息格式不能一次性改
  • 长任务不该靠拉长宽限期解决，而要做成可中断可重入：切成能保存进度的小步，被打断的那步由下一个副本接着做

D18 History Fidelity and Summarization, Multimodal Placeholders, Checkpointer Persistence

• What problem does a checkpointer solve in a multi-agent system, and what does it cost?checkpointer 在多 Agent 系统里解决了什么问题？它的代价是什么？

  Common in ChinaCommon overseasIntermediate#checkpointing#cost#operations

  How to reason about it · think before answering

  1. The second half is the real question. Answering it enables recovery and fault tolerance is a feature blurb any doc carries. The interviewer wants to know whether you have done the arithmetic and where it hurts.
  2. Make the value concrete in money and time: one multi-agent run with a review loop costs nine model calls. If the process is restarted for a deploy at call seven, without checkpoints all nine are wasted and the user is still watching a spinner. With them the run continues from the last signed-off point and no completed node re-runs. What you bought is a smaller unit of failure — a node instead of a whole run.
  3. Mention the three things it unlocks that retries alone cannot: human approval gates (pause before a node and the state simply waits), time-travel debugging (go back to just before the bad step and inspect state), and forking for comparison (run two variants from one checkpoint) — which is also the infrastructure evaluation is built on.
  4. Then the costs, all three. First, bigger state means slower writes, and it is written at every step: one request produces six checkpoints, so a byte added to state is six bytes written. Hence attachments hold references, not content, and retrieval results hold document ids, not full text.
  5. Second, the store has limits. With jsonb the hard cap is far away, but a row past roughly two kilobytes gets pushed to out-of-line storage and costs an extra IO on every read and write. The real engineering line is do not let a single checkpoint reach hundreds of kilobytes, not the theoretical cap.
  6. Third, the one people forget: version compatibility. Checkpoints are long-lived data, so every change to the state shape incurs migration debt, and missing fields usually do not throw — they silently yield undefined or NaN. This is why state fields should be reserved early: once a shape is persisted, changing a field is a data migration, not a code edit.
  7. Expect the follow-up: do checkpoints need cleanup? Yes — retention and archival per thread, or the table grows linearly with active users. Also treat it as sensitive data: graph state contains full conversations, so it must be included whenever you delete a user's data.

  分析过程 · 先想清楚再作答

  1. 这题的下半句才是考点。只答「能恢复、能容错」是功能介绍，任何文档都写着；面试官想听的是你有没有算过这笔账，以及知不知道它会在哪里疼。
  2. 先把价值说具体，用钱和时间说：一次带评审回路的多 Agent 执行要调九次模型，跑到第七次进程被换版本重启，没有检查点就是九次全废、用户界面还停在转圈。有检查点则从上一个签字点接着跑，已经跑完的节点一次都不重跑——它买的是「失败的粒度从一整次执行降到一个节点」。
  3. 顺带说清它解锁的另外三件事，这三样单靠重试做不到：**人工审批闸口**（在某个节点前停下等人点确认，状态就停在那儿）、**时间旅行调试**（回到出问题那一步之前看状态长什么样）、**分叉对比**（从同一个检查点跑两种走法，比较结果，这也是评估的基础设施）。
  4. 然后是代价，三笔要说全。第一笔，**状态越大写得越慢**，而且是每一步都写一份——一次请求写六个检查点，状态里多一个字节就要多写六遍。所以附件存引用不存内容，检索结果存文档 id 不存全文。
  5. 第二笔，**存储本身有上限**。用 jsonb 存的话，硬上限很远，但单行超过大约两 KB 就会被挪到外存、每次读写多一次 IO，所以真正的工程线是「别让单个检查点变成几百 KB」，而不是那个理论上限。
  6. 第三笔也是最容易被忽略的：**版本兼容**。检查点是长期存活的数据，你每改一次状态形状就欠下一笔迁移债，而缺字段读出来通常不报错，只是静默给出 undefined 或 NaN。这一条决定了状态字段要尽早占好位子——图状态的形状一旦被持久化，改字段就不是改代码，是数据迁移。
  7. 可以预期的追问：那检查点要不要清理？答要，按会话线设保留期与归档策略，否则这张表会随日活线性膨胀；另外要留意它是敏感数据——图状态里有完整对话，删除用户数据时这张表必须一起处理。

  Key points

  • Core value: it shrinks the unit of failure from a whole run to a single node, so a nine-call run is not wasted by one restart
  • It also unlocks three things retries cannot: human approval gates, time-travel debugging, and forking from one checkpoint to compare variants — the substrate evaluation is built on
  • Cost one: bigger state writes slower, and it is written at every step — hence references for attachments and document ids for retrieval results
  • Cost two: storage limits — a jsonb row past roughly two kilobytes goes out-of-line and costs an extra IO, so the practical line is keeping a checkpoint well under hundreds of kilobytes
  • Cost three: version compatibility — every change to the state shape is migration debt, and missing fields silently yield undefined or NaN, which is why fields should be reserved early
  • Operationally you need retention and archival, and you must treat it as sensitive data: graph state holds full conversations and must be purged with the user's data

  答题要点

  • 核心价值：把失败的粒度从「一整次执行」降到「一个节点」，九次模型调用的执行不会因为一次重启全废
  • 还解锁三件重试做不到的事：人工审批闸口、时间旅行调试、从同一个检查点分叉对比（也是评估的基础设施）
  • 代价一，状态越大写得越慢，而且每一步都写一份——所以附件存引用、检索结果存文档 id
  • 代价二，存储有上限：jsonb 单行超过约两 KB 就外存、多一次 IO，工程线是别让单个检查点到几百 KB
  • 代价三，版本兼容：状态形状改一次就欠一笔迁移债，缺字段静默给出 undefined 或 NaN；所以字段要尽早占位
  • 运维上还要有保留期与归档，并把它当敏感数据处理——图状态里有完整对话，删用户数据时必须一起删