面试题库

30 天从前端工程师到 Agent 工程师

D14 部署运维：compose 多 worker、心跳、健康检查、优雅停机、dev/prod 隔离；W2 复盘

• 系统设计：请设计一个 IM Agent 平台——用户在即时通讯软件里和一个 AI 助手对话，助手能调用工具、记住长期偏好、还能定时主动推送。要求支撑十万日活。System design: design an IM agent platform where users chat with an AI assistant inside a messaging app. The assistant calls tools, remembers long-term preferences, and proactively pushes scheduled messages. Target 100k daily active users.

  国内高频海外高频深入#system-design#distributed-systems#cost#operations

  分析过程 · 先想清楚再作答

  1. 先别画图。系统设计题最常见的死法是听完就开始画框，二十分钟后面试官发现你解的是另一道题。花三到五分钟问清四件事：一是流量形状（十万日活对应多少并发会话、峰谷比多少），二是延迟要求（首字节要多快，是否必须流式），三是工具的性质（只读查询还是有写操作和副作用），四是主动推送的合规边界（能不能在深夜推、每天上限几条）。这四个答案会实质改变架构，问它们本身就是分数。
  2. 然后给主干，一句话先定形状：**接入层无状态、消息总线解耦、Worker 有状态且按用户分片、状态全在数据库**。接着按数据流走一遍：IM 平台的 webhook 打到接入层，接入层只做鉴权、限流、落库、投递四件事，立刻返回 202；执行侧从总线取活、跑 Agent 循环、把输出片段回传；主动推送由一个中心调度器按时间投递进同一条总线。**关键论点是接入层耗时确定、执行层耗时不确定，把它们放在一个进程里意味着一次慢的模型调用会占住一个本该毫秒级返回的连接**——这是整道题的立论基础，要主动说出来。
  3. 再逐个模块给出选择和理由。存储：sessions / runs / messages 三张表，runs 单独存在是因为只有它能回答「这次到底跑完没有」，幂等靠 runs 上的唯一约束而不是先查后插。总线：Redis Streams 的消费组做分摊，语义是至少一次，恰好一次靠消费端幂等做出来；反复失败的消息投递三次后进死信流。顺序：消费组的分配单位是一条消息而业务要求的串行单位是一个用户，所以按 userId 哈希到固定数量分片，每个分片同一时刻只有一个 Worker 持有租约。记忆：pgvector 存 embedding，检索包成一个工具交给模型自己决定要不要查，且不给它身份参数——身份只能来自会话。
  4. 主动推送这一块要单独讲透，因为它是这道题区别于普通聊天服务的地方。中心调度器命中时间点后只投一条消息，执行侧照旧；幂等键锚在「计划触发的那一分钟」，所以调度器崩溃重启后回看重放不会重复推送。合规上要有时区、静默时段、每日上限三道闸，而且这三道闸必须在投递前判断而不是在推送时判断——否则你已经花了模型调用的钱才发现不该推。
  5. 然后主动给出容量和成本的数字感，这是高级候选人的分水岭。十万日活、人均十轮对话是一百万次模型调用；按输入输出各一千 token、每百万 token 输入 0.15 美元输出 0.60 美元估算，一天大约七百五十美元。这个数字立刻推出三件事必须做：token 用量要按调用记账并换算成美元（否则你无法定位是哪个用户或哪个功能在烧钱）、要有分层降级（超预算的用户切便宜模型而不是直接拒绝）、以及上下文长度是主要成本杠杆（所以要压缩历史、控制检索条数）。
  6. 最后收在可运维性上，也就是这一周的落点：多副本部署、心跳发现假死、就绪探针只查自己必需的依赖、优雅停机让发版不掐断对话、dev 与 prod 用键名前缀隔离。**每个机制都要配一句「它失效时会怎样」**——租约会脑裂所以要有自杀规则和护栏令牌、心跳会误判所以面板转红只告警不自动摘流量、停机会超时所以等待要有上限。说不出失效模式的机制，面试官会认为你只是读过。
  7. 可以预期的追问，按出现频率排：单点在哪（调度器无状态可重启，Redis 和 Postgres 靠托管服务的主备）；怎么灰度（新旧 Worker 同时在线，靠消息里的版本字段决定走哪套提示词）；用户在助手回复中途又发一句怎么办（三十秒内的改口合并进同一次执行，而不是并发开两个）；成本再降一半怎么做（缓存高频问答、压缩历史、把简单意图路由到小模型）。

  How to reason about it · think before answering

  1. Do not start drawing. The most common way to fail a design question is to hear the prompt and immediately sketch boxes, only for the interviewer to realise twenty minutes later that you solved a different problem. Spend three to five minutes on four questions: traffic shape (how many concurrent sessions does 100k DAU imply, and what is the peak-to-trough ratio), latency (how fast must first byte be, is streaming required), the nature of the tools (read-only lookups, or writes with side effects), and the compliance boundary on proactive pushes (may you push at night, what is the daily cap). All four change the architecture materially, so asking them is itself worth points.
  2. Then state the trunk in one sentence: stateless ingress, a message bus for decoupling, stateful workers sharded by user, all state in the database. Walk the data flow: the messaging platform's webhook hits ingress, which does only auth, rate limiting, persistence and publish, and returns 202 immediately; the execution side pulls work, runs the agent loop, and streams output fragments back; proactive pushes come from a central scheduler publishing onto the same bus. The load-bearing argument is that ingress latency is bounded while execution latency is not, so putting them in one process means one slow model call occupies a connection that should have returned in milliseconds — say this out loud, it is the premise of the whole answer.
  3. Then justify each module. Storage: sessions, runs and messages, with runs existing separately because only it can answer whether this attempt actually finished; idempotency comes from a unique constraint on runs, not from check-then-insert. Bus: Redis Streams consumer groups for fan-out, at-least-once semantics, with exactly-once manufactured by consumer-side idempotency, and messages that fail three times moved to a dead-letter stream. Ordering: the consumer group's unit of assignment is one message while the business requires serialisation per user, so hash userId into a fixed set of shards and let exactly one worker hold each shard's lease. Memory: embeddings in pgvector, retrieval wrapped as a tool the model chooses to call, with no identity parameter — identity only ever comes from the session.
  4. Treat proactive push as its own section, because it is what separates this from an ordinary chat service. The central scheduler publishes one message on a time match and the execution side is unchanged; the idempotency key is anchored to the scheduled minute, so replaying after a scheduler restart cannot double-send. For compliance you need timezone, quiet hours and a daily cap — and all three must be evaluated before publishing rather than at send time, or you have already paid for the model call before discovering you should not have pushed.
  5. Then volunteer capacity and cost numbers, which is what separates senior candidates. 100k DAU at ten turns each is a million model calls; at roughly a thousand tokens in and out, with input at $0.15 and output at $0.60 per million tokens, that is about $750 a day. That number immediately implies three requirements: meter token usage per call and convert to dollars (otherwise you cannot tell which user or feature is burning money), build tiered degradation (push over-budget users to a cheaper model rather than refusing them), and recognise that context length is the dominant cost lever (so compress history and cap retrieved items).
  6. Land on operability, which is this week's payoff: multiple replicas, heartbeats to surface zombies, readiness probes that only check their own hard dependencies, graceful shutdown so deploys do not cut conversations, and dev/prod isolation via key prefixes. Pair every mechanism with what happens when it fails — leases can split-brain so you need a self-fencing rule and fencing tokens, heartbeats produce false positives so a red dashboard alerts a human rather than auto-draining, shutdown can time out so the wait needs a ceiling. A mechanism without a stated failure mode reads as something you only read about.
  7. Expect, in rough order of frequency: where are the single points (the scheduler is stateless and restartable; Redis and Postgres rely on managed primary/replica); how do you roll out safely (old and new workers coexist and a version field in the message selects the prompt set); what if the user sends another message mid-reply (merge a change of mind within thirty seconds into the same execution rather than running two concurrently); and how would you halve the cost (cache frequent answers, compress history, route simple intents to a smaller model).

  答题要点

  • 先用三到五分钟问清四件事：流量形状、延迟要求、工具是否有副作用、主动推送的合规边界——它们会实质改变架构
  • 主干一句话：接入层无状态、消息总线解耦、Worker 有状态且按用户分片、状态全在数据库；立论是接入层耗时确定而执行层不确定
  • 存储 sessions / runs / messages 三张表，幂等靠 runs 上的唯一约束；总线用 Redis Streams 消费组，至少一次加消费端幂等，三次失败进死信
  • 顺序靠 userId 哈希分片加租约：消费组的分配单位是一条消息，而业务要求的串行单位是一个用户
  • 记忆用 pgvector 并包成工具交给模型自己决定是否检索，不给身份参数——身份只能来自会话
  • 主动推送由中心调度器投递，幂等键锚在计划触发的那一分钟；时区、静默时段、每日上限三道闸必须在投递前判断
  • 给出成本数字感：十万日活人均十轮约一百万次调用、一天约七百五十美元，由此推出计量记账、分层降级、压上下文三件事
  • 收在可运维性：多副本、心跳查假死、就绪探针只查自己的依赖、优雅停机、dev/prod 前缀隔离
  • 每个机制都配一句失效模式：租约会脑裂、心跳会误判、停机会超时——说不出失效模式等于只是读过

  Key points

  • Spend three to five minutes clarifying four things: traffic shape, latency targets, whether tools have side effects, and the compliance boundary on proactive pushes
  • State the trunk in one sentence: stateless ingress, bus for decoupling, stateful workers sharded by user, all state in the database — premised on bounded ingress latency versus unbounded execution latency
  • Storage is sessions/runs/messages with idempotency from a unique constraint on runs; the bus is Redis Streams consumer groups, at-least-once plus consumer idempotency, dead-lettering after three failures
  • Ordering comes from hashing userId into shards plus leases: the consumer group assigns per message while the business serialises per user
  • Memory is pgvector exposed as a tool the model may call, with no identity parameter — identity comes only from the session
  • Proactive push flows through a central scheduler with the idempotency key anchored to the scheduled minute; timezone, quiet hours and daily caps are enforced before publishing
  • Bring numbers: 100k DAU at ten turns is ~1M calls and ~$750/day, which implies metering, tiered degradation, and context length as the main cost lever
  • Land on operability: replicas, heartbeats for zombies, readiness probes scoped to own dependencies, graceful shutdown, dev/prod prefix isolation
  • Pair each mechanism with its failure mode — leases split-brain, heartbeats false-positive, shutdown times out; a mechanism without one reads as book knowledge

D21 评估与可观测：golden set、LLM-as-judge、tracing、失败率/成本面板；Pi vs LangGraph 总结；W3 复盘

• 系统设计：一个多 Agent 客服平台已经上线，团队每周改几次提示词，但没人说得清质量是变好还是变差，成本也只有一个月底的总数。请为它设计一套评估与可观测体系。System design: a multi-agent support platform is live, the team edits prompts several times a week, nobody can say whether quality is improving, and cost is only known as a month-end total. Design its evaluation and observability system.

  国内高频海外高频深入#system-design#evaluation#observability#cost

  分析过程 · 先想清楚再作答

  1. 先别画架构图。这道题的陷阱是它听起来像「搭一套监控」，于是很多人上来就报 Prometheus 加 Grafana——那答的是基础设施，不是这道题。花三到五分钟问清四件事：一是**改提示词的频率和发布方式**（每周几次、有没有灰度、能不能回滚）；二是**现在出问题是怎么发现的**（用户投诉？还是有人偶然看到？）；三是**有没有历史数据**（线上对话存了多久、能不能回放）；四是**谁来看这套东西**（工程师排障，还是老板看成本）。这四个答案会实质改变设计，问它们本身就是分数。
  2. 然后给主干，一句话定形状：**一份数据、两种读法。** 埋点只做一套（span），横着读是一次请求的调用树（排障用），竖着堆是面板（趋势和成本用）。**这条是地基**——两套数据来源迟早对不上，然后没有人相信任何一个。很多候选人在这里就分叉成「监控系统」和「评估系统」两套，那是后面所有麻烦的源头。
  3. 接着按三层展开。**第一层，离线回归**：建一个小而稳的 golden set（15 到 50 条），三层覆盖——每条路由都有人走、每种失败模式各一条（置信度不足落兜底、工具预算耗尽降级、下游挂掉）、以及历史上真出过事故的那几条。每条写清期望路由和必备信息清单。维护规矩是**只增不改**：改一条期望，历史分数全部作废。用 LLM-as-judge 对照清单打分，**judge 换一个模型、rubric 版本化并随每条记录存下来**。这一层挂在 CI 上，每次改提示词跑一遍，产出一个能和上次比的数字。
  4. **第二层，在线观测**：每次请求落一棵 span 树，必须记路由理由（模型做的决策，当时不记就永远丢了）、每个节点的 token 与耗时、以及降级和兜底事件。面板回答四个问题：错了多少、慢在哪、花了多少、**钱花在哪个角色身上**。最后一个是多 Agent 特有的，也最有用。
  5. **第三层，在线采样评估**：离线的 15 条覆盖不了真实流量分布，所以按比例采样线上请求（比如 1%）跑同一套 judge，得到一条真实质量曲线。**这一层是前两层的桥**：离线告诉你有没有改坏已知的东西，在线告诉你真实用户遇到了什么。
  6. 成本这块要给数字感，这是区分层级的地方。**多 Agent 一次用户请求可能产生 5 到 10 次模型调用**，所以「每次调用多少钱」比真实单价小一个数量级，**必须按请求算钱**。给个算式：日活一万、人均三次会话、每次 5 次调用就是 15 万次调用；按输入 2000 输出 500 token、$0.15/$0.60 每百万算，一天约 90 美元。这个数立刻推出两件事：按节点分摊能定位省钱的地方，以及**评估本身的成本要单独记**——judge 调用和被评估系统一个量级，它决定你每次提交都跑还是每天跑一次。
  7. 最后收在「怎么让它真的被用起来」，这是很多人漏的一层：把评估结果接进发布流程（通过率跌破阈值就挡住发布）、把 rubric 和 golden set 放进代码仓库走 code review、以及**给每个机制配一句失效模式**——judge 会偏向同源模型、golden set 会被针对性优化（有人为了让它绿而调提示词，那一刻它就失去了意义）、采样会漏掉长尾。说不出失效模式的方案，面试官会认为你只是读过。
  8. 可以预期的追问，按频率排：judge 用什么模型（比被评估的强一档，且必须异源）；golden set 从哪来（先从线上捞一批人工标注，再逐次把事故补进去）；这套东西自己出问题怎么办（面板发现 rubric 混版直接拒绝聚合，而不是给一个没含义的平均分）；多久能上线（第二层一周、第一层两周、第三层一个月，因为它依赖前两层）。

  How to reason about it · think before answering

  1. Do not draw an architecture diagram yet. The trap is that this sounds like build monitoring, so many candidates open with Prometheus and Grafana — that answers infrastructure, not this question. Spend three to five minutes on four things: how often prompts change and how they ship (weekly cadence, canary, rollback); how problems surface today (user complaints, or someone happening to notice); what history exists (how long conversations are retained, whether they can be replayed); and who consumes this (engineers debugging, or an executive watching spend). All four materially change the design, so asking them scores.
  2. Then the trunk, in one sentence: one dataset, two readings. Instrument once, as spans; read across for a single request's call tree (debugging) and stack them for a dashboard (trends and cost). This is the foundation — two data sources will eventually disagree and then nobody trusts either. Many candidates fork here into a monitoring system and an evaluation system, which is the source of every later problem.
  3. Then three layers. Layer one, offline regression: a small stable golden set (15 to 50), covering three things — every route exercised, one item per failure mode (low-confidence fallback, tool budget exhaustion, downstream outage), and the cases behind real past incidents. Each item declares its expected route and a checklist of required facts. The maintenance rule is add, never edit: changing an expectation voids all historical scores. Score with an LLM-as-judge using a different model, and version the rubric, storing that version on every record. This layer runs in CI on every prompt change and emits a number comparable to last time.
  4. Layer two, online observability: every request writes a span tree recording the routing rationale (a model decision, lost forever if not captured), per-node tokens and latency, and degradation and fallback events. The dashboard answers four questions: how much is wrong, where it is slow, what it cost, and which role spent it — that last one is multi-agent specific and the most actionable.
  5. Layer three, online sampled evaluation: fifteen offline cases cannot cover the real traffic distribution, so sample a fraction of live requests (say 1%) through the same judge to get a true quality curve. This layer bridges the other two: offline tells you whether you broke something known, online tells you what real users encountered.
  6. Bring numbers on cost, which is what separates levels. A multi-agent request can produce five to ten model calls, so per-call price is an order of magnitude below the real unit cost and you must price per request. Give the arithmetic: 10k DAU at three sessions each and five calls per session is 150k calls a day; at 2000 input and 500 output tokens, $0.15 and $0.60 per million, that is roughly $90 a day. That number implies two things: per-node attribution shows where to optimise, and evaluation's own cost must be tracked separately, since judge calls are the same order as the system itself and decide whether you evaluate per commit or nightly.
  7. Close on adoption, which many candidates omit: wire evaluation into the release process (block a deploy when pass rate drops below threshold), keep the rubric and golden set in the repository under code review, and pair every mechanism with a failure mode — judges favour same-family models, golden sets get gamed (someone tunes prompts to make it green, and at that moment it is worthless), sampling misses the long tail. A proposal with no stated failure modes reads as book knowledge.
  8. Expect, by frequency: which model judges (one tier above the system under test, and necessarily a different family); where the golden set comes from (start with human-labelled production samples, then append every incident); what happens when this system itself misbehaves (the dashboard refuses to aggregate mixed rubric versions rather than emitting a meaningless average); and how long to build (layer two in a week, layer one in two, layer three in a month since it depends on both).

  答题要点

  • 先用三到五分钟问清四件事：改提示词的频率与发布方式、现在问题怎么被发现、有无历史数据可回放、这套东西给谁看
  • 主干是「一份数据、两种读法」：埋点只做一套 span，横着读是调用树、竖着堆是面板；两套数据源迟早对不上
  • 第一层离线回归：小而稳的 golden set，三层覆盖（每条路由、每种失败模式、历史事故），只增不改，挂 CI
  • 第二层在线观测：span 树记路由理由、每节点 token 与耗时、降级兜底事件；面板回答错了多少/慢在哪/花了多少/钱花在哪个角色
  • 第三层在线采样评估：按比例采样线上请求跑同一套 judge，补上离线覆盖不到的真实分布
  • 成本必须按请求算而非按调用：一次请求 5 到 10 次调用，给出日活一万约 90 美元一天的算式；评估自身成本单独记
  • 收在落地：通过率跌破阈值挡发布、rubric 与 golden set 进仓库走 review
  • 每个机制配失效模式：judge 偏向同源、golden set 会被针对性优化、采样漏长尾——说不出失效模式等于只是读过

  Key points

  • Spend three to five minutes clarifying four things: prompt change cadence and release process, how problems surface today, what replayable history exists, and who the audience is
  • The trunk is one dataset, two readings: instrument once as spans, read across for a call tree and stack for a dashboard; two sources will disagree
  • Layer one, offline regression: a small stable golden set covering every route, every failure mode and past incidents, add-never-edit, wired into CI
  • Layer two, online observability: span trees recording routing rationale, per-node tokens and latency, degradation events; the dashboard answers wrong/slow/cost/which-role
  • Layer three, sampled online evaluation through the same judge, covering the real distribution the offline set cannot
  • Price per request, not per call: five to ten calls per request, with arithmetic showing ~$90/day at 10k DAU; track evaluation's own cost separately
  • Close on adoption: block releases when pass rate drops, keep rubric and golden set in the repo under review
  • Pair every mechanism with a failure mode: judge self-preference, golden set gaming, sampling missing the tail — omitting these reads as book knowledge

D26 系统设计专题：Agent 平台 / 客服 Agent / 多租户 / 成本控制

• 系统设计：请设计一个电商客服 Agent。它要能查订单和物流、按规则拟退款方案、回答商品与政策问题，并在搞不定时转人工。System design: design an e-commerce customer support agent. It looks up orders and shipments, drafts refunds by policy, answers product and policy questions, and escalates to a human when it cannot resolve the issue.

  国内高频海外高频深入#system-design#customer-support#escalation

  分析过程 · 先想清楚再作答

  1. 先说这题和「设计一个 Agent 平台」的区别，否则你会把它答成一道基础设施题。平台题考的是怎么把执行跑稳，这题考的是**怎么保证不把用户困在机器人里**——面试官心里的评分点在业务出口上，不在消息总线上。所以主线要一开口就钉死：任何一通会话最后只能落到三条出口之一，自助解决、转人工、留工单。
  2. 第一步澄清 5 分钟，问四件事：日活与并发会话数（决定要不要拆执行层）、人工坐席有没有夜班（决定出口三存不存在）、退款是 Agent 直接执行还是只拟方案（决定要不要人工确认档）、知识库有多大且多久更新一次（决定检索是不是本题的重点）。第三个问题尤其关键，它直接决定这道题是不是带副作用。
  3. 第二步估算 3 分钟，现场算：单轮 2000 输入加 500 输出，输入 2000 除以一百万乘 0.15 等于 0.0003 美元，输出 500 除以一百万乘 0.60 也等于 0.0003 美元，一轮约 0.0006 美元；日活 1 万、人均 5 轮就是 5 万轮，一天约 30 美元、一个月约 900 美元，模型费不含机器。并发按峰谷比 3、单轮 6 秒算，峰值在途约 11 次执行，每个 worker 并发 4 就是 3 个副本。报数字之前先报算式，面试官插的那句一定是「这个数怎么来的」。
  4. 第三步草图 8 分钟，四块：接入层只做鉴权、限流、落库、投递并立刻返回；执行层从消息总线取活跑 Agent 循环、片段带序号回传；存储是会话、执行、消息三张表加一张知识库切块表；可观测是 tracing 加成本台账。在这张图上额外标出三条出口的分叉点在哪一个节点上——这是本题独有的一笔，画上去面试官立刻知道你答的是客服而不是通用平台。
  5. 第四步深入 15 分钟，优先讲转人工这一支，因为它是本题的题眼。判据必须量化，四条任一命中就转：连续 2 轮未解决、用户明确要求、涉及金额超过自动执行上限（本课口径 50 元）、情绪词命中。接着讲交接形状——不是把 40 轮原文丢给客服，而是一段结构化摘要：用户诉求一句、已核实事实几条、Agent 已做过的动作、失败原因，附原始对话链接。知识库那一支一句话带过混合检索加重排加引用，重点落在「引用为空时走出口二或三，而不是让模型编一个答案」，这是最容易被追的一句。多轮那一支同样一句话：历史超七成预算触发压缩且切口对齐到一轮开头，用户中途改口则 30 秒内合并进同一次执行。
  6. 第五步权衡 5 分钟，说三件事：转人工的判据宁可偏松，因为误转的代价是一次人工会话，把用户困住的代价是一个流失客户加一条差评，两者不在一个量级；退款只拟方案不直接执行，是拿一次人工点头换掉一整类不可逆事故；以及什么规模会推翻这个设计——坐席团队大到需要技能路由和排队策略时，转人工就不再是一个布尔判断，而是另一套调度系统。
  7. 可以预期的追问，按频率排：用户说「我要投诉」算不算情绪词命中（算，且这一类要单独统计，它是产品问题的信号）；转人工之后 Agent 还要不要继续在旁边听（要，用来生成小结和给坐席提示，但不允许再发言）；怎么防止用户被反复转来转去（同一通会话只允许转一次，第二次直接留工单）；以及知识库更新后旧答案怎么办（回答里带引用编号和版本，出问题能倒查是哪一版说错的）。

  How to reason about it · think before answering

  1. Start by separating this from 'design an agent platform', or you will answer an infrastructure question. The platform question is about running execution reliably; this one is about not trapping users inside a bot. The rubric lives in the business exits, not the message bus. So pin the thesis in your first sentence: every conversation must end in exactly one of three exits — self-served, handed to a human, or filed as a ticket.
  2. Clarify for 5 minutes, asking four things: daily actives and concurrent sessions (does execution need to be split out), whether human agents work nights (does exit three exist), whether the agent executes refunds or only drafts them (do you need an approval tier), and how large the knowledge base is and how often it changes (is retrieval the centre of this problem). The third question matters most: it decides whether this system has irreversible side effects.
  3. Estimate for 3 minutes, out loud: 2000 input plus 500 output per turn, input is 2000 over a million times $0.15 which is $0.0003, output is 500 over a million times $0.60 which is also $0.0003, so about $0.0006 per turn. 10k daily actives at five turns is 50k turns, roughly $30 a day and $900 a month in model spend, machines excluded. For concurrency, a peak factor of 3 and 6 seconds per turn gives about 11 in-flight executions at peak, which is 3 worker replicas at 4 concurrent each. State the arithmetic before the result — the interviewer's next line is always 'where did that number come from'.
  4. Sketch for 8 minutes, four blocks: ingress does auth, rate limiting, persistence and publish, then returns immediately; execution pulls from the bus, runs the agent loop, and streams sequenced fragments back; storage is sessions, runs and messages plus a chunk table for the knowledge base; observability is tracing plus a cost ledger. Then mark on the diagram which node decides between the three exits — that single annotation tells the interviewer you are answering the support question rather than the generic platform one.
  5. Go deep for 15 minutes, starting with escalation because that is the crux. The criteria must be quantified, any one of four triggering a handoff: two consecutive unresolved turns, an explicit user request, an amount above the auto-execution ceiling (50 CNY in our setup), or a sentiment keyword hit. Then describe the handoff payload: not forty turns of raw transcript, but a structured summary — the user's ask in one line, verified facts, actions already taken, and the failure reason, with a link to the full transcript. Cover the knowledge base in one line (hybrid search, rerank, inline citations) and spend the weight on 'when the citation set comes back empty, take exit two or three rather than letting the model invent an answer' — that is the sentence they will push on. Cover multi-turn in one line too: compress once history passes 70% of budget, cut on a turn boundary, and merge a change of mind within 30 seconds into the same execution.
  6. Trade-offs for 5 minutes, three points: bias the escalation threshold toward escalating, because a false handoff costs one human conversation while trapping a user costs a churned customer and a bad review — different orders of magnitude. Drafting refunds instead of executing them trades one human approval for an entire class of irreversible incidents. And name what breaks the design: once the agent team is large enough to need skill-based routing and queueing, escalation stops being a boolean and becomes its own scheduling system.
  7. Expect, in rough order: does 'I want to file a complaint' count as a sentiment hit (yes, and track that class separately — it is a product signal); should the agent keep listening after handoff (yes, to summarise and prompt the human, but not to speak); how do you stop users being bounced repeatedly (allow one handoff per conversation, then file a ticket); and what happens to old answers when the knowledge base changes (cite chunk ids and versions so you can trace which revision was wrong).

  答题要点

  • 主线一句话：任何一通会话只能落到三条出口之一——自助解决、转人工、留工单
  • 澄清必问四件事：并发会话数、人工有没有夜班、退款是执行还是只拟方案、知识库规模与更新频率
  • 估算带算式：单轮约 0.0006 美元，日活 1 万人均 5 轮约 30 美元一天、900 美元一月；峰值并发约 11、3 个 worker 副本
  • 转人工判据必须量化，四条任一命中：连续 2 轮未解决、用户明确要求、金额超自动执行上限、情绪词命中
  • 交接给人工的是结构化摘要（诉求、已核实事实、已做动作、失败原因）加原始对话链接，不是 40 轮原文
  • 知识库检索不到时走出口二或三，绝不让模型自由发挥编答案；回答带引用编号
  • 多轮沿用压缩与 30 秒打断合并，退款只拟方案不直接执行，用一次人工点头换掉一类不可逆事故
  • 权衡：判据宁可偏松，因为误转和困住用户的代价不在一个量级

  Key points

  • Thesis: every conversation ends in exactly one of three exits — self-served, escalated to a human, or filed as a ticket
  • Clarify four things: concurrent sessions, whether humans cover nights, whether refunds are executed or only drafted, and knowledge base size and churn
  • Estimate with arithmetic: about $0.0006 per turn, so 10k DAU at five turns is roughly $30/day and $900/month; peak concurrency about 11, meaning 3 worker replicas
  • Quantify escalation: two consecutive unresolved turns, an explicit request, an amount over the auto-execution ceiling, or a sentiment keyword
  • Hand over a structured summary — ask, verified facts, actions taken, failure reason — plus a transcript link, not forty raw turns
  • When retrieval returns no citations, take exit two or three instead of letting the model improvise; answers carry citation ids
  • Reuse compression and 30-second merge for multi-turn; draft refunds rather than executing them, trading one approval for a class of irreversible incidents
  • Trade-off: bias toward escalating, because a false handoff and a trapped user cost different orders of magnitude

14 天 RAG：从检索到可信回答

D1 为什么要检索：幻觉、知识截止与长上下文的代价，以及一个纯关键词的最小 RAG

• 上下文窗口已经做到上百万 token 了，检索这一步会被淘汰吗？Context windows are now in the millions of tokens. Does that make the retrieval step obsolete?

  国内高频海外高频深入#long-context#cost#system-design

  分析过程 · 先想清楚再作答

  1. 这是一道立场题，容易答成非黑即白。判断你有没有做过的地方在于：会不会区分「技术上能不能塞进去」和「工程上该不该每次都塞」，只谈前者的答案一听就是纸上谈兵。
  2. 先承认对方有道理的部分：窗口变大确实吃掉了检索的一部分场景。几十篇文档、更新不频繁、调用量不大的内部工具，直接全塞是最省事的选择，为它建一套检索系统是过度设计。
  3. 再给三条它吃不掉的理由。第一是成本：材料是按次计费的，同一份材料被问一万次就要付一万次，而检索只付取回的那几段；预填充缓存能缓解但不能消除，缓存也有有效期和命中率。
  4. 第二是规模：企业知识库动辄几十万篇，再大的窗口也塞不下，检索是唯一的入口。第三是归因与权限：答案要指回具体某一段，以及不同的人只能看到自己有权访问的材料——这两件事必须在把材料喂给模型之前完成，窗口再大也不解决。
  5. 还要补一条经验事实：材料变多之后，模型在长上下文里定位关键信息的稳定性会下降，出现「读了但没读到」。所以「全塞」并不总是等于「效果更好」，很多时候少而准反而更好。
  6. 可预期的追问：那检索的形态会不会变？会——窗口变大之后，取回的块可以更大、条数可以更多，重排与压缩的压力变小，检索从「精挑几句」变成「粗筛一批」。趋势是检索的粒度变粗，不是检索消失。

  How to reason about it · think before answering

  1. This is a position question and it is easy to answer as a binary. The signal is whether you separate what fits technically from what is worth paying for on every request.
  2. Concede the valid half first: bigger windows genuinely absorb part of the use case. For an internal tool over a few dozen stable documents with low traffic, stuffing everything in is the right call and building a retrieval stack would be over-engineering.
  3. Then give three reasons it does not absorb the rest. Cost is the first: context is billed per request, so the same corpus is paid for on every one of ten thousand queries, whereas retrieval only pays for the passages it returns. Prompt caching softens this but does not remove it.
  4. Scale is the second: enterprise corpora run to hundreds of thousands of documents and no window holds them. Attribution and access control are the third: pointing an answer at a specific passage, and showing each user only what they are permitted to see, both have to happen before the material reaches the model.
  5. Add the empirical point: as the supplied material grows, models become less reliable at locating the one relevant fact inside it. More context is not automatically better; fewer and more precise passages often win.
  6. Expected follow-up: does retrieval change shape? Yes. Larger windows allow bigger chunks and more of them, which relieves pressure on reranking and compression. Retrieval gets coarser, it does not disappear.

  答题要点

  • 先区分「能不能塞进去」和「该不该每次都塞」，前者是技术问题，后者是成本问题。
  • 小规模、低频、少变的语料确实可以直接全塞，为它建检索系统是过度设计。
  • 检索不会被淘汰的三个理由：按次计费的成本、几十万篇塞不下的规模、必须在喂给模型之前完成的归因与权限过滤。
  • 材料越多，模型定位关键信息的稳定性越差，全塞不等于效果更好。
  • 趋势是检索粒度变粗——块更大、条数更多、重排压力变小，而不是检索消失。

  Key points

  • Separate whether it fits from whether it is worth paying for on every request.
  • Small, stable, low-traffic corpora can legitimately be stuffed whole; building retrieval for them is over-engineering.
  • Three reasons retrieval survives: per-request cost, corpora too large for any window, and attribution plus access control that must happen before the model sees the material.
  • More supplied context reduces the reliability of locating a single fact, so stuffing everything is not automatically better.
  • The trend is coarser retrieval — bigger chunks, more of them, less reranking pressure — not the removal of retrieval.

D14 综合项目与复盘：多租户企业知识库问答，一张 RAG 决策地图与面试专题

• 给你一个五百万文档、要求秒级响应、准确率优先的知识库场景，你会怎么设计这套系统？You are handed a knowledge base of five million documents that must answer in about a second, with accuracy as the top priority. How would you design it?

  国内高频海外高频深入#system-design#scaling#latency-budget

  分析过程 · 先想清楚再作答

  1. 这题的题眼不在「你会用什么技术」，而在「你有没有一套从约束推配置的方法」。开口就报架构图和技术栈的答案会被判成背方案；拿到分的答法是先把约束翻译成数字，再让每个选择被某个数字逼出来。
  2. 先把三个约束量化：五百万文档按一篇四五块估，是两千多万块，单精度 1536 维就是上百 GB，**索引塞不进单机内存**，这一条直接决定了存储选型；秒级响应意味着从收到问题到第一个字的预算大约一秒，而生成本身通常就吃掉七八百毫秒，检索侧只剩两三百毫秒；准确率优先意味着可以拿延迟和钱换指标，但只能换到那两三百毫秒为止。
  3. 然后逐项落地，每一项都挂在上面某个数字上：存储上专用向量库或分区加半精度量化（半精度的召回损失通常落在重跑噪声里，索引却小四成，这是白捡的）；检索保留关键词与向量两路加倒数排名融合，因为精确匹配的文档号、错误码、人名是向量的固定盲区；重排只作用于融合后的前二十条——它买的是排序质量，一次同步往返，秒级预算里放得下一次，放不下两次。
  4. 接着讲两个「不上」的决定，这一段比上面更能显出做过工程：**Agentic 检索不作为默认路径**，它的收益集中在多跳题上而代价摊给全部问题，秒级预算下更是直接超支——正确做法是先用一次便宜的分类把多跳分流出来，只让那一小部分进循环；**上下文块头之类的手法先不上**，因为它对关键词一路是稀释、对向量一路才是补位，方向相反，得在自己的真实 embedding 上测过再说。
  5. 准确率优先必须落成可验收的东西，否则是空话：一份不少于一百题的标准答案集（其中多跳与无答案各占一成以上）、召回率与排序质量分开看、无答案题的拒答率单独一栏、引用由代码回查而不是靠提示词自觉。**报数字时把最难看的那一栏也报出来**，比只报总分可信得多。
  6. 可预期的追问：五百万文档怎么建第一版索引？答案是这笔钱是一次性大额支出，要按批做、可断点续跑，并且从第一天就上基于内容指纹的增量同步——否则每次改配置都等于把整个知识库重买一遍。再追问就谈灰度：新旧两套向量双写在两列上，用同一份标准答案集在两列上各跑一遍再切流量，回滚只是改一个配置项。

  How to reason about it · think before answering

  1. The real subject here is not which technologies you know, it is whether you have a repeatable way to derive a configuration from constraints. Opening with an architecture diagram reads as a memorized answer; the way to score is to turn each constraint into a number first, then let every choice be forced by one of those numbers.
  2. Quantify the three constraints. Five million documents at roughly four or five chunks each is over twenty million chunks; at 1536 float dimensions that is hundreds of gigabytes, so the index does not fit in one machine's memory — that alone settles storage. A one-second budget to first token, with generation typically eating seven or eight hundred milliseconds, leaves only two or three hundred for retrieval. Accuracy first means you may trade latency and money for metrics, but only within that remaining budget.
  3. Now derive each knob from one of those numbers: a dedicated vector store or partitioning, plus half precision (its recall loss usually sits inside run-to-run noise while the index shrinks by about forty percent — essentially free); keep both keyword and vector routes with reciprocal rank fusion, because exact matches on document ids, error codes and names are a permanent blind spot for embeddings; rerank only the top twenty after fusion, since it buys ranking quality at the cost of one synchronous round trip, and a one-second budget affords exactly one.
  4. Then state two things you deliberately do not build, which is the part that reads as field experience. Agentic retrieval is not the default path: its gains concentrate on multi-hop questions while its cost is spread over every question, and it blows a one-second budget outright — the right move is a cheap classifier that routes only the multi-hop minority into the loop. Contextual chunk headers and similar tricks also wait, because they dilute the keyword route while helping the vector route; the directions are opposite, so measure on your own embeddings before committing.
  5. Accuracy first has to become something you can sign off on. That means a golden set of at least a hundred questions with multi-hop and unanswerable each above ten percent, recall and ranking quality read separately, abstention rate on unanswerable questions as its own column, and citations verified by code rather than trusted from the model. Reporting the ugliest column alongside the headline number is far more credible than reporting a single score.
  6. Expected follow-up: how do you build the first index over five million documents? It is a one-off large expense, so batch it, make it resumable, and put content-hash incremental sync in from day one, or every config change means buying the whole corpus again. Push further and you get to rollout: dual-write the new embeddings into a second column, evaluate both columns on the same golden set, then shift traffic, so rollback is a config flip rather than an eight-hour rebuild.

  答题要点

  • 先把约束翻译成数字：两千多万块决定索引塞不进单机内存，一秒预算里检索侧只剩两三百毫秒。
  • 存储用专用库或分区加半精度；检索保留关键词与向量两路加倒数排名融合，重排只作用于前二十条。
  • 明确说出「不上」的两项：Agentic 只对分流出来的多跳开，块头这类方向相反的手法先测再说。
  • 准确率优先要落成一百题以上的标准答案集、拒答率单独一栏、引用由代码回查。
  • 第一版建索引是一次性大额支出：分批可续跑，并从第一天就上增量同步；换模型走双写切列。

  Key points

  • Translate constraints into numbers first: twenty million chunks means the index will not fit one machine, and a one-second budget leaves retrieval two to three hundred milliseconds.
  • Dedicated store or partitions plus half precision; keep keyword and vector routes with RRF, and rerank only the top twenty after fusion.
  • Name the two things you will not ship: agentic only for a routed multi-hop minority, and chunk headers only after measuring on your own embeddings.
  • Turn accuracy-first into a hundred-plus question golden set, abstention rate as its own column, and code-verified citations.
  • First index build is a one-off large expense: batch it, make it resumable, add incremental sync on day one, and dual-write columns for model swaps.