Interview Bank

From Frontend Engineer to Agent Engineer in 30 Days

D21 Evaluation and Observability: a Golden Set, LLM-as-Judge, Tracing, a Failure-Rate/Cost Dashboard; Pi vs. LangGraph Summary; Week Three Retrospective

• How do you evaluate an agent's quality, and how does it differ from testing a conventional backend service?怎么评估一个 Agent 的效果？和传统后端服务的测试有什么不同？

  Common in ChinaCommon overseasIntermediate#evaluation#testing#agent-quality

  How to reason about it · think before answering

  1. The hinge is differ. Answering build a test set and measure accuracy is the textbook ML answer and misses the point; the interviewer wants to know whether you can articulate what makes agents special here.
  2. The root difference is one sentence: the same input does not guarantee the same output. Conventional tests assert equality, but an agent's output has no single correct answer, only good enough. Once the assertion changes from equality to scoring, the whole methodology changes with it.
  3. That difference cascades into three consequences, and covering all three secures the question. First, whether it ran tells you nothing about quality — the flow not throwing does not mean the reply stated the refund conclusion. Second, the blast radius of a change is diffuse: a one-word prompt edit may affect only one class of request, and hand-checking five samples that happen to miss that class yields no impact, then you ship. Third, multi-agent adds a layer: one request passes routing, planning, parallel execution, review and aggregation, and any one of them going wrong surfaces as that last paragraph seems off — without measuring each stage you cannot tell which to fix.
  4. So frame it: evaluation is not testing. Evaluation establishes a comparable baseline for a stochastic system. Its output is not pass or fail but a number you can compare against last time — and to compare, the sample set must be frozen.
  5. Then get concrete: a small stable golden set (15 items here), each declaring its expected route and a checklist of facts the reply must contain; an LLM-as-judge scoring against that checklist; and evaluation results joined to tracing on one dashboard. The checklist is the key move — it converts is this a good answer, which cannot be verified, into were these facts stated, which can.
  6. Expect: do you still need unit tests? Yes, with a clean split — deterministic parts (tool functions, state transitions, reducers) keep asserting equality in unit tests, while evaluation covers only the model-generated segment. Merge the two and you get a suite that fails randomly, after which everyone starts ignoring CI.

  分析过程 · 先想清楚再作答

  1. 题眼是「不同」。只答「建一个测试集跑准确率」拿不到分——那是机器学习的标准答案，面试官想看你能不能说清 Agent 这个场景特殊在哪。
  2. 根子上的差别只有一句：**同样的输入，Agent 不保证给同样的输出**。传统测试的断言是「等于」，而 Agent 的产出没有唯一正确答案，只有「够不够好」。断言从等值变成了判分，整套方法论跟着变。
  3. 这条差别连锁出三个后果，说全了这题就稳了：一是**跑没跑通判断不了质量**——流程没抛错，不等于回复里写明了退款结论；二是**改动的影响是弥散的**，改一个字的提示词可能只影响一类请求，人肉抽查五条恰好没覆盖到，你会得出「没影响」然后上线；三是**多 Agent 又难一层**，一次请求走路由、拆分、并行执行、评审、汇总五道工序，任何一道歪了都表现成「最后那段话不太对」，不分开量就不知道该改哪块。
  4. 所以给出定位：**评估不是测试，评估是给一个随机系统建立一条可比较的基线。** 它的产物不是「通过」或「不通过」，而是一个能和上一次比的数字。既然要比，样本集就必须固定。
  5. 然后落到具体做法：一个小而稳的 golden set（本课 15 条），每条写清期望走哪条路由和一份必备信息清单；用 LLM-as-judge 对照清单打分；把评估结果和链路追踪接到同一块面板上。**清单是关键**——它把「这答得好吗」这种没法验的问题，换成了「这几件事写没写」这种能验的问题。
  6. 可以预期的追问：那还需要单元测试吗？需要，而且分工很清楚——工具函数、状态迁移、reducer 这些确定性的部分照旧用单元测试断言等值，评估只负责模型产出那一段。把两者混成一套，你会得到一堆随机失败的测试，然后所有人开始无视 CI。

  Key points

  • The root difference: identical input does not guarantee identical output, so the assertion shifts from equality to good enough
  • Three consequences: running is not quality, change impact is diffuse (sampling misses it), and in multi-agent any of five stages failing looks like the same symptom
  • Framing: evaluation is not testing — it establishes a comparable baseline for a stochastic system, yielding a number rather than pass/fail
  • Method: a small stable golden set, a required-facts checklist per item, an LLM-as-judge, and a dashboard sharing tracing's data source
  • The checklist is the key move: it converts is this good into were these facts stated — unverifiable into verifiable
  • Unit tests remain for deterministic parts; merging the two makes CI fail randomly until everyone ignores it

  答题要点

  • 根本差别：同样的输入 Agent 不保证同样的输出，断言从「等于」变成「够不够好」
  • 三个后果：跑通不等于质量合格、改动影响弥散（抽查会漏）、多 Agent 里五道工序任一歪了都表现成同一个症状
  • 定位：评估不是测试，是给随机系统建一条可比较的基线，产物是能和上次比的数字而不是通过与否
  • 做法：小而稳的 golden set + 每条的必备信息清单 + LLM-as-judge 打分 + 与 tracing 同源的面板
  • 清单是关键，它把「答得好吗」换成「这几件事写没写」，从没法验变成能验
  • 单元测试仍然需要，负责确定性部分；两者混在一起会让 CI 随机变红，最后被所有人无视
• What makes LLM-as-judge unreliable, and what do you do about it?用大模型给大模型的输出打分（LLM-as-judge），有哪些不可靠的地方？怎么办？

  Common in ChinaCommon overseasDeep dive#evaluation#llm-as-judge#reliability

  How to reason about it · think before answering

  1. This screens for whether you have actually used it. People who have can name specific failure shapes with magnitudes; people who have not just say it might be inaccurate.
  2. First, self-preference: when the judge and the evaluated agent share a model, it favours its own output — the same model has a consistent notion of what a good answer looks like, so asking it to review what it just wrote gets an approving verdict. Measured: on the same batch of deliberately degraded outputs, a same-model judge gave 14/15 while a different model gave 12/15, and the extra passes were exactly the borderline cases worth catching. This is not confined to judges — every model-grading-model position has it, and a Critic node is the same problem.
  3. Second, length bias: judges reward longer answers. Measured: padding a correct 33-character reply with 141 characters of irrelevant pleasantries moved an impression-based rubric from 2 to 4 without changing a word of substance.
  4. Third, rubric drift: scores shift wholesale when the judge prompt is tweaked. The same output scored 2 under one rubric and 5 under another. Hence the hard rule: scores are comparable only within one judge prompt, and cross-version comparison is meaningless.
  5. Match each remedy to its failure rather than saying run it a few more times. Freeze and version the judge prompt — every score record carries its rubric version and judge model, which are its coordinates, and a dashboard that finds two rubrics mixed should refuse to aggregate rather than emit a meaningless average. Default to a different model as judge, as a default and not an option. Keep a small human-labelled calibration set and re-run it whenever the rubric changes, comparing verdicts (pass or fail) rather than score deltas — one point of drift is fine, a flipped verdict is an incident.
  6. And one deeper fix: replace impressionistic criteria with a checkable list, which also dissolves length bias — counting items off a list gives padding nothing to earn. Measured, that padded reply scored 5 both before and after under the checklist rubric.
  7. Expect: is a judge cheaper than humans? The judge's cost is the same order as the system being evaluated, so what a full evaluation run costs decides whether you run it per commit or nightly. Human cost is not money but latency — it cannot give you feedback at the speed of one prompt edit, which is why humans belong on the calibration set only.

  分析过程 · 先想清楚再作答

  1. 这题筛的是「你是真用过，还是听说过」。用过的人能报出具体的失效形态和量级，没用过的人只会说「可能不准」。
  2. 第一种，**同源偏差**：judge 和被评估的 Agent 用同一个模型时，它偏向认可自己的输出——同一个模型对「什么算好答案」的偏好是一致的，让它复核自己刚写的东西，它当然觉得没问题。实测数量级：同一批被改坏的产出，同源 judge 给 14/15，换个模型只给 12/15，被多放过去的正是最该抓的边缘产出。这个坑不止在 judge，**凡是「模型评模型」的位置都有**，Critic 节点是同一个问题。
  3. 第二种，**长度偏好**：judge 倾向给篇幅大的答案更高分。实测：一条 33 字的正确回复灌上 141 字无关客套话，凭印象打分的提示词就从 2 分涨到 4 分，内容一个字没变。
  4. 第三种，**评分提示词漂移**：judge 的评分随提示词微调整体移动。同一份产出，两套评分提示词一套给 2 分一套给 5 分。所以有条硬纪律——**分数只在同一套 judge 提示词内部可比**，跨版本比较是没有意义的。
  5. 解药要一一对应，别笼统说「多测几次」：固定 judge 提示词并版本化（每条评分记录带上 rubric 版本与 judge 模型，那是它的坐标；面板发现混了两套口径应当直接拒绝聚合，而不是算出一个没含义的平均分）；默认用不同的模型当 judge，而且这该是默认值不是可选项；留一小批人工标注做校准集，每次改评分提示词拿它对一遍，**比的是结论（过或不过）而不是分数差**——差 1 分无所谓，结论翻了就是事故。
  6. 还有一条更根本的：**把评分标准从主观印象换成可核对的清单**，它同时解掉长度偏好——照清单逐条数，灌水加不了分。实测那条灌水回复在清单口径下前后都是 5 分，纹丝不动。
  7. 可以预期的追问：judge 便宜还是人工便宜？答：judge 的成本和被评估的系统本身一个量级，所以「跑一次全量评估多少钱」是你决定每次提交都跑还是每天跑一次的依据；而人工的成本不在钱在延迟——它给不了你改一次提示词就想看一次结果的反馈速度，所以人工只该用在校准集上。

  Key points

  • Self-preference: a same-model judge inflates scores (14/15 vs 12/15 cross-model), and it applies to every model-grading-model spot including Critic
  • Length bias: 141 characters of padding moved an impression score from 2 to 4 with no substantive change
  • Rubric drift: the same output scored 2 and 5 under two rubrics, so scores compare only within one judge prompt
  • Remedies map one-to-one: version the rubric and store it alongside each record, refuse to aggregate mixed rubrics, default to a different judge model
  • Keep a human-labelled calibration set and compare verdicts, not score deltas — a point of drift is fine, a flipped verdict is an incident
  • The deeper fix is a checkable list instead of impressions, which also removes length bias (the padded reply scored 5 both ways)

  答题要点

  • 同源偏差：judge 与被评估 Agent 同模型会虚高（实测 14/15 vs 异源 12/15），且凡「模型评模型」的位置都有，Critic 同理
  • 长度偏好：灌水 141 字能让印象分从 2 涨到 4，内容一字未变
  • 评分提示词漂移：同一产出两套 rubric 一个 2 分一个 5 分，所以分数只在同一套提示词内部可比
  • 解药一一对应：rubric 版本化并随记录存坐标、面板发现混口径直接拒绝聚合、默认换模型当 judge
  • 留人工标注校准集，比结论（过/不过）而不是比分数差——差 1 分无所谓，结论翻了是事故
  • 更根本的是把主观印象换成可核对的清单，同时解掉长度偏好（清单口径下灌水前后都是 5 分）
• System design: a multi-agent support platform is live, the team edits prompts several times a week, nobody can say whether quality is improving, and cost is only known as a month-end total. Design its evaluation and observability system.系统设计：一个多 Agent 客服平台已经上线，团队每周改几次提示词，但没人说得清质量是变好还是变差，成本也只有一个月底的总数。请为它设计一套评估与可观测体系。

  Common in ChinaCommon overseasDeep dive#system-design#evaluation#observability#cost

  How to reason about it · think before answering

  1. Do not draw an architecture diagram yet. The trap is that this sounds like build monitoring, so many candidates open with Prometheus and Grafana — that answers infrastructure, not this question. Spend three to five minutes on four things: how often prompts change and how they ship (weekly cadence, canary, rollback); how problems surface today (user complaints, or someone happening to notice); what history exists (how long conversations are retained, whether they can be replayed); and who consumes this (engineers debugging, or an executive watching spend). All four materially change the design, so asking them scores.
  2. Then the trunk, in one sentence: one dataset, two readings. Instrument once, as spans; read across for a single request's call tree (debugging) and stack them for a dashboard (trends and cost). This is the foundation — two data sources will eventually disagree and then nobody trusts either. Many candidates fork here into a monitoring system and an evaluation system, which is the source of every later problem.
  3. Then three layers. Layer one, offline regression: a small stable golden set (15 to 50), covering three things — every route exercised, one item per failure mode (low-confidence fallback, tool budget exhaustion, downstream outage), and the cases behind real past incidents. Each item declares its expected route and a checklist of required facts. The maintenance rule is add, never edit: changing an expectation voids all historical scores. Score with an LLM-as-judge using a different model, and version the rubric, storing that version on every record. This layer runs in CI on every prompt change and emits a number comparable to last time.
  4. Layer two, online observability: every request writes a span tree recording the routing rationale (a model decision, lost forever if not captured), per-node tokens and latency, and degradation and fallback events. The dashboard answers four questions: how much is wrong, where it is slow, what it cost, and which role spent it — that last one is multi-agent specific and the most actionable.
  5. Layer three, online sampled evaluation: fifteen offline cases cannot cover the real traffic distribution, so sample a fraction of live requests (say 1%) through the same judge to get a true quality curve. This layer bridges the other two: offline tells you whether you broke something known, online tells you what real users encountered.
  6. Bring numbers on cost, which is what separates levels. A multi-agent request can produce five to ten model calls, so per-call price is an order of magnitude below the real unit cost and you must price per request. Give the arithmetic: 10k DAU at three sessions each and five calls per session is 150k calls a day; at 2000 input and 500 output tokens, $0.15 and $0.60 per million, that is roughly $90 a day. That number implies two things: per-node attribution shows where to optimise, and evaluation's own cost must be tracked separately, since judge calls are the same order as the system itself and decide whether you evaluate per commit or nightly.
  7. Close on adoption, which many candidates omit: wire evaluation into the release process (block a deploy when pass rate drops below threshold), keep the rubric and golden set in the repository under code review, and pair every mechanism with a failure mode — judges favour same-family models, golden sets get gamed (someone tunes prompts to make it green, and at that moment it is worthless), sampling misses the long tail. A proposal with no stated failure modes reads as book knowledge.
  8. Expect, by frequency: which model judges (one tier above the system under test, and necessarily a different family); where the golden set comes from (start with human-labelled production samples, then append every incident); what happens when this system itself misbehaves (the dashboard refuses to aggregate mixed rubric versions rather than emitting a meaningless average); and how long to build (layer two in a week, layer one in two, layer three in a month since it depends on both).

  分析过程 · 先想清楚再作答

  1. 先别画架构图。这道题的陷阱是它听起来像「搭一套监控」，于是很多人上来就报 Prometheus 加 Grafana——那答的是基础设施，不是这道题。花三到五分钟问清四件事：一是**改提示词的频率和发布方式**（每周几次、有没有灰度、能不能回滚）；二是**现在出问题是怎么发现的**（用户投诉？还是有人偶然看到？）；三是**有没有历史数据**（线上对话存了多久、能不能回放）；四是**谁来看这套东西**（工程师排障，还是老板看成本）。这四个答案会实质改变设计，问它们本身就是分数。
  2. 然后给主干，一句话定形状：**一份数据、两种读法。** 埋点只做一套（span），横着读是一次请求的调用树（排障用），竖着堆是面板（趋势和成本用）。**这条是地基**——两套数据来源迟早对不上，然后没有人相信任何一个。很多候选人在这里就分叉成「监控系统」和「评估系统」两套，那是后面所有麻烦的源头。
  3. 接着按三层展开。**第一层，离线回归**：建一个小而稳的 golden set（15 到 50 条），三层覆盖——每条路由都有人走、每种失败模式各一条（置信度不足落兜底、工具预算耗尽降级、下游挂掉）、以及历史上真出过事故的那几条。每条写清期望路由和必备信息清单。维护规矩是**只增不改**：改一条期望，历史分数全部作废。用 LLM-as-judge 对照清单打分，**judge 换一个模型、rubric 版本化并随每条记录存下来**。这一层挂在 CI 上，每次改提示词跑一遍，产出一个能和上次比的数字。
  4. **第二层，在线观测**：每次请求落一棵 span 树，必须记路由理由（模型做的决策，当时不记就永远丢了）、每个节点的 token 与耗时、以及降级和兜底事件。面板回答四个问题：错了多少、慢在哪、花了多少、**钱花在哪个角色身上**。最后一个是多 Agent 特有的，也最有用。
  5. **第三层，在线采样评估**：离线的 15 条覆盖不了真实流量分布，所以按比例采样线上请求（比如 1%）跑同一套 judge，得到一条真实质量曲线。**这一层是前两层的桥**：离线告诉你有没有改坏已知的东西，在线告诉你真实用户遇到了什么。
  6. 成本这块要给数字感，这是区分层级的地方。**多 Agent 一次用户请求可能产生 5 到 10 次模型调用**，所以「每次调用多少钱」比真实单价小一个数量级，**必须按请求算钱**。给个算式：日活一万、人均三次会话、每次 5 次调用就是 15 万次调用；按输入 2000 输出 500 token、$0.15/$0.60 每百万算，一天约 90 美元。这个数立刻推出两件事：按节点分摊能定位省钱的地方，以及**评估本身的成本要单独记**——judge 调用和被评估系统一个量级，它决定你每次提交都跑还是每天跑一次。
  7. 最后收在「怎么让它真的被用起来」，这是很多人漏的一层：把评估结果接进发布流程（通过率跌破阈值就挡住发布）、把 rubric 和 golden set 放进代码仓库走 code review、以及**给每个机制配一句失效模式**——judge 会偏向同源模型、golden set 会被针对性优化（有人为了让它绿而调提示词，那一刻它就失去了意义）、采样会漏掉长尾。说不出失效模式的方案，面试官会认为你只是读过。
  8. 可以预期的追问，按频率排：judge 用什么模型（比被评估的强一档，且必须异源）；golden set 从哪来（先从线上捞一批人工标注，再逐次把事故补进去）；这套东西自己出问题怎么办（面板发现 rubric 混版直接拒绝聚合，而不是给一个没含义的平均分）；多久能上线（第二层一周、第一层两周、第三层一个月，因为它依赖前两层）。

  Key points

  • Spend three to five minutes clarifying four things: prompt change cadence and release process, how problems surface today, what replayable history exists, and who the audience is
  • The trunk is one dataset, two readings: instrument once as spans, read across for a call tree and stack for a dashboard; two sources will disagree
  • Layer one, offline regression: a small stable golden set covering every route, every failure mode and past incidents, add-never-edit, wired into CI
  • Layer two, online observability: span trees recording routing rationale, per-node tokens and latency, degradation events; the dashboard answers wrong/slow/cost/which-role
  • Layer three, sampled online evaluation through the same judge, covering the real distribution the offline set cannot
  • Price per request, not per call: five to ten calls per request, with arithmetic showing ~$90/day at 10k DAU; track evaluation's own cost separately
  • Close on adoption: block releases when pass rate drops, keep rubric and golden set in the repo under review
  • Pair every mechanism with a failure mode: judge self-preference, golden set gaming, sampling missing the tail — omitting these reads as book knowledge

  答题要点

  • 先用三到五分钟问清四件事：改提示词的频率与发布方式、现在问题怎么被发现、有无历史数据可回放、这套东西给谁看
  • 主干是「一份数据、两种读法」：埋点只做一套 span，横着读是调用树、竖着堆是面板；两套数据源迟早对不上
  • 第一层离线回归：小而稳的 golden set，三层覆盖（每条路由、每种失败模式、历史事故），只增不改，挂 CI
  • 第二层在线观测：span 树记路由理由、每节点 token 与耗时、降级兜底事件；面板回答错了多少/慢在哪/花了多少/钱花在哪个角色
  • 第三层在线采样评估：按比例采样线上请求跑同一套 judge，补上离线覆盖不到的真实分布
  • 成本必须按请求算而非按调用：一次请求 5 到 10 次调用，给出日活一万约 90 美元一天的算式；评估自身成本单独记
  • 收在落地：通过率跌破阈值挡发布、rubric 与 golden set 进仓库走 review
  • 每个机制配失效模式：judge 偏向同源、golden set 会被针对性优化、采样漏长尾——说不出失效模式等于只是读过

D24 RAG, Level Up: Hybrid Search, Reranking, Citations, Recall Evaluation

• How do you evaluate retrieval quality in a RAG system, and how should the evaluation set be built?怎么评估一个 RAG 系统的检索效果？评估集应该怎么构造？

  Common in ChinaCommon overseasDeep dive#rag#evaluation#recall

  How to reason about it · think before answering

  1. This is a very common question in the Chinese market and the fastest way to expose someone who has assembled RAG but never tuned it. The test: does your answer contain concrete metric names and an annotation granularity?
  2. First separate what is being evaluated — the step people most often conflate. Retrieval evaluation asks 'was it found'; generation evaluation asks 'was the answer right'. Keep two separate sets. Merge them and, when the score drops, you cannot tell whether retrieval missed or the model fumbled — and those have completely different fixes.
  3. Shape of the set: about 20 queries, each annotated with 1-3 chunk ids that must be retrieved. Annotate at chunk level, not document level — chunks are the retrieval unit, and document-level labels inflate the numbers. Cover the real query mix, especially the types you know break: codes, paraphrase, cross-document.
  4. Three metrics, three questions. recall@5 is what actually reaches the model, so it is the number you care about. recall@20 is the ceiling — if it does not move, the problem is on the recall side and no reranker will save you. MRR is sensitive to ordering and breaks ties when recall is equal.
  5. Production view: freeze the set once agreed, because changing samples destroys comparability — the same reason a factory keeps fixed reference samples. Pair it with online counterparts (empty-citation rate, hallucinated-citation rate, escalation rate), since passing offline does not mean passing in production.
  6. Expected follow-up: is 20 enough given the labelling cost? Not for statistical significance, but enough for regression — its job is to stop retrieval silently getting worse. Scale up before you settle an A/B, and grow it from failure cases rather than random additions.

  分析过程 · 先想清楚再作答

  1. 这题是国内面试的极高频题，也是最容易暴露「只搭过没调过」的一题。判据很简单：你的回答里有没有出现**具体的指标名和标注粒度**，没有就是没做过。
  2. 先把评估对象分清楚——这是最容易混的一步：**检索评估问「找得到找不到」，生成评估问「答得对不对」**。两套评估集要分开维护。混成一套的后果是分数掉了你分不清是检索漏了还是模型答砸了，而这两件事的修法完全不同。
  3. 评估集的形状：20 条左右的 query，每条**人工标注 1 到 3 个必须召回的 chunkId**。注意标注粒度是**块**不是文档——检索的单位就是块，标到文档级会让指标虚高。query 要覆盖真实分布，尤其要包含那些你知道会翻车的类型（编号、同义改写、跨文档）。
  4. 三个指标各回答一个问题：recall@5 是「进上下文的那几条覆盖了多少」，也就是你真正关心的数；recall@20 是天花板，它上不去说明问题在召回侧、重排再强也没用；MRR 对排序质量敏感，recall 打平时用它分高下。
  5. 生产视角：评估集一旦定下来就要冻结，换了样本分数就没有可比性——这和产线质检必须用固定的标准样品是同一个道理。同时线上要有对照指标（引用为空率、幻觉引用率、转人工率），因为离线过了不等于线上没事。
  6. 可预期的追问：标注成本这么高，20 条够吗？答：20 条不够做统计显著性，但足够做**回归**——它的作用是「改了检索之后别悄悄变差」。要做 A/B 定论再上规模，而且优先扩充失败案例，不是随机加样本。

  Key points

  • Retrieval and generation evaluation are two separate sets: 'was it found' versus 'was the answer right'.
  • Around 20 queries, each labelled with 1-3 chunk ids that must be retrieved — chunk level, not document level.
  • recall@5 is what the model actually sees, recall@20 is the ceiling, MRR measures ordering quality.
  • Freeze the set once agreed or scores stop being comparable; pair it with online empty-citation and hallucinated-citation rates.
  • Twenty cases is a regression guard, not a significance test; grow it from failure cases, not random samples.

  答题要点

  • 检索评估和生成评估是两套：前者问「找得到找不到」，后者问「答得对不对」，分开维护。
  • 评估集是 20 条左右的 query，每条人工标 1 到 3 个必须召回的 chunkId——标到块级，不是文档级。
  • recall@5 是真正关心的数（模型只看得到这几条），recall@20 是天花板，MRR 衡量排序质量。
  • 评估集一旦定下来就冻结，否则分数没有可比性；线上再配引用为空率、幻觉引用率做对照。
  • 20 条不够做显著性但够做回归；扩充时优先补失败案例，不是随机加样本。