面试题库

30 天从前端工程师到 Agent 工程师

D21 评估与可观测：golden set、LLM-as-judge、tracing、失败率/成本面板；Pi vs LangGraph 总结；W3 复盘

• 怎么评估一个 Agent 的效果？和传统后端服务的测试有什么不同？How do you evaluate an agent's quality, and how does it differ from testing a conventional backend service?

  国内高频海外高频进阶#evaluation#testing#agent-quality

  分析过程 · 先想清楚再作答

  1. 题眼是「不同」。只答「建一个测试集跑准确率」拿不到分——那是机器学习的标准答案，面试官想看你能不能说清 Agent 这个场景特殊在哪。
  2. 根子上的差别只有一句：**同样的输入，Agent 不保证给同样的输出**。传统测试的断言是「等于」，而 Agent 的产出没有唯一正确答案，只有「够不够好」。断言从等值变成了判分，整套方法论跟着变。
  3. 这条差别连锁出三个后果，说全了这题就稳了：一是**跑没跑通判断不了质量**——流程没抛错，不等于回复里写明了退款结论；二是**改动的影响是弥散的**，改一个字的提示词可能只影响一类请求，人肉抽查五条恰好没覆盖到，你会得出「没影响」然后上线；三是**多 Agent 又难一层**，一次请求走路由、拆分、并行执行、评审、汇总五道工序，任何一道歪了都表现成「最后那段话不太对」，不分开量就不知道该改哪块。
  4. 所以给出定位：**评估不是测试，评估是给一个随机系统建立一条可比较的基线。** 它的产物不是「通过」或「不通过」，而是一个能和上一次比的数字。既然要比，样本集就必须固定。
  5. 然后落到具体做法：一个小而稳的 golden set（本课 15 条），每条写清期望走哪条路由和一份必备信息清单；用 LLM-as-judge 对照清单打分；把评估结果和链路追踪接到同一块面板上。**清单是关键**——它把「这答得好吗」这种没法验的问题，换成了「这几件事写没写」这种能验的问题。
  6. 可以预期的追问：那还需要单元测试吗？需要，而且分工很清楚——工具函数、状态迁移、reducer 这些确定性的部分照旧用单元测试断言等值，评估只负责模型产出那一段。把两者混成一套，你会得到一堆随机失败的测试，然后所有人开始无视 CI。

  How to reason about it · think before answering

  1. The hinge is differ. Answering build a test set and measure accuracy is the textbook ML answer and misses the point; the interviewer wants to know whether you can articulate what makes agents special here.
  2. The root difference is one sentence: the same input does not guarantee the same output. Conventional tests assert equality, but an agent's output has no single correct answer, only good enough. Once the assertion changes from equality to scoring, the whole methodology changes with it.
  3. That difference cascades into three consequences, and covering all three secures the question. First, whether it ran tells you nothing about quality — the flow not throwing does not mean the reply stated the refund conclusion. Second, the blast radius of a change is diffuse: a one-word prompt edit may affect only one class of request, and hand-checking five samples that happen to miss that class yields no impact, then you ship. Third, multi-agent adds a layer: one request passes routing, planning, parallel execution, review and aggregation, and any one of them going wrong surfaces as that last paragraph seems off — without measuring each stage you cannot tell which to fix.
  4. So frame it: evaluation is not testing. Evaluation establishes a comparable baseline for a stochastic system. Its output is not pass or fail but a number you can compare against last time — and to compare, the sample set must be frozen.
  5. Then get concrete: a small stable golden set (15 items here), each declaring its expected route and a checklist of facts the reply must contain; an LLM-as-judge scoring against that checklist; and evaluation results joined to tracing on one dashboard. The checklist is the key move — it converts is this a good answer, which cannot be verified, into were these facts stated, which can.
  6. Expect: do you still need unit tests? Yes, with a clean split — deterministic parts (tool functions, state transitions, reducers) keep asserting equality in unit tests, while evaluation covers only the model-generated segment. Merge the two and you get a suite that fails randomly, after which everyone starts ignoring CI.

  答题要点

  • 根本差别：同样的输入 Agent 不保证同样的输出，断言从「等于」变成「够不够好」
  • 三个后果：跑通不等于质量合格、改动影响弥散（抽查会漏）、多 Agent 里五道工序任一歪了都表现成同一个症状
  • 定位：评估不是测试，是给随机系统建一条可比较的基线，产物是能和上次比的数字而不是通过与否
  • 做法：小而稳的 golden set + 每条的必备信息清单 + LLM-as-judge 打分 + 与 tracing 同源的面板
  • 清单是关键，它把「答得好吗」换成「这几件事写没写」，从没法验变成能验
  • 单元测试仍然需要，负责确定性部分；两者混在一起会让 CI 随机变红，最后被所有人无视

  Key points

  • The root difference: identical input does not guarantee identical output, so the assertion shifts from equality to good enough
  • Three consequences: running is not quality, change impact is diffuse (sampling misses it), and in multi-agent any of five stages failing looks like the same symptom
  • Framing: evaluation is not testing — it establishes a comparable baseline for a stochastic system, yielding a number rather than pass/fail
  • Method: a small stable golden set, a required-facts checklist per item, an LLM-as-judge, and a dashboard sharing tracing's data source
  • The checklist is the key move: it converts is this good into were these facts stated — unverifiable into verifiable
  • Unit tests remain for deterministic parts; merging the two makes CI fail randomly until everyone ignores it