Interview Bank

MCP in 7 Days: Wire Tools Into Any Agent

D7 Productionizing and Retrospective: Writing Evals for Tools, Versioning, Publishing to npm and a Registry, Observability, and a Capstone Project

• How do you evaluate whether an MCP tool is any good, and what categories of test cases would you design?怎么评估一个 MCP 工具做得好不好？你会设计哪几类测试用例？

  Common in ChinaCommon overseasIntermediate#eval#tooling

  How to reason about it · think before answering

  1. The screen is whether you have shipped tools. 'Write unit tests' answers the wrong question: unit tests check that given arguments produce the right output, while the first thing to break on an MCP tool is the model not selecting it at all, so the arguments never reach your function.
  2. Define the target: an eval measures selection accuracy, meaning given a user utterance and a tool list, does the model pick the right tool. Correctness of the tool itself belongs to unit tests, and the two layers should not be blurred.
  3. Then the three categories, which is the direct answer. Happy path: unambiguous intent, such as 'find me the release process doc'. Edge: intent carried by semantics rather than keywords, such as 'what does release-process say', which has no verb cue and only a slug-shaped token, and is most often misread as a search. Traps: cases where nothing should be called, such as 'thanks, no need to look it up', 'how do you say document in English', and 'what can you do'. I usually weight them four, three, three.
  4. Stress that the third category is the dividing line: an eval set of only happy paths reports a comfortable hundred percent while measuring nothing about over-triggering, which is what most production complaints actually are, and over-triggering has side effects.
  5. Then the assertion, where the classic bug lives: treating an expected value of null as 'anything goes', which makes negatives permanently green. The correct assertion judges both directions. I also rerun the whole set with a deliberately wrong selector that always picks the same tool and require every negative to fail, which validates the assertion rather than the selector.
  6. Finally the engineering constraints: one command, under a minute, on a cheap small model, because a slow eval is no eval, since whoever edits a description will not wait. And report per category rather than one number: happy-path drops mean a vague description, edge drops mean the sentence distinguishing two similar tools is missing, trap drops mean the description over-claims.
  7. Likely follow-ups: when do you run it? On any change to descriptions, schemas, or the tool set, wired into CI. And what if the model changes? The eval set is a cross-model asset, so you rebaseline on a model switch, which is part of why it pays for itself.

  分析过程 · 先想清楚再作答

  1. 这题在筛「有没有真的上线过工具」。答「写单元测试」是答错了赛道——单元测试测的是给定参数输出对不对，而 MCP 工具最先出问题的地方是模型压根没选它，参数根本到不了你的函数。
  2. 先把要评估的对象说清：评估评的是**选中率**，也就是给一句用户的话和一张工具表，模型会不会选中该选的那个。工具本身的正确性归单元测试，两层不要混。
  3. 然后给三类用例，这是本题的正面回答。正例：意图明确时选得中，比如「帮我搜一下发布流程文档」。边界：意图靠语义而不是关键词，比如「release-process 这篇讲了什么」——没有任何动词提示，只有一个像 slug 的词，最容易被误判成搜索。诱导误选（负例）：不该调的时候一个都不调，比如「谢谢，不用查文档了」「文档这个词英文怎么说」「你都能干什么」。占比我一般给四三三。
  4. 第三类是分水岭，要主动强调：只有正例的评估集会给你一个 100% 的假象，它测不出过度触发，而线上大多数投诉恰恰是过度触发——用户随口一句否定，助手转头就去干了，还带副作用。
  5. 接着讲断言，这里有个最常见的写法错误：expected 为 null 的用例被当成「随便都行」，于是负例永远绿。正确的断言两个方向都判：期待某个工具时选中它才算过，期待不调用时什么都不选才算过。我会额外拿一个「总是选同一个工具」的假选择器再跑一遍，要求负例全部失败——这验的不是选择器，是我的断言真的在起作用。
  6. 最后是工程约束：评估集要能一条命令跑完，一分钟以内，用便宜的小模型跑。跑得慢的评估集等于没有，因为改描述的人不会等。结果也不要只看总分，按三类分开看才有行动价值：正例掉了说明描述写糊了，边界掉了说明缺了区分相似工具的那句话，负例掉了说明描述写得太热情。
  7. 可预期的追问：什么时候跑？描述、schema、工具增删这三类改动都必须跑，把它挂进 CI。再追问会问到「模型换了怎么办」，答案是评估集是跨模型的资产，换模型时先跑一遍拿到基线，这也是它值得投入的原因之一。

  Key points

  • Evals measure selection accuracy, not tool correctness; the latter is unit-tested and the layers must not blur
  • Three categories are mandatory: happy path, edge, and traps, weighted roughly four three three
  • Assert both directions: an expected null must mean nothing was called, and validate the assertion with a deliberately wrong selector
  • One command, under a minute, scored per category, and triggered by any description or schema change

  答题要点

  • 评估评的是选中率，不是工具正确性；后者归单元测试，两层不能混
  • 三类用例缺一不可：正例、边界、诱导误选，建议四三三
  • 断言两个方向都判：期待 null 时必须什么都不选；再用故意选错的选择器验证断言本身
  • 一条命令一分钟内跑完，按类别分开看分数，描述与 schema 改动必须触发