Interview Bank

MCP in 7 Days: Wire Tools Into Any Agent

D7 Productionizing and Retrospective: Writing Evals for Tools, Versioning, Publishing to npm and a Registry, Observability, and a Capstone Project

• A user reports that one of your MCP tools is 'always getting it wrong' in production. In what order do you investigate?线上有人反馈某个 MCP 工具「总是调不对」。你按什么顺序排查？

  Common in ChinaCommon overseasIntermediate#observability#debugging

  How to reason about it · think before answering

  1. This tests ordering, not breadth. Anyone who opens with logs and stack traces gets asked how they know the problem is server-side at all.
  2. Step zero is translating 'getting it wrong' into three mutually exclusive symptoms, without which everything after is guesswork: it was never called, it was called with wrong arguments, or it was called correctly and returned the wrong thing. Asking whether it did nothing or did the wrong thing, or simply checking whether a call was logged, separates them.
  3. Each symptom has its own path. Never called means the description is at fault: run the eval set and see whether happy paths or edges dropped, since happy-path drops mean a vague description and edge drops mean the sentence distinguishing similar tools is missing. Wrong arguments means the schema is at fault: ambiguous field names, unstated formats, wrong required markers. The signal is a persistently high tool-execution error rate, which usually means the schema is unclear rather than the model being dumb. Only a wrong result is a code problem, and only then do unit tests and logs matter.
  4. State the metric ordering too: error rate first, selection accuracy second. A normal error rate with unhappy users almost always means the tool is not being chosen; a spiking error rate sends you to the code and upstream. Watch P95, not the mean, because a remote tool degrading from 200 milliseconds to 8 seconds barely moves an average.
  5. Volunteer two commonly missed causes. Aggregation collisions: the client is connected to several servers, two tools share a name, and the model picked the other one, so your server was never called and investigating it will never find anything. And version or caching: list results carry ttlMs cache hints, so the client may hold a stale tool list, and refresh depends on a listChanged notification.
  6. Conclusion: the chain has four links, description, schema, aggregation and caching, and implementation. Walk it in the order the model sees it, because the earliest links produce no error logs and are therefore the ones people skip.
  7. Likely follow-up: how do you keep evidence? Emit one structured log per call with tool name, an argument digest plus field names, the isError flag, duration, and whether a human confirmed the call, that last column being the only way to distinguish user intent from the model acting on its own.

  分析过程 · 先想清楚再作答

  1. 这题考的是排查的**顺序**，不是知识点的多少。上来就贴日志和堆栈的人会被追问「你怎么知道问题在服务端」。
  2. 第零步是把「调不对」翻译成三种互斥的现象，这一步不做后面全是猜：一是**没被调**（模型压根没选这个工具）；二是**调了但参数错**；三是**调了参数也对，但结果不对**。问一句「那次它是没动，还是动了但做错了」，或者直接去日志里看有没有这条调用记录，就能分开。
  3. 对应三条不同的路。没被调，问题在**描述**：去跑评估集，看正例还是边界掉了；正例掉说明描述写糊，边界掉说明缺了区分相似工具的那句话。参数错，问题在 **schema**：看字段名是不是有歧义、描述里有没有写清格式、必填项是不是标对了；这类问题的信号是错误率里工具执行错误持续偏高——那通常不是模型笨，是 schema 没说清。结果不对才是代码问题，这时候才轮到单元测试和日志。
  4. 指标层面的顺序也说一下：**先看错误率，再看选中率**。错误率正常但用户说不好用，八成是选不中；错误率飙了才去看代码和上游。耗时看 P95 不看平均值，远程服务端上一个工具从 200 毫秒退化到 8 秒，平均值可能只动一点点。
  5. 还有两条容易被忽略但很常见的原因，要主动提。一是**聚合冲突**：客户端连了多个服务端，两个工具重名，模型选中的是另一个服务端的那个——这时候「你的工具」根本没被调，查你的服务端永远查不出来。二是**版本或缓存**：列表结果带 ttlMs 缓存提示，客户端可能拿着旧的工具清单；工具清单变了要靠 listChanged 通知才会重新拉。
  6. 结论：这条链上有四个环节——描述、schema、聚合与缓存、实现。**按模型看得见的顺序从前往后查**，因为越靠前的环节越不产生错误日志，也就越容易被跳过。
  7. 可预期的追问：怎么留证据？每次调用记一条结构化日志，字段里要有工具名、参数摘要与字段名、是否 isError、耗时、以及这次调用有没有经过人工确认；最后那一栏是事后区分「用户授意」和「模型自作主张」的唯一依据。

  Key points

  • First split 'getting it wrong' into never called, wrong arguments, or wrong result; the split decides where to look
  • Never called points at the description and the eval set; wrong arguments at the schema; only a wrong result at the code
  • Check error rate before selection accuracy, and read P95 rather than the mean
  • Do not miss aggregation collisions, where another server's same-named tool was chosen, or a stale cached tool list

  答题要点

  • 先把「调不对」分成没被调、参数错、结果错三种互斥现象，再决定查哪里
  • 没被调查描述并跑评估集；参数错查 schema；结果错才轮到代码与日志
  • 指标顺序是先错误率再选中率；耗时看 P95 不看平均值
  • 别漏掉聚合重名（选中的是别的服务端的同名工具）和工具清单缓存这两类原因