Interview Bank

Agent Skills in 7 Days: Turn Experience Into Reusable Capability

D7 Capstone and Retrospective: Turning a Team's Conventions Into a Skill Pack and Driving a Subagent Through a Real Task

• How do you prove a skill actually helps rather than just feeling better?你怎么证明一个 skill 真的有用，而不是感觉上更好？

  Common in ChinaCommon overseasDeep dive#agent-skills#evaluation#methodology

  How to reason about it · think before answering

  1. This tests evaluation skill and honesty. Saying it felt better ends the answer; the interviewer wants a reproducible comparison.
  2. Give the structure first: one set of cases, two arms differing in exactly one variable, per-assertion judging, and a pass-rate comparison. The conclusion is a single number, the delta.
  3. Then explain how to keep the comparison clean, the half most people skip. Never test in the session you spent two hours debugging: that context is littered with convention snippets you typed and corrections you made, so good output reflects you, not the skill. Use a fresh subagent, with the two definitions differing only in which skills are preloaded.
  4. Describe the case mix: positive, boundary and negative roughly five to three to two. Negatives are non-negotiable because they measure whether the trigger surface is too wide, which is the most common way a skill goes wrong. Without them, a skill that grabs everything scores perfectly.
  5. Assertions are the core. Decidable means checking facts, not quality: the type field is one of six values, the scope equals a real directory in the repository, the first line is under fifty characters. Written clearly is not decidable. One assertion checks one thing so failures point somewhere.
  6. Close on honesty: some judgments resist reliable assertions, such as whether a review comment found the real problem. Forcing an assertion yields false green. Mark those as human-judged, sample a few, and say so in the conclusion.
  7. Expected follow-up: does a small sample support the claim? Be candid. A small sample supports a claim about that batch of tasks only, so every number carries its measurement conditions and is never extrapolated into a general efficiency gain.

  分析过程 · 先想清楚再作答

  1. 这题在考评估能力，也在考诚实。答「我试了几次感觉好多了」直接出局，面试官要的是一个可复现的对照。
  2. 先给整体结构：同一批用例、两组只差一个变量、逐条判定、比通过率。**结论只有一个数：通过率差值。**
  3. 然后讲对照怎么做干净，这是本题最容易被忽略的一半。**绝对不要在你调试了两小时的那个会话里试**——那个上下文里散落着你手打的规范片段和你纠正过的措辞，模型产出得好是因为你自己把答案说了一遍。要用一个上下文干净的子代理，两份定义只差「预加载哪几个 skill」这一行，模型、工具集、提示词全部一致。
  4. 再讲用例集怎么配：正例、边界例、负例大约五比三比二。负例不能省，它测的是触发面有没有过宽，而**过宽是 skill 最常见的坏法**——少了负例，一个什么都抢的 skill 也能拿满分。
  5. 断言是全部重点。可判定的意思是不看好坏、只看事实成不成立：「类型字段取自那六个值之一」「范围等于仓库里真实存在的目录名」「首行不超过 50 个字符」是可判定的；「写得清楚」不是。一条断言只查一件事，失败时才知道是哪一条挂了。
  6. 最后补诚实这一层：有些判断写不出可靠断言，比如「这条评审意见有没有抓住真问题」。硬凑只会得到假绿，老实标成人工判定、抽查几条、并在结论里注明有几条是人工判的。**一份诚实的部分自动化评估远好过一份全绿的假评估。**
  7. 可预期的追问是「样本量这么小，结论站得住吗」。答话要坦率：小样本只能支撑「在这一批任务上」的结论，所以每个数字都要带测量条件，不要外推成通用效率提升。

  Key points

  • Same cases, two arms differing in one variable, judged per assertion, compared by pass rate.
  • The comparison needs a context-clean subagent, never the session you debugged in.
  • The two subagent definitions differ only in preloaded skills; model, tools and prompt are identical.
  • Include negative cases: they measure an over-wide trigger surface, the most common failure.
  • Assertions must be decidable and single-purpose; mark human-judged cases honestly in the conclusion.

  答题要点

  • 同一批用例、两组只差一个变量、逐条判定、比通过率差值。
  • 对照必须用上下文干净的子代理，不能在调试过的会话里试。
  • 两份子代理定义只差预加载 skill 那一行，模型、工具、提示词全部一致。
  • 用例要含负例，它测触发面有没有过宽，过宽是最常见的坏法。
  • 断言要可判定、一条只查一件事；判不了的老实标人工判定并在结论里注明。