面试题库

7 天 Agent Skills：把经验做成可复用能力

D3 设计方法：从重复任务提炼、检查清单式与参考手册式、四种反模式与触发测试

• 怎么测一个 skill 的 description 好不好？自己试几句话够吗？How do you evaluate whether a skill description is good? Is trying a few prompts yourself enough?

  国内高频海外高频进阶#agent-skills#evaluation

  分析过程 · 先想清楚再作答

  1. 题眼在后半句。答「自己试几句就行」直接出局，但只答「要写测试集」也不够——面试官要看你知不知道这个测试集该怎么设计。
  2. 先说为什么抽查不够：一个 skill 时够用，装到第十个就不行了，因为你既记不住十个描述之间会不会互相抢，也没法在改完一句话后判断是改好了还是改坏了。
  3. 然后给三件东西。第一是带标注的查询集，约 20 条，正负各半。正例要在措辞、显式程度、详略、复杂度四个维度上铺开；**最有价值的正例是那些确实该用但字面看不出来的**，字面已经念了一遍功能的查询任何描述都能命中，测不出区别。
  4. 负例是设计的重点：毫无重叠的句子测不出任何东西，真正有用的是近似负例——共享关键词或概念但目标动词不同。对 CSV 分析 skill，「改 Excel 预算表的公式」和「把 CSV 每行写进数据库」都是好负例。
  5. 第二是重复跑取触发率：模型是不确定的，每条跑三次算命中比例，阈值取 0.5。第三是训练验证拆分，六比四，验证集全程不看。
  6. 可预期的追问是「怎么判断一条查询触发了没有」。答案是把所有 skill 的名字与描述拼成目录，连同这句话交给模型问它该用哪一个——这正是客户端在发现阶段做的事，只是单独拎出来跑。

  How to reason about it · think before answering

  1. The hinge is the second half. Saying a few prompts is enough fails immediately, but saying write a test set is not enough either; the interviewer wants the design.
  2. Explain why spot checks fail: fine with one skill, useless at ten, because you cannot hold ten descriptions in your head nor tell whether an edit helped or hurt.
  3. Then give three ingredients. First, a labeled query set of about twenty, balanced positive and negative. Vary positives along phrasing, explicitness, detail and complexity; the most valuable positives are the ones where the skill applies but the wording does not say so.
  4. Negatives are where the design effort goes: unrelated sentences test nothing. Near-misses that share keywords but need something else are what matters, such as editing Excel formulas or loading CSV rows into a database for a CSV-analysis skill.
  5. Second, repeat runs for a trigger rate, since model behavior is nondeterministic: three runs per query with a 0.5 threshold. Third, a roughly sixty-forty train and validation split with the validation set untouched.
  6. Expected follow-up: how do you decide whether a query triggered? Build the catalog of names and descriptions, hand it plus the query to the model and ask which skill applies. That is exactly what a client does at discovery, run standalone.

  答题要点

  • 抽查在一个 skill 时够用，多个 skill 互相干扰时完全不够。
  • 约 20 条带标注查询，正负各半，正例在措辞、显式程度、详略、复杂度四维上铺开。
  • 负例必须是近似负例：共享关键词但目标动词不同，无关句子测不出东西。
  • 每条跑三次取触发率，阈值 0.5，因为模型行为不确定。
  • 训练验证六四拆分，训练集指导改写，验证集只用来选版本。

  Key points

  • Spot checks work for one skill and break down once several skills compete.
  • About twenty labeled queries, balanced, with positives varied by phrasing, explicitness, detail and complexity.
  • Negatives must be near-misses that share keywords but need a different action.
  • Three runs per query for a trigger rate with a 0.5 threshold, because behavior is nondeterministic.
  • Split roughly sixty-forty; train guides revision, validation picks the winning version.
• 优化 description 的时候怎么避免过拟合到你自己写的那几条测试查询？When optimizing a description, how do you avoid overfitting to the very queries you wrote?

  国内高频海外高频深入#agent-skills#evaluation

  分析过程 · 先想清楚再作答

  1. 这题是机器学习的老概念换了个场景，考的是你能不能把它迁移过来。能说出「验证集」三个字只是起点，真正的区分度在你怎么描述那个具体的错误动作。
  2. 先点明过拟合在这里长什么样：一条查询没触发，你把它的原话抄进描述，于是这一条过了，换一句同义的又不过。**抄原话就是过拟合的动作本身。**
  3. 正确做法是归纳：找出这条失败查询代表的**那一类说法**，然后把这一类补进去。比如「这几个文件我要提交了」失败了，该补的不是这句话，是「不含专业词的口语提交请求」这一类。
  4. 结构上靠拆分兜底：查询集按六比四拆成训练与验证，只用训练集的失败项指导改写，验证集全程不参与优化过程，两份都要保持正负比例接近，拆完固定不再洗牌。
  5. 还有两条实操经验。**挑版本按验证集通过率挑，不是按迭代顺序挑**——后面几轮往往在往训练集上过拟合，最好的可能是第三版而不是第五版。改五轮左右还不动就该停，问题多半在查询集本身而不在描述。
  6. 可预期的追问是「怎么知道是查询集的问题」。答案是看那些在两种配置下都失败或都成功的条目：都成功说明这条太容易、没有信息量，都失败说明要么标注错了要么要求超出模型能力，两类都该换掉。

  How to reason about it · think before answering

  1. This is a familiar machine learning idea in a new setting. Saying validation set is only the start; the discriminator is describing the exact wrong move.
  2. Name what overfitting looks like here: a query fails, you paste its wording into the description, that query passes, and a synonymous one fails. Pasting the wording is the overfitting act itself.
  3. The right move is to generalize: identify the category the failing query represents and cover that. If a casual phrasing failed, cover casual phrasings, not that sentence.
  4. Structurally, rely on the split: roughly sixty-forty, revise only from train-set failures, keep the validation set out of the loop, preserve label balance in both, and freeze the split across iterations.
  5. Two practical rules: pick the version by validation pass rate rather than by recency, since later rounds tend to overfit, and stop after about five iterations if nothing moves, because the problem is then in the queries.
  6. Expected follow-up: how do you know the queries are the problem? Look at items that pass or fail in every configuration. Always-pass items carry no information; always-fail items are mislabeled or beyond the model.

  答题要点

  • 过拟合的具体动作是把失败查询的原话抄进描述，要改成补它代表的那一类说法。
  • 查询集六四拆分，只用训练集指导改写，验证集全程不看。
  • 两个集合都要保持正负比例接近，拆完固定，不要每轮重洗。
  • 按验证集通过率挑版本，最好的那版不一定是最后一版。
  • 五轮不动就停，去查查询集本身是不是太容易、太难或标注错了。

  Key points

  • The overfitting move is pasting a failing query verbatim; generalize to its category instead.
  • Split roughly sixty-forty and revise only from train-set failures.
  • Keep label balance in both splits and freeze the split across iterations.
  • Select the version by validation pass rate; the best is not always the last.
  • If five rounds change nothing, inspect the query set for triviality, impossibility or mislabeling.

D7 综合与复盘：把一套团队规范做成 skill 包并驱动子代理完成一次真实任务

• 你怎么证明一个 skill 真的有用，而不是感觉上更好？How do you prove a skill actually helps rather than just feeling better?

  国内高频海外高频深入#agent-skills#evaluation#methodology

  分析过程 · 先想清楚再作答

  1. 这题在考评估能力，也在考诚实。答「我试了几次感觉好多了」直接出局，面试官要的是一个可复现的对照。
  2. 先给整体结构：同一批用例、两组只差一个变量、逐条判定、比通过率。**结论只有一个数：通过率差值。**
  3. 然后讲对照怎么做干净，这是本题最容易被忽略的一半。**绝对不要在你调试了两小时的那个会话里试**——那个上下文里散落着你手打的规范片段和你纠正过的措辞，模型产出得好是因为你自己把答案说了一遍。要用一个上下文干净的子代理，两份定义只差「预加载哪几个 skill」这一行，模型、工具集、提示词全部一致。
  4. 再讲用例集怎么配：正例、边界例、负例大约五比三比二。负例不能省，它测的是触发面有没有过宽，而**过宽是 skill 最常见的坏法**——少了负例，一个什么都抢的 skill 也能拿满分。
  5. 断言是全部重点。可判定的意思是不看好坏、只看事实成不成立：「类型字段取自那六个值之一」「范围等于仓库里真实存在的目录名」「首行不超过 50 个字符」是可判定的；「写得清楚」不是。一条断言只查一件事，失败时才知道是哪一条挂了。
  6. 最后补诚实这一层：有些判断写不出可靠断言，比如「这条评审意见有没有抓住真问题」。硬凑只会得到假绿，老实标成人工判定、抽查几条、并在结论里注明有几条是人工判的。**一份诚实的部分自动化评估远好过一份全绿的假评估。**
  7. 可预期的追问是「样本量这么小，结论站得住吗」。答话要坦率：小样本只能支撑「在这一批任务上」的结论，所以每个数字都要带测量条件，不要外推成通用效率提升。

  How to reason about it · think before answering

  1. This tests evaluation skill and honesty. Saying it felt better ends the answer; the interviewer wants a reproducible comparison.
  2. Give the structure first: one set of cases, two arms differing in exactly one variable, per-assertion judging, and a pass-rate comparison. The conclusion is a single number, the delta.
  3. Then explain how to keep the comparison clean, the half most people skip. Never test in the session you spent two hours debugging: that context is littered with convention snippets you typed and corrections you made, so good output reflects you, not the skill. Use a fresh subagent, with the two definitions differing only in which skills are preloaded.
  4. Describe the case mix: positive, boundary and negative roughly five to three to two. Negatives are non-negotiable because they measure whether the trigger surface is too wide, which is the most common way a skill goes wrong. Without them, a skill that grabs everything scores perfectly.
  5. Assertions are the core. Decidable means checking facts, not quality: the type field is one of six values, the scope equals a real directory in the repository, the first line is under fifty characters. Written clearly is not decidable. One assertion checks one thing so failures point somewhere.
  6. Close on honesty: some judgments resist reliable assertions, such as whether a review comment found the real problem. Forcing an assertion yields false green. Mark those as human-judged, sample a few, and say so in the conclusion.
  7. Expected follow-up: does a small sample support the claim? Be candid. A small sample supports a claim about that batch of tasks only, so every number carries its measurement conditions and is never extrapolated into a general efficiency gain.

  答题要点

  • 同一批用例、两组只差一个变量、逐条判定、比通过率差值。
  • 对照必须用上下文干净的子代理，不能在调试过的会话里试。
  • 两份子代理定义只差预加载 skill 那一行，模型、工具、提示词全部一致。
  • 用例要含负例，它测触发面有没有过宽，过宽是最常见的坏法。
  • 断言要可判定、一条只查一件事；判不了的老实标人工判定并在结论里注明。

  Key points

  • Same cases, two arms differing in one variable, judged per assertion, compared by pass rate.
  • The comparison needs a context-clean subagent, never the session you debugged in.
  • The two subagent definitions differ only in preloaded skills; model, tools and prompt are identical.
  • Include negative cases: they measure an over-wide trigger surface, the most common failure.
  • Assertions must be decidable and single-purpose; mark human-judged cases honestly in the conclusion.
• 让子代理带着 skill 去执行任务，和在主会话里执行有什么区别？What is the difference between running a task in a subagent with skills and running it in the main session?

  国内高频海外高频深入#agent-skills#subagent#evaluation

  分析过程 · 先想清楚再作答

  1. 这题考的是上下文隔离的价值，答得浅会变成「子代理就是开个新会话」。要说清它带来的三件事，以及每一件对应什么问题。
  2. 第一件是**上下文干净**。子代理不继承主会话的对话历史，你说过什么、纠正过什么它一概不知道。这一条在做评估时是决定性的：在调试了两小时的会话里试 skill，模型产出得好往往是因为你自己在会话里把答案说了一遍，这是评估 skill 时最常见的自欺。
  3. 第二件是**能力可以精确限定**。子代理定义里能声明可用工具、模型，也能直接声明预加载哪几个 skill。做对照时两份定义只差这一行，其它完全一致——任何第二个差异都会让结论说不清是谁的功劳。
  4. 第三件是**噪音不进主会话**。翻文件、试错、跑脚本这些过程留在子代理自己的上下文里，只把结论交回来。主会话的窗口因此能留给真正要连贯推进的那条线。
  5. 还要说清代价，只说好处会显得没做过。子代理拿不到主会话的上下文，意味着**交接摘要要写清楚**，任务描述含糊时它比主会话更容易跑偏；而且它多跑一遍系统提示与技能目录，不是免费的。
  6. 补一个实现细节能显出实感：skill 进子代理有两条路，一是在定义里预加载、启动时就注入完整正文，二是让它在执行中自己发现并激活。做对照实验用预加载，因为它把变量固定住了；做真实任务用自动发现，更接近日常。
  7. 可预期的追问是「什么时候不该用子代理」。答案是任务需要跟用户来回确认、或强依赖前面几十轮的上下文时——隔离带来的干净，这时候正好是缺陷。

  How to reason about it · think before answering

  1. This tests the value of context isolation. A shallow answer reduces it to opening a new session. Name three effects and the problem each solves.
  2. First, a clean context. A subagent does not inherit the main conversation, so it knows nothing you said or corrected. This is decisive for evaluation: testing a skill in a session you debugged for two hours usually measures your own hints, the most common self-deception here.
  3. Second, precisely bounded capability. A subagent definition can declare its tools, its model, and which skills to preload. For a controlled comparison the two definitions differ only in that line, because any second difference makes the result unattributable.
  4. Third, noise stays out. File reading, trial and error and script runs live in the subagent's own context, and only the conclusion comes back, leaving the main window for the thread that must stay coherent.
  5. Name the costs too. Without the main context, the handoff prompt must be explicit, and a vague task description sends a subagent off course faster than the main session. It also pays for its own system prompt and skill catalog.
  6. An implementation detail shows real experience: skills reach a subagent either by preloading in the definition, which injects the full body at startup, or by letting it discover and activate them during execution. Use preloading for controlled comparisons and discovery for real work.
  7. Expected follow-up: when should you not use one? When the task needs back-and-forth with the user or depends heavily on dozens of earlier turns. There, isolation is the defect rather than the feature.

  答题要点

  • 子代理有独立上下文窗口，不继承主会话历史，这是做干净对照的前提。
  • 定义里能限定工具、模型与预加载的 skill，对照时两份定义只差那一行。
  • 过程噪音留在子代理里，只把结论交回主会话。
  • 代价是交接摘要必须写清楚，任务含糊时更容易跑偏，且多付一次系统提示的开销。
  • 预加载适合做对照实验，自动发现更接近真实使用；需要与用户反复确认的任务不适合隔离。

  Key points

  • A subagent has its own context window and no inherited history, which is what makes a clean comparison possible.
  • Its definition bounds tools, model and preloaded skills, so a controlled pair differs in one line.
  • Process noise stays inside the subagent; only the conclusion returns.
  • The costs are an explicit handoff prompt, more drift on vague tasks, and paying for another system prompt.
  • Preload for controlled experiments, discovery for real work, and skip isolation when the task needs user back-and-forth.