面试题库

Codex 与 OpenAI Agents SDK 高效使用

D5 Claude 与 Codex 选型与协作：同一任务双工具实测对比、一个写一个审的混用工作流

• 怎么评价一个 coding agent 这次任务的输出质量，而不只是看它跑没跑通？How do you judge the quality of a coding agent's output on a task, beyond whether it ran?

  国内高频海外高频深入#coding-agent#evaluation#quality

  分析过程 · 先想清楚再作答

  1. 这题考的是你有没有把「测试绿了」当终点。答「看测试」是及格线，区分度在测试之外。
  2. 拆成四层：正确性（测试是否覆盖了需求里的边界，比如 title 超长、done 传字符串）、契约（错误响应形状是否与需求一字不差，还是它自作主张改了）、范围（有没有改不该改的文件、有没有偷偷加依赖或改默认值）、可维护性（校验规则是否抽成常量、测试是否隔离、命名是否与仓库一致）。
  3. 再说怎么量：正确性看它写的测试之外你再补的反例能不能过；契约与范围看 diff 与需求逐条对照；可维护性交给第二家模型或人做结构化审查。
  4. 补一条随机性：单次结果不能下结论，同一需求跑三次看方差，方差大本身就是一个质量信号。
  5. 可预期的追问：它自己说「已完成并通过测试」能信吗？只信你能复现的部分——在你的机器上重跑测试、看 diff，agent 的汇报是线索不是证据。

  How to reason about it · think before answering

  1. This tests whether you treat green tests as the finish line; 'check the tests' is the pass mark, differentiation lies beyond it.
  2. Four layers: correctness (do the tests cover the requirement's edges such as overly long titles or a string for done), contract (does the error shape match the spec exactly or did it improvise), scope (did it touch forbidden files, add dependencies or change defaults silently), maintainability (constants extracted, tests isolated, naming consistent with the repo).
  3. How to measure: correctness by adding your own counterexamples beyond its tests; contract and scope by diffing against the requirement line by line; maintainability via a structured review by a second model or a person.
  4. Add variance: one run proves nothing; run the same requirement three times and treat high variance as a quality signal in itself.
  5. Expect the follow-up: can you trust its 'done, tests pass'? Only what you can reproduce; rerun tests and read the diff yourself, the agent's report is a lead, not evidence.

  答题要点

  • 四层：正确性、契约、范围、可维护性，测试绿只是正确性的一部分
  • 正确性用自己补的反例验证，契约与范围对照需求逐条看 diff，可维护性做结构化审查
  • 同一需求跑多次看方差，方差大本身是质量信号
  • agent 的汇报是线索不是证据，只信自己能复现的部分

  Key points

  • Four layers: correctness, contract, scope, maintainability; green tests cover only part of correctness
  • Verify correctness with your own counterexamples, contract and scope by diffing against the spec, maintainability via structured review
  • Run the same requirement several times; high variance is itself a quality signal
  • The agent's report is a lead, not evidence; trust only what you reproduce