Interview Bank

Prompt Engineering From Scratch in 5 Days

D5 Migrating Across Models: Differences Between Claude / GPT / Domestic Chinese Models, Organizing the System Prompt; Where to Go Next — the Claude Course or the Codex Course

• After migrating a prompt, how do you verify nothing regressed, and if the pass rate drops, what is your debugging order?迁移之后怎么验证效果没有退步？如果通过率降了，你的排查顺序是什么？

  Common in ChinaCommon overseasDeep dive#model-migration#evaluation

  How to reason about it · think before answering

  1. A combined D4/D5 question testing whether verification is a process. 'Run a few and see' is the floor; the interviewer wants baseline, identical cases, field-level triage.
  2. Verification needs a baseline: the same test set run on the source model beforehand with pass rate and failing cases recorded. Then run the identical cases on the target and put the columns side by side.
  3. Debugging order: normal versus edge failures first — edge failures point at the prompt; map failing fields to the four breakages and fix the vendor block, leaving the common block untouched; rerun; remaining failures that overlap the source model's are the prompt's own known regressions, unrelated to migration, handled through the D4 changelog.
  4. Conclusion: migration is done when the target matches the source pass rate with zero edits to the common block; editing the common block is a new prompt version and must be re-run on the source too.
  5. Follow-ups: single-run noise — run each case three times and take the majority, or use temperature zero. And keep dual-running for a while, because vendor model updates drift pass rates and the test set is the only thing that catches drift early.

  分析过程 · 先想清楚再作答

  1. 这题是 D4 与 D5 的合题，考的是「验证有没有流程」。答「多跑几条看看」是最低分；面试官要听的是基线、同一批用例、逐字段对号。
  2. 拆法：验证的前提是基线——迁移前在源模型上跑过同一份测试集并记录通过率与失败用例；没有基线就没有「退步」可言。迁移后用完全相同的用例在目标模型上跑，两列并排。
  3. 排查顺序：先看失败用例是正常还是边界——边界优先怀疑提示词；再把失败字段对四处断裂对号，改厂商适配块，通用块不动；再跑一遍；仍失败的用例看是否与源模型的失败重合——重合的是提示词自身的已知回退，与迁移无关，按 D4 的变更记录处理。
  4. 结论：达到与源模型相同的通过率、且通用规则块一个字没改，迁移才算完成；改了通用块就等于改了提示词版本，要重新在源模型上跑。
  5. 追问：真模型有随机性，一次运行的通过率能信吗？每条跑三次取多数或温度设 0；另一个追问是要不要在两家上长期并跑，答案是至少保留一段时间的双跑对比，因为模型版本更新会让通过率漂移，测试集是唯一能及时发现漂移的工具。

  Key points

  • Verification requires a baseline: the same test set run on the source model before migrating
  • Run identical cases on the target and compare pass rates and failing cases side by side
  • Triage: normal versus edge, map failing fields to the four breakages, fix the vendor block only, rerun, and treat failures shared with the source as known regressions
  • Any edit to the common block is a new version that must be re-run on the source; tame randomness with majority-of-three or temperature zero

  答题要点

  • 验证前提是基线：迁移前在源模型跑过同一份测试集
  • 同一批用例在目标模型上跑，两列并排看通过率与失败用例
  • 排查顺序：正常还是边界 → 失败字段对四处断裂 → 改厂商块不动通用块 → 重跑 → 与源模型重合的失败是已知回退
  • 通用块改了就是新版本，要回源模型重跑；随机性用多次取多数或温度 0 压住