Interview Bank

Prompt Engineering From Scratch in 5 Days

D4 Iteration and Evaluation: Small Test Sets, A/B Testing, Version Control, Common Anti-Patterns

• How do you version prompts? How does it differ from versioning code, and what is your first move when production misbehaves?提示词版本化怎么做？它和代码版本管理有什么不同？线上出问题你先做什么？

  Common in ChinaCommon overseasIntermediate#prompt-versioning#evaluation

  How to reason about it · think before answering

  1. One of the most frequent prompt-engineering interview questions abroad; it tests whether you have managed prompts as production assets. 'Put it in git' is the floor; the interviewer wants the changelog contents and why prompts differ from code.
  2. Shape first: prompt text in its own file, an id per version, decoupled from business code. Then the changelog's four items: what changed, why (which test cases failed), the measured pass rate, known regressions. The pass rate must come from the script.
  3. The difference is the differentiator: code changes are usually local, prompt changes are global — one added sentence can shift behavior on every input, so 'known regressions' is mandatory where commit messages have no such field. Also rollback is nearly free, just swap a string.
  4. Conclusion: when production misbehaves, roll back to the previous version first, then add the triggering input to the test set and investigate — which only works if you have version ids and a test set.
  5. Follow-ups: bind prompt versions to model versions? Yes — pass rates shift across model versions, so record which model was used. And canarying: route by version id and compare live metrics, same as code.

  分析过程 · 先想清楚再作答

  1. 这题是海外面试里提示词工程方向出现频率最高的一道，考的是「有没有把提示词当成生产资产管理过」。答「放进 git」是最低分，面试官要听的是变更记录里写什么、以及为什么和代码不一样。
  2. 拆法：先说形态——提示词正文独立成文件、每版有 id、与业务代码解耦；再说变更记录四件事——改了什么、为什么改（对应哪几条测试失败）、跑出来的通过率、已知回退。通过率必须是脚本跑出来的数字。
  3. 不同点是区分度所在：代码改动通常是局部的，提示词改动是全局的——加一句话可能改变所有输入的行为，所以「已知回退」是必填项而代码提交信息里没有这一栏；另一点是回滚成本几乎为零，只是换一个字符串。
  4. 结论：线上出问题第一步是回滚到上一版，再拿触发问题的输入补进测试集慢慢查——前提是你有版本号可回、有测试集可跑。
  5. 追问：提示词版本要不要和模型版本绑定？要——同一份提示词在不同模型版本上通过率会变，记录里要写清是在哪个模型上测的；另一个追问是多环境怎么灰度，答案是按版本 id 分流并对比两版的线上指标，跟代码灰度一样。

  Key points

  • Prompt text lives in its own file with a version id; the changelog records change, reason, measured pass rate, known regressions
  • Unlike code, prompt changes are global, so known regressions are mandatory; rollback is nearly free
  • On a production issue, roll back first, then add the triggering input to the test set
  • Record which model version was tested, since pass rates shift across models

  答题要点

  • 提示词正文独立成文件、每版有 id，变更记录写改动、原因、脚本跑出的通过率、已知回退
  • 与代码的不同：改动是全局的，所以「已知回退」必填；回滚成本几乎为零
  • 线上出问题先回滚上一版，再把触发输入加进测试集查
  • 版本要记录在哪个模型上测的，换模型版本通过率会变