面试题库

5 天提示词工程零基础

D4 迭代与评估：小样本测试集、A/B、版本管理、常见反模式

• 怎么给一个提示词建测试集？十条样本该怎么挑，标准答案从哪来？How do you build a test set for a prompt? How would you choose ten samples, and where do the expected answers come from?

  国内高频海外高频基础#evaluation#test-set

  分析过程 · 先想清楚再作答

  1. 这题在筛「有没有真的建过测试集」。答「多找一些输入跑一跑」的人没建过；建过的人第一句会说分布——因为提示词的错误全集中在边界上。
  2. 拆法：三类各占三四条。正常输入守底线，新版弄坏它们就是严重回退；边界输入（没写默认值、可选字段、不规范写法）测默认规则说清没说清；刁难输入（干扰信息、中途改口、夹带无关要求）测能不能抓住重点。可以再放一两条模型不可能知道的样本，看它是否老实说不知道。
  3. 标准答案只能人工标，这一步没有捷径；标错一条整份评估就偏，而且你会误以为是提示词的问题去反复改。每条写完再读一遍输入确认答案唯一。
  4. 结论：十条够起步，价值在分布不在数量；最好的来源是过去每一次「它又错了」的真实输入，一周就能攒出比想象出来的更真实的测试集。
  5. 可预期的追问：测试集怎么增长？每次想改提示词先把触发的那条输入加进去再改；以及「测试集会不会泄漏进提示词」——用例不能直接当 few-shot 示例，否则是在测记忆而不是泛化。

  How to reason about it · think before answering

  1. This screens for whether the candidate has actually built one. 'Collect some inputs and run them' means no; people who have start with distribution, because prompt errors cluster at the edges.
  2. Three classes with three or four each: normal inputs guard the baseline; edge inputs (missing defaults, optional fields, informal phrasing) test whether default rules are explicit; adversarial inputs (distractors, mid-sentence corrections, unrelated asks) test focus. Add one or two unknowable items to check honesty.
  3. Expected answers are labeled by hand, no shortcut; one mislabeled case skews the whole evaluation and sends you chasing a phantom prompt bug. Re-read each input after labeling to confirm the answer is unique.
  4. Conclusion: ten is enough to start, value lies in distribution not count, and the best source is every real 'it failed again' input from the past week.
  5. Follow-ups: how does the set grow? Add the triggering input before every prompt change. And leakage — test cases must not double as few-shot examples, or you are measuring memorization rather than generalization.

  答题要点

  • 价值在分布不在数量：正常、边界、刁难三类各三四条，再放一两条模型不可能知道的
  • 正常输入守底线，边界测默认规则，刁难测抓重点
  • 标准答案人工标注、逐条复核，标错一条整份评估就偏
  • 最好的来源是真实出错的输入；每次想改提示词先把那条加进测试集

  Key points

  • Distribution over count: three or four each of normal, edge and adversarial, plus a couple of unknowable items
  • Normal cases guard the baseline, edge cases test defaults, adversarial cases test focus
  • Expected answers are hand-labeled and re-checked; one wrong label skews everything
  • Best source is real failures; add the triggering input before each prompt change
• 提示词版本化怎么做？它和代码版本管理有什么不同？线上出问题你先做什么？How do you version prompts? How does it differ from versioning code, and what is your first move when production misbehaves?

  国内高频海外高频进阶#prompt-versioning#evaluation

  分析过程 · 先想清楚再作答

  1. 这题是海外面试里提示词工程方向出现频率最高的一道，考的是「有没有把提示词当成生产资产管理过」。答「放进 git」是最低分，面试官要听的是变更记录里写什么、以及为什么和代码不一样。
  2. 拆法：先说形态——提示词正文独立成文件、每版有 id、与业务代码解耦；再说变更记录四件事——改了什么、为什么改（对应哪几条测试失败）、跑出来的通过率、已知回退。通过率必须是脚本跑出来的数字。
  3. 不同点是区分度所在：代码改动通常是局部的，提示词改动是全局的——加一句话可能改变所有输入的行为，所以「已知回退」是必填项而代码提交信息里没有这一栏；另一点是回滚成本几乎为零，只是换一个字符串。
  4. 结论：线上出问题第一步是回滚到上一版，再拿触发问题的输入补进测试集慢慢查——前提是你有版本号可回、有测试集可跑。
  5. 追问：提示词版本要不要和模型版本绑定？要——同一份提示词在不同模型版本上通过率会变，记录里要写清是在哪个模型上测的；另一个追问是多环境怎么灰度，答案是按版本 id 分流并对比两版的线上指标，跟代码灰度一样。

  How to reason about it · think before answering

  1. One of the most frequent prompt-engineering interview questions abroad; it tests whether you have managed prompts as production assets. 'Put it in git' is the floor; the interviewer wants the changelog contents and why prompts differ from code.
  2. Shape first: prompt text in its own file, an id per version, decoupled from business code. Then the changelog's four items: what changed, why (which test cases failed), the measured pass rate, known regressions. The pass rate must come from the script.
  3. The difference is the differentiator: code changes are usually local, prompt changes are global — one added sentence can shift behavior on every input, so 'known regressions' is mandatory where commit messages have no such field. Also rollback is nearly free, just swap a string.
  4. Conclusion: when production misbehaves, roll back to the previous version first, then add the triggering input to the test set and investigate — which only works if you have version ids and a test set.
  5. Follow-ups: bind prompt versions to model versions? Yes — pass rates shift across model versions, so record which model was used. And canarying: route by version id and compare live metrics, same as code.

  答题要点

  • 提示词正文独立成文件、每版有 id，变更记录写改动、原因、脚本跑出的通过率、已知回退
  • 与代码的不同：改动是全局的，所以「已知回退」必填；回滚成本几乎为零
  • 线上出问题先回滚上一版，再把触发输入加进测试集查
  • 版本要记录在哪个模型上测的，换模型版本通过率会变

  Key points

  • Prompt text lives in its own file with a version id; the changelog records change, reason, measured pass rate, known regressions
  • Unlike code, prompt changes are global, so known regressions are mandatory; rollback is nearly free
  • On a production issue, roll back first, then add the triggering input to the test set
  • Record which model version was tested, since pass rates shift across models
• 用模型给模型打分靠谱吗？什么时候可以用，什么时候必须人工看？Is using a model to grade another model's output reliable? When is it acceptable, and when must a human look?

  国内高频海外高频进阶#evaluation#llm-as-judge

  分析过程 · 先想清楚再作答

  1. 这题在考「知道裁判也会错」。答「用更强的模型当裁判就行」的人没校准过裁判；答出「能用字段比对就不用裁判」才说明有判断力。
  2. 拆法：先分任务。输出是固定字段就用代码逐字段比对，不需要裁判；输出是自由文本（摘要、邮件、解释）才没有字段可比，这时裁判是唯一能规模化的办法。
  3. 再说裁判的偏差：偏向长的、格式漂亮的、和自己风格接近的回答；评分标准含糊时打分随意；对事实性错误不敏感。所以评分标准要像便签一样具体——列出信息点、每点一分——而不是「给这段摘要打 1 到 10 分」。
  4. 结论：裁判可以用，前提是先拿十条人工打过分的样本校准它，看它和人的一致率；上线后定期抽样复核；对涉及事实、安全、金额的输出必须人工看。
  5. 追问方向：裁判和被评的模型是同一家会怎样？会有自我偏好，尽量换一家或至少换一个版本；另一个追问是「裁判的成本」，每条评估都是一次完整调用，测试集大了要算钱，所以能用代码判的部分先用代码判掉。

  How to reason about it · think before answering

  1. This tests whether you know the judge is fallible too. 'Use a stronger model as the judge' means you never calibrated one; 'prefer field comparison whenever possible' shows judgment.
  2. Split by task: structured output gets field-by-field code comparison, no judge needed; free text (summaries, emails, explanations) has no fields, and a judge is the only scalable option.
  3. Judge biases: longer and prettier answers score higher, stylistic similarity gets rewarded, vague rubrics produce noisy scores, factual errors are under-penalized. So the rubric must be concrete — list the information points, one point each — not 'rate this summary 1 to 10'.
  4. Conclusion: usable once calibrated against ten human-scored samples with an agreement rate you accept; spot-check regularly; anything involving facts, safety or money still gets human review.
  5. Follow-ups: same vendor for judge and judged? Expect self-preference, so switch vendor or at least version. And cost — every judgment is a full call, so let code handle whatever it can first.

  答题要点

  • 能用字段比对就不用裁判；裁判只用于没有字段可比的自由文本
  • 裁判偏向长的、格式漂亮的回答，评分标准含糊就打得随意，所以标准要具体到信息点
  • 先用十条人工打分样本校准裁判，上线后定期抽样复核
  • 涉及事实、安全、金额的输出必须人工看；裁判尽量换一家或换版本以避免自我偏好

  Key points

  • Prefer field comparison; reserve the judge for free text with nothing to compare
  • Judges favor long, well-formatted answers and score noisily on vague rubrics, so rubrics must list concrete points
  • Calibrate against ten human-scored samples first, then spot-check regularly
  • Facts, safety and money always get human review; use a different vendor or version to avoid self-preference

D5 跨模型迁移：Claude / GPT / 国产模型的差异、system prompt 组织；进入 Claude 课与 Codex 课

• 迁移之后怎么验证效果没有退步？如果通过率降了，你的排查顺序是什么？After migrating a prompt, how do you verify nothing regressed, and if the pass rate drops, what is your debugging order?

  国内高频海外高频深入#model-migration#evaluation

  分析过程 · 先想清楚再作答

  1. 这题是 D4 与 D5 的合题，考的是「验证有没有流程」。答「多跑几条看看」是最低分；面试官要听的是基线、同一批用例、逐字段对号。
  2. 拆法：验证的前提是基线——迁移前在源模型上跑过同一份测试集并记录通过率与失败用例；没有基线就没有「退步」可言。迁移后用完全相同的用例在目标模型上跑，两列并排。
  3. 排查顺序：先看失败用例是正常还是边界——边界优先怀疑提示词；再把失败字段对四处断裂对号，改厂商适配块，通用块不动；再跑一遍；仍失败的用例看是否与源模型的失败重合——重合的是提示词自身的已知回退，与迁移无关，按 D4 的变更记录处理。
  4. 结论：达到与源模型相同的通过率、且通用规则块一个字没改，迁移才算完成；改了通用块就等于改了提示词版本，要重新在源模型上跑。
  5. 追问：真模型有随机性，一次运行的通过率能信吗？每条跑三次取多数或温度设 0；另一个追问是要不要在两家上长期并跑，答案是至少保留一段时间的双跑对比，因为模型版本更新会让通过率漂移，测试集是唯一能及时发现漂移的工具。

  How to reason about it · think before answering

  1. A combined D4/D5 question testing whether verification is a process. 'Run a few and see' is the floor; the interviewer wants baseline, identical cases, field-level triage.
  2. Verification needs a baseline: the same test set run on the source model beforehand with pass rate and failing cases recorded. Then run the identical cases on the target and put the columns side by side.
  3. Debugging order: normal versus edge failures first — edge failures point at the prompt; map failing fields to the four breakages and fix the vendor block, leaving the common block untouched; rerun; remaining failures that overlap the source model's are the prompt's own known regressions, unrelated to migration, handled through the D4 changelog.
  4. Conclusion: migration is done when the target matches the source pass rate with zero edits to the common block; editing the common block is a new prompt version and must be re-run on the source too.
  5. Follow-ups: single-run noise — run each case three times and take the majority, or use temperature zero. And keep dual-running for a while, because vendor model updates drift pass rates and the test set is the only thing that catches drift early.

  答题要点

  • 验证前提是基线：迁移前在源模型跑过同一份测试集
  • 同一批用例在目标模型上跑，两列并排看通过率与失败用例
  • 排查顺序：正常还是边界 → 失败字段对四处断裂 → 改厂商块不动通用块 → 重跑 → 与源模型重合的失败是已知回退
  • 通用块改了就是新版本，要回源模型重跑；随机性用多次取多数或温度 0 压住

  Key points

  • Verification requires a baseline: the same test set run on the source model before migrating
  • Run identical cases on the target and compare pass rates and failing cases side by side
  • Triage: normal versus edge, map failing fields to the four breakages, fix the vendor block only, rerun, and treat failures shared with the source as known regressions
  • Any edit to the common block is a new version that must be re-run on the source; tame randomness with majority-of-three or temperature zero