Interview Bank

Prompt Engineering From Scratch in 5 Days

D4 Iteration and Evaluation: Small Test Sets, A/B Testing, Version Control, Common Anti-Patterns

• How do you build a test set for a prompt? How would you choose ten samples, and where do the expected answers come from?怎么给一个提示词建测试集？十条样本该怎么挑，标准答案从哪来？

  Common in ChinaCommon overseasBasic#evaluation#test-set

  How to reason about it · think before answering

  1. This screens for whether the candidate has actually built one. 'Collect some inputs and run them' means no; people who have start with distribution, because prompt errors cluster at the edges.
  2. Three classes with three or four each: normal inputs guard the baseline; edge inputs (missing defaults, optional fields, informal phrasing) test whether default rules are explicit; adversarial inputs (distractors, mid-sentence corrections, unrelated asks) test focus. Add one or two unknowable items to check honesty.
  3. Expected answers are labeled by hand, no shortcut; one mislabeled case skews the whole evaluation and sends you chasing a phantom prompt bug. Re-read each input after labeling to confirm the answer is unique.
  4. Conclusion: ten is enough to start, value lies in distribution not count, and the best source is every real 'it failed again' input from the past week.
  5. Follow-ups: how does the set grow? Add the triggering input before every prompt change. And leakage — test cases must not double as few-shot examples, or you are measuring memorization rather than generalization.

  分析过程 · 先想清楚再作答

  1. 这题在筛「有没有真的建过测试集」。答「多找一些输入跑一跑」的人没建过；建过的人第一句会说分布——因为提示词的错误全集中在边界上。
  2. 拆法：三类各占三四条。正常输入守底线，新版弄坏它们就是严重回退；边界输入（没写默认值、可选字段、不规范写法）测默认规则说清没说清；刁难输入（干扰信息、中途改口、夹带无关要求）测能不能抓住重点。可以再放一两条模型不可能知道的样本，看它是否老实说不知道。
  3. 标准答案只能人工标，这一步没有捷径；标错一条整份评估就偏，而且你会误以为是提示词的问题去反复改。每条写完再读一遍输入确认答案唯一。
  4. 结论：十条够起步，价值在分布不在数量；最好的来源是过去每一次「它又错了」的真实输入，一周就能攒出比想象出来的更真实的测试集。
  5. 可预期的追问：测试集怎么增长？每次想改提示词先把触发的那条输入加进去再改；以及「测试集会不会泄漏进提示词」——用例不能直接当 few-shot 示例，否则是在测记忆而不是泛化。

  Key points

  • Distribution over count: three or four each of normal, edge and adversarial, plus a couple of unknowable items
  • Normal cases guard the baseline, edge cases test defaults, adversarial cases test focus
  • Expected answers are hand-labeled and re-checked; one wrong label skews everything
  • Best source is real failures; add the triggering input before each prompt change

  答题要点

  • 价值在分布不在数量：正常、边界、刁难三类各三四条，再放一两条模型不可能知道的
  • 正常输入守底线，边界测默认规则，刁难测抓重点
  • 标准答案人工标注、逐条复核，标错一条整份评估就偏
  • 最好的来源是真实出错的输入；每次想改提示词先把那条加进测试集