面试题库

5 天提示词工程零基础

D4 迭代与评估：小样本测试集、A/B、版本管理、常见反模式

• 用模型给模型打分靠谱吗？什么时候可以用，什么时候必须人工看？Is using a model to grade another model's output reliable? When is it acceptable, and when must a human look?

  国内高频海外高频进阶#evaluation#llm-as-judge

  分析过程 · 先想清楚再作答

  1. 这题在考「知道裁判也会错」。答「用更强的模型当裁判就行」的人没校准过裁判；答出「能用字段比对就不用裁判」才说明有判断力。
  2. 拆法：先分任务。输出是固定字段就用代码逐字段比对，不需要裁判；输出是自由文本（摘要、邮件、解释）才没有字段可比，这时裁判是唯一能规模化的办法。
  3. 再说裁判的偏差：偏向长的、格式漂亮的、和自己风格接近的回答；评分标准含糊时打分随意；对事实性错误不敏感。所以评分标准要像便签一样具体——列出信息点、每点一分——而不是「给这段摘要打 1 到 10 分」。
  4. 结论：裁判可以用，前提是先拿十条人工打过分的样本校准它，看它和人的一致率；上线后定期抽样复核；对涉及事实、安全、金额的输出必须人工看。
  5. 追问方向：裁判和被评的模型是同一家会怎样？会有自我偏好，尽量换一家或至少换一个版本；另一个追问是「裁判的成本」，每条评估都是一次完整调用，测试集大了要算钱，所以能用代码判的部分先用代码判掉。

  How to reason about it · think before answering

  1. This tests whether you know the judge is fallible too. 'Use a stronger model as the judge' means you never calibrated one; 'prefer field comparison whenever possible' shows judgment.
  2. Split by task: structured output gets field-by-field code comparison, no judge needed; free text (summaries, emails, explanations) has no fields, and a judge is the only scalable option.
  3. Judge biases: longer and prettier answers score higher, stylistic similarity gets rewarded, vague rubrics produce noisy scores, factual errors are under-penalized. So the rubric must be concrete — list the information points, one point each — not 'rate this summary 1 to 10'.
  4. Conclusion: usable once calibrated against ten human-scored samples with an agreement rate you accept; spot-check regularly; anything involving facts, safety or money still gets human review.
  5. Follow-ups: same vendor for judge and judged? Expect self-preference, so switch vendor or at least version. And cost — every judgment is a full call, so let code handle whatever it can first.

  答题要点

  • 能用字段比对就不用裁判；裁判只用于没有字段可比的自由文本
  • 裁判偏向长的、格式漂亮的回答，评分标准含糊就打得随意，所以标准要具体到信息点
  • 先用十条人工打分样本校准裁判，上线后定期抽样复核
  • 涉及事实、安全、金额的输出必须人工看；裁判尽量换一家或换版本以避免自我偏好

  Key points

  • Prefer field comparison; reserve the judge for free text with nothing to compare
  • Judges favor long, well-formatted answers and score noisily on vague rubrics, so rubrics must list concrete points
  • Calibrate against ten human-scored samples first, then spot-check regularly
  • Facts, safety and money always get human review; use a different vendor or version to avoid self-preference