Interview Bank

Prompt Engineering From Scratch in 5 Days

D2 Few-Shot, Chain of Thought, Step-by-Step, and Self-Checks; When None of These Work

• What are the signs of a task that no amount of prompt engineering will fix, and what do you do when you hit one?提示词写得再好也做不对的任务有哪些特征？遇到这类任务你会怎么办？

  Common in ChinaCommon overseasIntermediate#prompt-limits#failure-modes

  How to reason about it · think before answering

  1. This tests whether you know where prompting ends. Piling techniques onto a hopeless task signals poor judgment; saying 'this is not a prompting problem' signals maturity.
  2. Three failure classes. Missing knowledge: the fact postdates training or lives in your private data, and the model may fabricate a well-formatted answer. Missing tools: the task needs an action or query against the world. Wrong task: the requirement is contradictory or you actually want something else.
  3. One test each: could the model plausibly know something that appeared yesterday? Could a person who can only type complete this? Would two readers of the requirement do opposite things?
  4. Conclusion: paste the material or add retrieval for missing knowledge; add tool use or compute in code for missing tools; fix the requirement for a wrong task. None of these is a prompt change.
  5. Follow-ups: stricter formats make fabrications look more credible — require sources or verifiable identifiers and validate in code. And to catch these early, seed the test set with a few unknowable items and check that the model admits it does not know.

  分析过程 · 先想清楚再作答

  1. 这题考的是「知道提示词的边界在哪」。一直往提示词上堆技巧的候选人会被判为缺乏判断力；能说出「这题不该用提示词解」才是成熟的信号。
  2. 拆法：把失效分三类。缺知识——信息在模型训练截止之后或本来就在你的私有数据里，模型不可能知道，还可能编出格式正确的假答案；缺工具——任务需要对外部世界做动作或查询（跑命令、查库、发请求），文字生成做不到；任务写错——需求本身自相矛盾或者你要的其实是另一件事。
  3. 每类给一个判据：缺知识问「这信息是昨天才出现的，模型有可能知道吗」；缺工具问「一个只能打字的人能完成这件事吗」；任务写错问「两个人读这个需求会不会得出相反的做法」。
  4. 结论：缺知识就把资料贴进上下文或接检索；缺工具就接工具调用或在代码里做完再让模型解读；任务写错回去改需求。三种都不是提示词层面的解法。
  5. 追问几乎必然是「格式越严格假答案越像真的怎么办」——答案是对事实类输出要求带出处或可验证的标识，并在代码里校验；以及「怎么在评估里提前发现这类任务」，答案是测试集里放几条模型不可能知道的样本，看它是否老实说不知道。

  Key points

  • Three failure classes: missing knowledge, missing tools, and a wrongly specified task
  • Missing knowledge is dangerous because the model fabricates well-formatted answers, and stricter formats make them more convincing
  • Fixes live outside the prompt: paste material or add retrieval, add tool use or compute in code, or fix the requirement
  • Seed the test set with unknowable items to check the model admits ignorance

  答题要点

  • 三类失效：缺知识（截止日期之后或私有数据）、缺工具（需要对外部世界做动作）、任务写错（需求自相矛盾）
  • 缺知识的危险在于模型会编出格式正确的假答案，格式越严越像真的
  • 解法都在提示词之外：贴资料或接检索、接工具调用或代码先算、回去改需求
  • 测试集里放几条模型不可能知道的样本，检查它会不会老实说不知道