Interview Bank

RAG in 14 Days: From Retrieval to Trustworthy Answers

D6 The Generation Side: Ordering Context, Labeling Citations, When You Must Refuse to Answer, and Streaming Responses

• How do you make sure a model's citations are real rather than fabricated? Describe a scheme that does not rely on the model behaving well.怎么让模型的引用是真的而不是编的？说出一个不依赖模型自觉的方案。

  Common in ChinaCommon overseasIntermediate#citation-verification#grounding#hallucination

  How to reason about it · think before answering

  1. The phrase to catch is 'not relying on the model behaving well'. Any answer that boils down to 'tell the model to be accurate in the prompt' fails, because the prompt is exactly the part that cannot enforce this.
  2. Split the problem in two. Verifiability requires that a citation be a symbol from a closed set, not free text. So step one is numbering the blocks at assembly time and telling the model it may only cite the numbers it was given. 'According to the storage handbook' cannot be checked, because the title is a string the model can invent.
  3. Step two is post-hoc checking, with two gates. Gate one is existence: you handed out 1 through 5, so an 8 is fabricated, and that is a one-line check. Gate two is substantive overlap, which catches the sneakier case where the number is real but the block says something else. Measure what fraction of the sentence's terms appear in the cited block and reject below a threshold.
  4. Mention the trap in the overlap metric: drop terms that appear in most blocks first, otherwise generic words let any citation pass. It is the same reasoning behind inverse document frequency in BM25.
  5. On failure, feed the specific reason back and regenerate once, not repeatedly. Two fabricated drafts in a row means the material does not support the question, so refuse instead. Also verify against the original chunk text, never against a compressed or rewritten version, otherwise 'verified' says nothing about what the user sees.
  6. Expected follow-up: why not ask the model to self-check? Self-checking shares the generator's bias and has no independent source of truth, whereas number checking is deterministic, essentially free, and reproducible.

  分析过程 · 先想清楚再作答

  1. 题眼在「不依赖模型自觉」这半句。回答里只要出现「在提示词里强调请确保引用准确」，这题就答砸了——面试官问的正是提示词管不住的那部分。
  2. 先把问题拆成两半：引用要能验证，前提是它是一个**闭集里的符号**，不是一段自由文本。所以第一步是组装上下文时给每块材料一个编号，提示词里明确只能引用发出去的编号。让模型写「根据《某某手册》」是没法验证的，标题是它可以随口生成的字符串。
  3. 第二步是事后核对，两道闸缺一不可。第一道查编号存在性：发出去的是 1 到 5，出现 8 就一定是编的，一行代码判掉。第二道查实质重合：编号是真的、内容却对不上，这类更隐蔽，要算这句话的词元有多大比例能在被引块原文里找到，低于阈值判不通过。
  4. 算重合度时有个坑要主动说出来：先剔掉在多数块里都出现的高频词元，否则「文件」「系统」这种词会让随便哪一块都及格。这跟 BM25 用逆文档频率压常见词是同一个道理。
  5. 校验不过怎么办：把具体原因写成反馈打回去重生成一次，只给一次机会；连着两版都编说明材料本来就不支持，该走拒答而不是第三次重试。另外校验必须拿原文比对，不能拿压缩或改写过的材料比对，否则「校验通过」保证不了用户点开看到的东西。
  6. 可预期的追问：为什么不让模型自己再检查一遍？因为自检和生成是同一个模型的同一种倾向，它对自己编的东西没有独立信息源；而编号核对是一个确定性判断，成本几乎为零、结果可复现，这两点自检都做不到。

  Key points

  • Citations must be closed-set symbols such as block numbers, not free-text titles: verifiability comes from the closed set, not from wording.
  • Two gates: the number must exist, and the sentence must substantively overlap the cited block's original text, which is what catches real-number-wrong-content fabrication.
  • Strip terms that occur in most blocks before scoring overlap, or any citation will pass.
  • On failure, regenerate once with the concrete reason fed back; two bad drafts means refuse instead.
  • Always verify against the original text the user can open, never against a compressed or rewritten copy.

  答题要点

  • 引用必须是块编号这种闭集符号，不能是自由文本的文档标题——可验证性来自闭集，不来自措辞。
  • 两道闸：编号存在性，以及这句话与被引块原文的实质重合度，后者才拦得住「编号是真的、内容对不上」。
  • 算重合度前剔掉在多数块里都出现的高频词元，否则随便引哪一块都能及格。
  • 校验不过就带着具体原因打回重生成一次，只给一次机会，两版都编就转拒答。
  • 校验对象必须是用户能点开看到的原文，不是压缩或改写后的材料。