Interview Bank

RAG in 14 Days: From Retrieval to Trustworthy Answers

D7 Week One Capstone: Assembling Six Days of Parts Into a One-Command Question-Answering Service, and a Retrospective

• What three checks would you run before shipping a retrieval QA service, and why those three?一个检索问答服务上线前你会做哪三项检查？为什么偏偏是这三项？

  Common in ChinaCommon overseasIntermediate#production-readiness#citations#refusal

  How to reason about it · think before answering

  1. The discriminator is not how many checks you list but whether you can justify the three. Ten items with no ranking suggests you have never had to prioritise.
  2. Derive them by consequence: the failures that are invisible to users and most damaging go first.
  3. First, citations must be verifiable: every cited id resolves to a real chunk, and that chunk genuinely overlaps the sentence citing it. This ranks first because a wrong citation is undetectable by the user, and citations are the only source of trust this system has.
  4. Second, refusal must actually fire: ask a question the corpus cannot answer and confirm the system says so instead of inventing. Also invisible, and one discovered fabrication zeroes out trust in the whole product.
  5. Third, ingestion-to-retrieval consistency: freshly ingested documents are retrievable immediately, and the keyword and vector paths cover the same set. This guards against the 'one route finds it, the other does not' failure, which is the hardest to diagnose.
  6. Expected follow-up: why not latency and cost? Because those failures are visible. Users complain about slowness and the bill reports overspending; nobody will ever report the three above.

  分析过程 · 先想清楚再作答

  1. 这题的区分度不在你能列几项，而在你能不能说清「为什么是这三项」。列十项而每项都不给理由，反而说明你没有排过优先级。
  2. 推导方式是按后果排序：哪种故障用户看不出来、又损失最大，哪一项就该排在前面。
  3. 第一项是引用可查证：每条引用的编号都能回查到真实存在的块，且那一块确实与该句有实质重合。这一项排第一是因为引用错了用户根本发现不了，而它恰恰是这类系统唯一的信任来源。
  4. 第二项是该拒答时真的拒答：构造一个语料里没有答案的问题，看它是回那句拒答话术还是开始编。这一项也属于用户看不出来的故障，且一旦编造被发现，整个系统的可信度归零。
  5. 第三项是摄取到检索的一致性：摄取完之后新文档立刻能被检索到，且关键词与向量两路的覆盖数量对得上。这一项防的是「一路能查一路查不到」这种最难排查的故障。
  6. 可预期的追问：为什么延迟和成本不在前三？因为它们是**看得见**的故障——慢了用户会抱怨，贵了账单会告诉你；而上面三项不检查就永远不会有人告诉你。

  Key points

  • State the ranking rule first: prioritise failures users cannot see but that cost the most.
  • Check one, verifiable citations: every id resolves to a real chunk that overlaps the sentence citing it.
  • Check two, refusal actually fires on a question the corpus cannot answer.
  • Check three, ingestion and retrieval agree: new documents are immediately retrievable on both routes.
  • Latency and cost matter but rank lower because those failures announce themselves.

  答题要点

  • 先给排序依据：优先检查用户发现不了、但后果最重的故障。
  • 第一项引用可查证：编号能回查到真实的块，且该块与被引的那句话有实质重合。
  • 第二项拒答生效：用一个语料里没有答案的问题验证系统会说查不到，而不是开始编。
  • 第三项摄取与检索一致：新入库的文档立刻可检索，关键词与向量两路覆盖对得上。
  • 延迟和成本重要但排在后面，因为它们是看得见的故障，会自己找上门。