面试题库

14 天 RAG：从检索到可信回答

D14 综合项目与复盘：多租户企业知识库问答，一张 RAG 决策地图与面试专题

• RAG 系统上线后用户反馈「答得不准」，你的排查顺序是什么？Users report that your live RAG system 'answers inaccurately'. What is your triage order?

  国内高频海外高频进阶#debugging#failure-modes#observability

  分析过程 · 先想清楚再作答

  1. 这题几乎是必考题，而绝大多数人答成一堆并列的可能性：可能是切块问题、可能是提示词问题、可能是模型不行。并列不是排查，排查的意思是**有顺序、有判据、每一步能把可能性砍掉一半**。
  2. 先把「答得不准」这四个字拆开——它至少塞了四种病，而且修法互不通用：答非所问、只答得出片段、引用错位、更新不生效。所以第一个动作不是改配置，是**拿到具体的问题和回答，把它归到这四类里的一类**。
  3. 然后给顺序，而且要说清顺序的理由：**排查从右往左看、修复从左往右修**。从右往左是因为你最先看到的是生成结果；从左往右是因为上游的错会被下游放大——检索没捞到的东西，再好的提示词也救不回来。具体走法是：打印这一问的候选池和最终上下文，先看答案文档在不在候选池里。不在，是检索的债；在候选池但没过准入门槛，是门槛的债；过了门槛却没装进上下文预算，是块太大或预算太小；都进了而模型没用上，才轮到生成侧。
  4. 这里有一个容易写错的细节值得主动讲：**多跳题的诊断对象是缺的那几篇，不是「有没有捞到任意一篇」**。我们实验里有一道题要同时命中两篇，第一篇稳稳排第一、第二篇一次都没进候选池；用「任意一篇」去判会把它归成预算问题，然后你去调预算，调一整天也没用。这一条区分度很高，因为它只有真的按题排查过才想得到。
  5. 第四类「更新不生效」发生在问答之外，判据是另一条：先看对账认没认出这篇改了（内容指纹算之前有没有做换行归一化），再看缓存的 key 里有没有把索引版本和权限范围算进去。「什么时候必须失效」等价于「key 里有没有把那样东西算进去」，key 少放一样，那样东西变了缓存就不会失效。
  6. 可预期的追问：怎么让这套排查不靠人肉？答案是把分类做进评估面板——每一道没中的题自动标出它属于四类中的哪一类，并按租户分开统计。全局平均会把单个客户的塌方按人头摊薄，而线上会投诉的恰恰是那个客户。

  How to reason about it · think before answering

  1. This one is almost guaranteed to be asked, and most people answer with a flat list of possibilities: maybe chunking, maybe the prompt, maybe the model. A list is not triage. Triage means an order, a decision rule at each step, and each step eliminating half the search space.
  2. First decompose the complaint. 'Inaccurate' hides at least four distinct failures whose fixes do not transfer: off-topic answers, partial answers, misaligned citations, and stale content. So the first action is not to change a setting, it is to obtain the specific question and answer and classify it into one of those four.
  3. Then give the order along with its justification: read the pipeline right to left, fix it left to right. Right to left because the generated answer is what you see first; left to right because upstream errors are amplified downstream — no prompt can recover a document retrieval never fetched. Concretely: dump the candidate pool and the final context for that question, and check whether the answer document is in the pool at all. Absent means a retrieval debt; present but below the admission gate means a gate debt; admitted but never packed into the context budget means chunks too large or budget too small; all present and still unused means it is finally a generation problem.
  4. One detail worth volunteering because it is easy to get wrong: for multi-hop questions, diagnose the documents that are missing, not whether any one of them was retrieved. In our experiment one question needed two documents; the first ranked first every time and the second never entered the candidate pool at all. Judging by 'any of them' labels it a budget problem, and you can spend a full day tuning budgets to no effect. This distinction only occurs to someone who has actually triaged question by question.
  5. The fourth class, stale content, happens outside the question path and has its own rule: first check whether reconciliation even noticed the edit (was the content hash computed after line-ending normalization?), then check whether the cache key includes the index version and the permission scope. 'When must this expire' is equivalent to 'is that thing part of the key' — leave something out of the key and changes to it will never invalidate the entry.
  6. Expected follow-up: how do you stop relying on manual triage? Build the classification into the evaluation panel so every missed question is automatically labelled with one of the four classes, and report it per tenant. A global average dilutes one customer's collapse across the whole population, and that customer is exactly the one who will file the complaint.

  答题要点

  • 先把「答得不准」归类成四种病：答非所问、只答得出片段、引用错位、更新不生效——修法互不通用。
  • 排查从右往左看、修复从左往右修：先打印候选池与最终上下文，看答案文档卡在哪一层。
  • 四层判据依次是：没进候选池、进了没过门槛、过了没装进预算、都进了模型没用上。
  • 多跳题只诊断缺的那几篇；用「有没有捞到任意一篇」会把「根本没捞到」误判成预算问题。
  • 「更新不生效」查对账与缓存 key：什么时候必须失效，等价于 key 里有没有算进那样东西。

  Key points

  • Classify the complaint into four failures first — off-topic, partial, misaligned citation, stale — because their fixes do not transfer.
  • Read right to left, fix left to right: dump the candidate pool and final context and find which layer the answer document stalls at.
  • The four rules in order: never retrieved, retrieved but below the gate, admitted but squeezed out of the budget, packed but unused by the model.
  • For multi-hop, diagnose only the missing documents; judging by 'any one retrieved' mislabels a never-retrieved case as a budget problem.
  • For stale content, check reconciliation and the cache key: what must expire is exactly what the key must contain.