Interview Bank

RAG in 14 Days: From Retrieval to Trustworthy Answers

D7 Week One Capstone: Assembling Six Days of Parts Into a One-Command Question-Answering Service, and a Retrospective

• What is the biggest risk in the RAG service you just assembled, and how would you prove that judgment?你刚拼出来的这个检索问答系统，现在最大的风险在哪里？你打算怎么证明这个判断？

  Common in ChinaCommon overseasDeep dive#evaluation#risk-assessment#retrospective

  How to reason about it · think before answering

  1. There are two halves here and the second is the real question. Naming a risk is easy; giving a method that could falsify your own claim is what separates answers from opinions.
  2. Rule out two common wrong answers: 'hallucination' is too vague to act on, and 'latency' mistakes a visible problem for the biggest one.
  3. The biggest risk is the absence of evaluation. Chunk size, top-k, thresholds and route weights were all guessed, and that makes every other risk unverifiable: you cannot even say whether a change helped.
  4. How to prove it: build a question set from the corpus with known answer documents, deliberately including unanswerable and multi-hop questions; implement recall and ranking metrics; produce a baseline for the current configuration; then move one parameter back and forth and watch whether the metrics move. If they do not move at all, the evaluation set is wrong, not the system.
  5. Add the accounting rule: every optimisation reports three numbers, metric gain, latency added and cost added. A claim with only the first is not usable.
  6. Expected follow-up: how large must the set be? Start with roughly twenty questions covering the main question types to catch obvious regressions, then grow toward the real distribution once you have actual user questions. Chasing size first only yields questions you invented yourself.

  分析过程 · 先想清楚再作答

  1. 这题有两半，后半句才是题眼。说出一个风险不难，难的是给出一个能证伪你自己判断的方法——答不出后半句，前半句就只是意见。
  2. 先排除两个常见的错误答案：说「幻觉」太笼统，没有指向任何可动的地方；说「延迟」则是把看得见的问题当成最大风险。
  3. 真正的最大风险是**没有评估**：切块大小、取几条、门槛定多少、两路怎么加权，全是拍出来的。它最重要的地方在于它让所有其他风险都无法验收——你连「改了之后变好还是变坏」都说不出口。
  4. 怎么证明：先从语料反向出一份带标准答案文档的问题集，刻意掺进无答案问题和需要跨文档的多跳问题；再实现召回率与排序指标，给当前配置跑出一个基线；然后把一个参数来回改两次，看指标动不动。如果指标对参数完全不敏感，说明是评估集有问题，不是系统没问题。
  5. 补一句成本口径：每一项优化都要同时报三笔账——指标涨了多少、延迟涨了多少、钱涨了多少。只报第一笔的结论不能用。
  6. 可预期的追问：评估集多大才够？先做二十题能覆盖主要问题类型的小集，用它挡住明显的退步；等真实用户问题攒起来，再按真实分布扩到几百题。一上来就追求规模，只会得到一堆自己出的、跟真实用法无关的题。

  Key points

  • The biggest risk is having no evaluation: every parameter was guessed, so no change can be judged.
  • Prove it by building a golden set with known answer documents, including unanswerable and multi-hop questions, then baseline the current configuration.
  • Validate the set itself by perturbing parameters: metrics that never move mean the questions are wrong.
  • Report three numbers per optimisation: metric gain, added latency, added cost.
  • Start small but well covered, then grow toward the real question distribution.

  答题要点

  • 最大的风险是没有评估：所有参数都是拍的，导致任何改动的好坏都无法判断。
  • 证明方式是先建标准答案集，刻意包含无答案问题与多跳问题，再跑出当前配置的基线。
  • 用参数扰动反过来验证评估集本身：指标对参数完全不敏感，说明题出得有问题。
  • 每项优化同时报三笔账：指标、延迟、成本；只报指标的结论不能用。
  • 评估集先小而全，覆盖问题类型即可，等真实问题攒起来再按真实分布扩大。

D14 Capstone Project and Retrospective: A Multi-Tenant Enterprise Knowledge-Base Q&A, a RAG Decision Map, and an Interview Deep Dive

• Looking back at the RAG project you built, which decision would you change now, and why?你做过的这个 RAG 项目里，哪个决定你现在会改？为什么？

  Common in ChinaCommon overseasDeep dive#retrospective#evidence#chunking

  How to reason about it · think before answering

  1. This looks like a soft question but it separates people sharply. Saying 'nothing yet' admits you never ran a retrospective; a long list of self-criticism reads as poor judgement. What the interviewer is listening for is whether you can chain four things together: the decision, the evidence you had then, the evidence you got later, and your current call.
  2. How to pick: choose a decision that was justified at the time and later overturned by data, not one you always knew was a shortcut. The first proves you measure; the second only proves you were behind schedule. So the answer has a fixed four-part shape — what you chose, on what basis, what you measured later, and what you now believe.
  3. This course supplies a ready example. One day measured that prepending a heading-path header to every chunk left hit rate unchanged, grew index tokens by about ten percent, and pushed the answer document's mean rank from 2.88 to 3.25 — hence 'headers hurt keyword retrieval'. A later day re-ran the same comparison under structure-aware chunking and the rank regression did not reproduce. The reason was the chunker: with fixed-length cuts, chunk boundaries do not line up with section boundaries, so the header injects heading terms that do not belong to that chunk; with structure-aware cuts, each chunk already sits inside one section and the header largely restates what is already there. The correct statement is therefore not 'headers hurt' but 'headers hurt when chunk boundaries are misaligned with document structure'.
  4. The value of the chain is that it demonstrates a reusable habit: attach the premises to every conclusion. Change a premise and you owe a re-run; you may not pair new settings with an old conclusion. The same reasoning yields a second example: an earlier claim that 'the vector route is clearly a net gain' collapsed once the two routes were counted separately before fusion — the vector-only candidates were a small share and contained the answer document zero times, so the improvement was never semantic at all.
  5. Expected follow-up: how will you avoid this class of error in future? Give two concrete practices. Write the bound premises next to every number — corpus, question set, budget, chunker. And before publishing any conclusion, ask whether it was measured by this experiment or forced by the structure of the implementation; the first needs its boundaries stated, only the second can be asserted flatly.

  分析过程 · 先想清楚再作答

  1. 这题看着是软性问题，其实区分度极高。答「暂时没有」等于承认没做过复盘；答成一长串自我批评又会显得没有判断力。面试官真正在听的是：你能不能把一个决定、它当时的依据、后来的证据、以及新的判断，四样东西串成一条链子说清楚。
  2. 怎么拆：挑一个**当时有理由、后来被数据推翻**的决定，而不是一个「当时就知道是凑合」的决定。前者证明你有量化的习惯，后者只证明你赶过工期。所以答案的骨架固定是四段——当时选了什么、依据是什么、后来量到了什么、现在的判断是什么。
  3. 本课里有一个现成的样本：某一天先量到「给每个块拼上标题块头之后，命中率不变、索引 token 涨一成、答案文档平均名次从 2.88 退到 3.25」，据此写下「块头对关键词检索是负收益」。后来换成按文档结构切块再复核，这条名次退化**没有复现**。原因是切法变了：固定长度硬切时块边界跟小节边界不对齐，块头会把不属于这一块的标题词塞进来；按结构切时块本身就落在一个小节里，块头补的信息跟块里已有的高度重合。所以正确的表述不是「块头有害」，而是「**块头在块边界与结构不对齐时才有害**」。
  4. 这条链子的价值在于它演示了一个可复用的动作：**给每个结论标出它绑定的前提**。前提变了就要重跑，不能拿新配置去配旧结论。顺着这个思路还能给出第二个例子：曾经写过「向量侧确实是正收益」，后来把融合前的两路拆开数了一遍才发现，向量路独有的候选只占很小一部分，其中含答案文档的次数是零——那个「涨」根本不是语义检索带来的，于是这条结论被自己推翻。
  5. 可预期的追问：那你以后怎么避免这类错误？答两条具体的：一是每个数字旁边写清它绑定了哪几个前提（语料、题集、预算、切法），二是报结论前先问自己一句「这是这次实验测出来的，还是这个结构必然导致的」——前者要标边界，后者才能直接讲。

  Key points

  • Structure the answer in four beats: the choice, the evidence then, the evidence later, the call now.
  • Pick a decision that was defensible at the time and later overturned by data, not one you knew was a shortcut.
  • Worked example: 'headers hurt keyword retrieval' was corrected to 'headers hurt when chunk boundaries misalign with structure', because the chunker premise changed.
  • Record the premises bound to every number; when a premise changes you owe a re-run rather than a reinterpretation.
  • Classify before asserting: measured by this experiment, or forced by the implementation's structure — the former needs its boundaries stated.

  答题要点

  • 答案要串成四段：当时选了什么、依据是什么、后来量到了什么、现在的判断是什么。
  • 挑一个当时有理由、后来被数据推翻的决定，而不是一个当时就知道在凑合的决定。
  • 样本：块头从「对关键词检索有害」修正成「块边界与结构不对齐时才有害」，因为切法这个前提变了。
  • 每个数字旁边写清它绑定的前提；前提变了就必须重跑，不能新配置配旧结论。
  • 报结论前先分类：这是实验测出来的，还是实现结构必然导致的——前者要标边界。