面试题库

30 天从前端工程师到 Agent 工程师

D6 消息、上下文工程与压缩、会话存储/恢复/分叉（dg M06/M08/M09/M10）

• 上下文工程和 RAG 检索是什么关系？只做 RAG 不做上下文管理会出什么问题？How do context engineering and RAG relate, and what breaks if you do RAG without context management?

  国内高频海外高频深入#context-engineering#rag#retrieval

  分析过程 · 先想清楚再作答

  1. 题眼在「关系」。把两者说成并列的两种技术是最常见的失分答法——正确的框架是包含关系：上下文工程是「决定这次请求里放什么」，RAG 是它的一种供给手段，负责「从外部捞该放进去的东西」。
  2. 拆开看职责就清楚了：RAG 解决的是「信息不在模型参数里、也不在当前对话里」，靠检索把它取回来；上下文工程解决的是「取回来的东西、加上历史、加上工具定义，一共放不放得下、该按什么优先级放」。前者管来源，后者管预算。
  3. 所以只做 RAG 不做上下文管理会出三类问题，最好按这个顺序说。第一是挤占：检索回来的文档动辄几千 token，直接拼进去把对话历史挤没了，模型记得住资料却忘了用户三句话前说过什么。第二是干扰：召回条数调大看着安全，实际上不相关的片段会稀释模型注意力，准确率不升反降。第三是成本：检索结果每一轮都重发，一段两千 token 的资料聊十轮就付了十次。
  4. 给出正确的组合姿势：先给历史和检索结果各划一条预算线，检索结果只保留 top-k 且不跨轮重复注入，历史超线就压缩，两条线加起来必须留出输出空间。这套「分账」的说法比笼统的「要平衡」有说服力得多。
  5. 可以预期的追问：检索结果该放在系统提示词里还是当成一条 user 消息？答放在靠近当前问题的位置通常效果更好，而且要标注来源便于模型区分「资料」和「用户说的话」；顺带说清它是一次性上下文，不该被写进长期会话历史里反复重发。

  How to reason about it · think before answering

  1. The hinge word is 'relate'. Treating them as two parallel techniques is the standard weak answer — the right frame is containment: context engineering decides what goes into this request, and RAG is one supply mechanism that fetches what should go in.
  2. Separate the responsibilities and it becomes obvious: RAG solves 'the information is neither in the weights nor in this conversation' by retrieving it; context engineering solves 'the retrieved chunks plus the history plus the tool definitions all have to fit, in some priority order'. One owns sourcing, the other owns budget.
  3. So RAG without context management breaks in three ways, best delivered in this order. Crowding: retrieved documents run to thousands of tokens and squeeze out the conversation, so the model knows the manual but forgot what the user said three turns ago. Interference: raising top-k feels safe but irrelevant chunks dilute attention and accuracy drops instead of rising. Cost: retrieved text is resent every turn, so a 2k-token passage costs ten times over ten turns.
  4. Give the correct combination: budget history and retrieval separately, keep retrieval to top-k without re-injecting the same chunks every turn, compress history when it crosses its line, and make sure both lines together still leave room for output. This 'separate budgets' framing lands much better than a vague 'you need to balance them'.
  5. Expect the follow-up on placement: putting retrieved context near the current question usually works better, and it should be labeled with its source so the model can tell reference material from what the user actually said. Note too that it is single-turn context and should not be written into the persisted history and resent forever.

  答题要点

  • 不是并列关系而是包含关系：上下文工程决定这次请求放什么，RAG 是给它供货的一种手段，负责把不在模型和对话里的信息检索回来
  • 只做 RAG 会挤占历史：几千 token 的检索结果把对话挤没，模型记得住资料却忘了用户刚说的话
  • 召回条数越大越准是错觉：不相关片段会稀释注意力，准确率反而下降，应控制 top-k
  • 检索结果每轮重发会持续计费，属于一次性上下文，不该写进持久化历史反复重发
  • 正确姿势是给历史和检索各划一条预算线，历史超线就压缩，两条线之外还要留出输出空间

  Key points

  • They are not parallel: context engineering decides what enters the request, and RAG is one supply mechanism for information that is neither in the weights nor in the conversation
  • RAG alone crowds out history — multi-thousand-token retrievals evict the conversation, so the model knows the docs but forgot the user's last request
  • A bigger top-k is not safer: irrelevant chunks dilute attention and accuracy drops, so cap retrieval
  • Retrieved text is single-turn context; persisting it into the history means paying for it on every subsequent turn
  • Budget history and retrieval on separate lines, compress history when it crosses its line, and leave room for the output on top of both

D12 长期记忆：pgvector、embedding、chunking、memory_search 工具

• 为什么 Agent 需要额外的长期记忆，而不是把历史全部塞进上下文？Why does an agent need a separate long-term memory instead of stuffing all history into the context window?

  国内高频海外高频基础#long-term-memory#rag#cost

  分析过程 · 先想清楚再作答

  1. 这题最容易答成「因为窗口装不下」。那只答对了一半，而且是不值钱的那一半——窗口一年比一年大，光靠这条理由，面试官会追问「等窗口到一百万 token 呢」，你就没词了。
  2. 先把两个问题拆开：上下文压缩解决的是「同一次会话里这一轮塞不下」，长期记忆解决的是「上个月说过的事想不起来」。前者在组装请求时做减法，后者做加法，触发时机、数据去向、失败后果都不同。能主动区分这两件事，是这题最大的区分度。
  3. 然后给成本账：200 条记忆、每条约 400 token 就是 8 万 token，按输入价 0.15 美元每百万 token 算，每一轮多付 0.012 美元；一天 20 轮就是 0.24 美元一个用户。只检索最相关的 5 条是 2000 token、每轮 0.0003 美元，差 40 倍。而且这笔钱是每轮重复付的，不是一次性的。
  4. 再给比钱更硬的理由：无关信息会降低命中率。200 条里跟这一轮相关的可能只有 1 条，剩下 199 条是噪声，模型会被带偏去回答一个用户没问的问题。**所以哪怕窗口无限大、token 免费，也该检索而不是全塞。** 这一句是这题的最优解。
  5. 落到做法上：把跨会话的用户事实与偏好抽成陈述句存进向量库，每轮按语义检索最相关的三五条注入请求——这就是 RAG 最小的一环。
  6. 可以预期的追问：什么信息该进长期记忆？答三问——跨会话之后还需要吗、会不会随时间失效、能不能靠检索捞回来。「用户住上海」三条都满足，「把刚才那段改成三句话」一条都不满足。

  How to reason about it · think before answering

  1. The tempting answer is 'the window is too small'. That is half right and it is the cheap half — windows keep growing, and the interviewer will ask what you would do at a million tokens.
  2. Separate the two problems first: context compression solves 'this turn does not fit in one session', long-term memory solves 'I cannot recall what was said last month'. One subtracts at request-assembly time, the other adds. Naming that distinction unprompted is where the signal is.
  3. Then quantify: 200 memories at roughly 400 tokens each is 80k tokens; at 0.15 USD per million input tokens that is 0.012 USD every single turn, about 0.24 USD per user per day at 20 turns. Retrieving the top 5 is 2k tokens, 0.0003 USD per turn — a 40x gap, and it repeats every turn.
  4. Give the reason that beats cost: irrelevant context lowers accuracy. If one of 200 memories is relevant, the other 199 are noise that pull the model toward answering something nobody asked. So even with an infinite free window, you would still retrieve rather than dump.
  5. Land on practice: distil cross-session user facts and preferences into standalone statements, store them as vectors, and inject the three to five most relevant per turn — the minimal form of RAG.
  6. Expect the follow-up: what belongs in long-term memory? Three tests — is it still needed across sessions, does it expire, can retrieval find it again. 'Lives in Shanghai' passes all three; 'shorten that paragraph' passes none.

  答题要点

  • 压缩管「这一轮塞不下」，长期记忆管「上个月说过的事想不起来」，是两个问题、两套机制
  • 全塞的成本是每轮重复付的：200 条约 8 万 token，每轮多 0.012 美元；检索 5 条只要 0.0003 美元
  • 更硬的理由是准确率：无关记忆是噪声，会把模型带偏，所以窗口再大也该检索而不是全塞
  • 做法是把跨会话的事实抽成陈述句、向量化存储，每轮按语义检索最相关的三五条注入
  • 判断一条信息该不该进长期记忆：跨会话还需要吗、会不会失效、能不能被检索到

  Key points

  • Compression handles 'this turn does not fit'; long-term memory handles 'what did they say last month' — different problems, different machinery
  • Dumping everything costs on every turn: 200 memories is about 80k tokens and 0.012 USD per turn versus 0.0003 USD for five retrieved ones
  • The stronger reason is accuracy — irrelevant memories are noise, so you would retrieve even with an infinite window
  • Practice: distil cross-session facts into statements, embed them, inject the top three to five per turn
  • Admission test for a memory: still needed across sessions, does not expire, and is findable by retrieval
• chunking 的切分策略会怎么影响检索效果？切多大合适？How does the chunking strategy affect retrieval quality, and how do you pick a chunk size?

  国内高频海外高频进阶#chunking#rag#retrieval-quality

  分析过程 · 先想清楚再作答

  1. 题眼在「怎么影响」。只回答一个数字（比如「切 500 字」）会被追着问为什么，所以要先把两个方向的失效模式讲出来，数字才有落点。
  2. 切太碎的失效模式：单张卡片脱离上下文。「他说要 42 码」检索命中了也没用，代词失去指代，模型拿到一句悬空的话反而更容易编。
  3. 切太整的失效模式更反直觉，也是这题真正的区分点：一块横跨三个主题时，它的向量是这几个主题的平均值，结果对哪个 query 都不太像，命中率反而下降。**块越大信息越全，却越难被检索到**——能说出这句话基本就过了。
  4. 然后给可操作的口径：目标 400 字符、相邻块重叠 80 字符，并优先在句号、换行这类自然边界收尾。重叠的作用要说清楚——一句关键的话被切口劈开时，两块各拿半句，重叠保证它至少在其中一块里是完整的。
  5. 补上代价，这是工程视角：重叠 80 除以 400 等于 20% 的存储放大，向量也跟着多一份；内容高度重叠的两块可能一起被检索出来，白占返回名额，所以要按内容去重。
  6. 可以预期的追问：怎么验证切分策略好不好？答案是准备一批 query 与标注好的期望命中，量召回率和 top-k 命中率，改切分参数后重跑对比——切分是可以被度量的，不该靠感觉调。第二个追问是「对话数据要不要原样切」，答不要：先让模型抽成陈述句再切，否则大量寒暄句会把向量拉平。

  How to reason about it · think before answering

  1. The hinge is 'how does it affect'. Naming a number alone invites a why, so describe both failure modes first and let the number follow.
  2. Too small: a chunk loses its context. 'He wants size 42' retrieves fine but resolves to nothing — pronouns dangle and the model is more likely to fabricate.
  3. Too large is the counter-intuitive half and the real discriminator: a chunk spanning three topics gets a vector that averages them, so it looks only vaguely like any query and recall drops. Bigger chunks carry more information yet are harder to retrieve.
  4. Give an operational default: target 400 characters with 80 characters of overlap, ending on natural boundaries such as sentence stops or newlines. Explain the overlap — when a key sentence lands on a cut, each side holds half of it, and the overlap guarantees at least one chunk holds it whole.
  5. Add the costs: 80 over 400 is 20% storage amplification plus an extra vector per duplicated span, and near-duplicate chunks can both surface and waste result slots, so deduplicate by content before returning.
  6. Expect: how do you validate a chunking strategy? Build a query set with labelled expected hits and measure recall and top-k hit rate, then re-run after changing parameters — chunking is measurable, not a matter of taste. Second follow-up: should raw dialogue be chunked as-is? No — have the model distil it into standalone statements first, or filler turns flatten the vectors.

  答题要点

  • 切太碎：单块脱离上下文，代词失去指代，命中了也用不上
  • 切太整：一块横跨多个主题，向量被平均，对任何 query 都不够像，命中率反而下降
  • 可操作口径：目标 400 字符、重叠 80 字符，优先在句号或换行这类自然边界收尾
  • 重叠的作用是保证被切口劈开的句子至少在一块里完整；代价是约 20% 的存储放大和可能的重复命中
  • 别直接切对话原文，先抽成陈述句；切分效果要用标注好的 query 集测召回率，而不是凭感觉

  Key points

  • Too small: chunks lose context, pronouns dangle, and a hit is useless
  • Too large: one chunk spans several topics, its vector averages them, and recall drops for every query
  • Working default: target 400 characters with 80 characters of overlap, cutting on sentence or newline boundaries
  • Overlap keeps a split sentence whole in at least one chunk, at roughly 20% storage amplification plus possible duplicate hits
  • Distil dialogue into standalone statements before chunking, and validate with a labelled query set measuring recall

D24 RAG 进阶：hybrid search、rerank、引用、recall 评估

• 为什么单纯的向量检索不够，还要加一路关键词检索？Why isn't pure vector search enough — what does keyword search add?

  国内高频海外高频基础#rag#hybrid-search#retrieval

  分析过程 · 先想清楚再作答

  1. 这题的区分度不在「你知不知道有 hybrid search」，而在**你能不能说出一个向量检索一定会漏的具体例子**。答不出例子的，一听就是只看过架构图。
  2. 推导链只有一句：向量检索比的是语义距离，所以它的强项和弱项都来自「压缩成语义」这一步——同义词能对上（运费 / 邮费），而没有语义的字符串会被压到一起（E4032、SF-3000、订单号、人名）。
  3. 关键词那一路（BM25）的性质正好相反：一个词在本文档里越频繁越相关、在全语料里越常见越不值钱，所以它对低频稀有词极准，对同义改写完全无能。
  4. 结论要说成「两者的盲区不重叠，而且是由计算原理决定的不重叠」——不是「多一路更保险」这种模糊说法。举一个实测例子最有说服力：查「E4032 是什么意思」，向量 top5 里没有那篇讲支付错误码的文档，关键词 top1 就是它。
  5. 可预期的追问一：那怎么合并两路结果？答 RRF，并说清为什么不能加权求和（见 q02）。
  6. 可预期的追问二：中文怎么做关键词检索？答 Postgres 默认分词器对中文等于不分词，最简可用的兜底是 bigram（相邻两字切开），但英文与编号必须整词保留；生产要上专门的中文分词扩展。这一条能答出来，基本就说明你真动手做过。

  How to reason about it · think before answering

  1. The discriminator is not whether you know the term 'hybrid search' — it is whether you can name a concrete query that vector search will always miss. No example means you have only read architecture diagrams.
  2. One causal chain: vector search compares semantic distance, so both its strength and its weakness come from that compression step. Synonyms match (shipping fee vs postage), but strings with no semantics collapse together — error codes, SKUs, order ids, person names.
  3. BM25 has the mirror-image profile: a term matters more when it is frequent in this document and rare across the corpus. So it nails low-frequency literals and fails completely on paraphrase.
  4. State the conclusion as 'their blind spots do not overlap, and that follows from how each one computes' — not the vague 'two channels are safer'. A measured example lands best: for 'what does E4032 mean', the correct doc is absent from the vector top-5 and is the keyword top-1.
  5. Expected follow-up 1: how do you merge the two rankings? Answer RRF, and explain why weighted sums fail (see q02).
  6. Expected follow-up 2: how do you do keyword search over Chinese? Postgres's default parser effectively does not tokenize Chinese; the cheapest workable fallback is character bigrams, keeping ASCII words and codes whole. Production needs a real Chinese tokenizer extension. Answering this usually proves you actually built it.

  答题要点

  • 向量检索比的是语义距离，强在同义改写，弱在型号、错误码、订单号这类没有语义的字符串。
  • BM25 强在低频稀有词的字面命中，弱在同义改写——两者的盲区由各自的计算原理决定，不重叠。
  • 所以第一轮开两路、各取 20 条，用 RRF 融合，把两边的盲区互相补上。
  • 举实测例子：E4032 那条 query 向量 top5 漏掉正确文档，关键词 top1 就是它；「邮费」那条反过来只有向量能召回。
  • 中文关键词那一路要处理分词，最简兜底是 bigram，字母数字整词保留，生产上专门的中文分词扩展。

  Key points

  • Vector search compares semantic distance: strong on paraphrase, weak on SKUs, error codes and order ids that carry no semantics.
  • BM25 is strong on rare literal terms and weak on paraphrase — the blind spots follow from the algorithms and do not overlap.
  • So run both channels wide (top 20 each) and fuse with RRF so each covers the other's gap.
  • Give a measured example: for the E4032 query the correct chunk is missing from vector top-5 but is keyword top-1; a 'postage vs shipping fee' query is the reverse.
  • Chinese keyword search needs tokenization: character bigrams as the cheap fallback, ASCII words kept whole, a real tokenizer extension in production.
• 两路检索结果怎么合并？为什么不能直接加权求和？How do you merge two retrieval rankings, and why not just take a weighted sum of the scores?

  国内高频海外高频进阶#rag#rrf#ranking

  分析过程 · 先想清楚再作答

  1. 题眼在后半句。前半句答「RRF」谁都会，后半句「为什么不能加权求和」才是筛人的地方——它考的是你有没有真的看过两路分数的分布。
  2. 怎么拆：先问自己两个分数是不是同一个量纲。余弦相似度有界（0 到 1）且分布密集，同一批候选常常只差 0.02；BM25 无上界，命中几个稀有词就能到 12 分。**不同量纲的数相加，等于让量纲大的那一路单方面决定结果**，权重只是在调「它说了算的程度」。
  3. 更麻烦的是它不稳定：权重在这批语料上调好了，换一批语料分布就变了，得重调。这是一个永远还不完的技术债。
  4. 结论：改用名次。RRF 把每一路的名次折算成 `1/(k + rank)` 再相加，k 取 60。名次是无量纲的，不需要任何标定。k 的作用是压平头部差距，让「两路都进前列」压过「一路排第一」——共识优先于单点自信。
  5. 一个能当场手算的例子很加分：两路排名 [a,b,c] 与 [c,d,a]，a 得 1/61 + 1/63 ≈ 0.0323；而分数直接相加的版本会把 BM25 里 12 分的 c 顶到第一。
  6. 可预期的追问：同分了怎么办？必须显式定序（比如按 id），否则结果取决于哈希表遍历顺序，同一份输入在不同语言、不同运行里给出不同排序——评估集量出来的数字也就不可复现了。这一条答出来会非常加分，因为它说明你真的跑过多次。

  How to reason about it · think before answering

  1. The second half is the real question. Anyone can say 'RRF'; explaining why weighted sums fail is what separates people who have looked at the score distributions.
  2. Decompose it: are the two scores even the same unit? Cosine similarity is bounded in 0 to 1 and tightly clustered — candidates often differ by 0.02. BM25 is unbounded and a few rare-term hits reach 12. Adding them lets the larger-magnitude channel decide everything; the weight only tunes how much it dominates.
  3. Worse, it is unstable. Weights tuned on one corpus drift on the next, so you re-tune forever.
  4. Conclusion: fuse ranks, not scores. RRF maps each rank to 1/(k + rank) and sums, with k = 60. Ranks are unitless and need no calibration. k flattens the head of the list so that 'top-ranked in both channels' beats 'first in one channel' — consensus over single-source confidence.
  5. A hand-checkable example helps: rankings [a,b,c] and [c,d,a] give a = 1/61 + 1/63 ≈ 0.0323, while a raw score sum promotes c on the strength of its BM25 12.
  6. Expected follow-up: what about ties? You must break them explicitly, e.g. by id. Otherwise ordering depends on hash-map iteration order and differs across languages and runs, which makes your evaluation numbers irreproducible. Mentioning this signals you actually ran it more than once.

  答题要点

  • 用 RRF：每一路的名次折算成 1/(k + rank) 再相加，k 取 60。
  • 不能加权求和是因为两个分数量纲不同——余弦有界密集、BM25 无上界，相加等于让 BM25 单方面决定结果。
  • 而且权重不可迁移：这批语料调好，换一批就得重调，是还不完的债。
  • 名次是无量纲的，不需要标定；k 压平头部差距，让两路共识压过单路自信。
  • 同分必须显式定序（按 id），否则结果依赖哈希表遍历顺序，评估数字不可复现。

  Key points

  • Use RRF: map each channel's rank to 1/(k + rank) and sum, with k = 60.
  • Weighted sums fail because the scores are different units — bounded, tightly clustered cosine versus unbounded BM25, so BM25 decides the outcome.
  • Weights also do not transfer: tuned on one corpus, they drift on the next.
  • Ranks are unitless and need no calibration; k flattens the head so cross-channel consensus outweighs single-channel confidence.
  • Break ties explicitly (by id) or ordering depends on hash iteration order and your evaluation numbers stop being reproducible.
• 重排（rerank）一般怎么实现？它解决了初步检索的什么问题，代价是什么？How is reranking usually implemented, what problem does it solve, and what does it cost?

  国内高频海外高频进阶#rag#rerank#latency

  分析过程 · 先想清楚再作答

  1. 这题最容易答成「再排一次序，更准」。面试官想听的是**为什么第一轮不能直接排准**，以及**为什么重排不能对全库做**。
  2. 怎么拆：第一轮的排序依据是「检索信号」——余弦距离或词频统计，它们是为了能在百万条里快速筛选而设计的，代价就是粗。重排换了一种算法：把 query 和候选**拼在一起**送进同一个模型算相关度（cross-encoder），精度高得多，但复杂度是每条候选一次前向，没法对全库做。所以它必须跟在一个宽召回后面。
  3. 结论要区分两种实现：教学 / 原型可以用 LLM 批量打分（一次调用给 40 条打 0 到 10 分），生产用专门训练的 cross-encoder 重排模型。**代价说清楚：多一次 100 到 300 毫秒的调用，外加一台推理机器**——它不是按 token 计费的 API，是要占资源的。
  4. 把它放进漏斗里说最清楚：召回决定天花板，重排决定天花板上的东西能不能排到前五。实测的样子是——加了关键词那一路，recall@20 从 83% 涨到 95%（天花板抬高）；再加重排，recall@20 只到 98%，但 recall@5 从 80% 跳到 91%、MRR 从 0.732 到 0.908。
  5. 可预期的追问一：重排能不能提高召回？不能。它不引入新候选，只重排已有的那批——所以看 recall@20 判断重排效果是错的指标。
  6. 可预期的追问二：为什么不用 LLM 打分上生产？延迟不可控、成本按 token 走、分数会随提示词措辞漂移，而且没法做批量离线蒸馏。

  How to reason about it · think before answering

  1. The lazy answer is 'sort again, more accurately'. What the interviewer wants is why the first pass cannot rank well, and why reranking cannot run over the whole corpus.
  2. Decompose: the first pass ranks by retrieval signals — cosine distance or term statistics — which are designed to scan millions of items fast, and coarseness is the price. Reranking changes the algorithm: query and candidate go into one model together (a cross-encoder), which is far more accurate but costs one forward pass per candidate. Hence it must sit behind a wide recall stage.
  3. Distinguish two implementations. For teaching or prototypes, batch-score with an LLM (0-10 for 40 candidates in one call). Production uses a trained cross-encoder reranker. Name the cost: an extra 100-300 ms hop plus an inference box — it is not a per-token API, it consumes capacity.
  4. Framing it as a funnel is clearest: recall sets the ceiling, reranking decides whether what is under the ceiling reaches the top five. Measured: adding the keyword channel lifts recall@20 from 83% to 95%; adding reranking moves recall@20 only to 98%, but recall@5 jumps from 80% to 91% and MRR from 0.732 to 0.908.
  5. Expected follow-up 1: does reranking improve recall? No. It introduces no new candidates, so recall@20 is the wrong metric to judge it by.
  6. Expected follow-up 2: why not ship LLM scoring to production? Unpredictable latency, per-token cost, scores that drift with prompt wording, and no clean path to offline distillation.

  答题要点

  • 第一轮按检索信号粗排（余弦、词频），为的是能在大库里快速筛，代价是粗。
  • 重排把 query 和候选拼在一起过同一个模型（cross-encoder），精度高但每条一次前向，只能对几十条做。
  • 教学版可用 LLM 批量打 0 到 10 分；生产用专用重排模型，代价是多一次 100 到 300 毫秒的调用加一台推理机器。
  • 重排不提高召回，它提高的是 recall@5 与 MRR——实测 80% → 91%、0.732 → 0.908，而 recall@20 只从 95% 到 98%。
  • 所以判断重排效果要看前 k 小的指标，不要看 recall@20。

  Key points

  • The first pass ranks by retrieval signals so it can scan a large index fast; coarseness is the trade.
  • Reranking feeds query and candidate through one model together (cross-encoder): much sharper, but one forward pass per candidate, so only tens of items.
  • Batch LLM scoring works for teaching; production uses a dedicated reranker, costing an extra 100-300 ms hop plus an inference box.
  • Reranking does not raise recall — it raises recall@5 and MRR (measured 80% to 91%, 0.732 to 0.908) while recall@20 barely moves from 95% to 98%.
  • So judge a reranker by small-k metrics, never by recall@20.
• 怎么评估一个 RAG 系统的检索效果？评估集应该怎么构造？How do you evaluate retrieval quality in a RAG system, and how should the evaluation set be built?

  国内高频海外高频深入#rag#evaluation#recall

  分析过程 · 先想清楚再作答

  1. 这题是国内面试的极高频题，也是最容易暴露「只搭过没调过」的一题。判据很简单：你的回答里有没有出现**具体的指标名和标注粒度**，没有就是没做过。
  2. 先把评估对象分清楚——这是最容易混的一步：**检索评估问「找得到找不到」，生成评估问「答得对不对」**。两套评估集要分开维护。混成一套的后果是分数掉了你分不清是检索漏了还是模型答砸了，而这两件事的修法完全不同。
  3. 评估集的形状：20 条左右的 query，每条**人工标注 1 到 3 个必须召回的 chunkId**。注意标注粒度是**块**不是文档——检索的单位就是块，标到文档级会让指标虚高。query 要覆盖真实分布，尤其要包含那些你知道会翻车的类型（编号、同义改写、跨文档）。
  4. 三个指标各回答一个问题：recall@5 是「进上下文的那几条覆盖了多少」，也就是你真正关心的数；recall@20 是天花板，它上不去说明问题在召回侧、重排再强也没用；MRR 对排序质量敏感，recall 打平时用它分高下。
  5. 生产视角：评估集一旦定下来就要冻结，换了样本分数就没有可比性——这和产线质检必须用固定的标准样品是同一个道理。同时线上要有对照指标（引用为空率、幻觉引用率、转人工率），因为离线过了不等于线上没事。
  6. 可预期的追问：标注成本这么高，20 条够吗？答：20 条不够做统计显著性，但足够做**回归**——它的作用是「改了检索之后别悄悄变差」。要做 A/B 定论再上规模，而且优先扩充失败案例，不是随机加样本。

  How to reason about it · think before answering

  1. This is a very common question in the Chinese market and the fastest way to expose someone who has assembled RAG but never tuned it. The test: does your answer contain concrete metric names and an annotation granularity?
  2. First separate what is being evaluated — the step people most often conflate. Retrieval evaluation asks 'was it found'; generation evaluation asks 'was the answer right'. Keep two separate sets. Merge them and, when the score drops, you cannot tell whether retrieval missed or the model fumbled — and those have completely different fixes.
  3. Shape of the set: about 20 queries, each annotated with 1-3 chunk ids that must be retrieved. Annotate at chunk level, not document level — chunks are the retrieval unit, and document-level labels inflate the numbers. Cover the real query mix, especially the types you know break: codes, paraphrase, cross-document.
  4. Three metrics, three questions. recall@5 is what actually reaches the model, so it is the number you care about. recall@20 is the ceiling — if it does not move, the problem is on the recall side and no reranker will save you. MRR is sensitive to ordering and breaks ties when recall is equal.
  5. Production view: freeze the set once agreed, because changing samples destroys comparability — the same reason a factory keeps fixed reference samples. Pair it with online counterparts (empty-citation rate, hallucinated-citation rate, escalation rate), since passing offline does not mean passing in production.
  6. Expected follow-up: is 20 enough given the labelling cost? Not for statistical significance, but enough for regression — its job is to stop retrieval silently getting worse. Scale up before you settle an A/B, and grow it from failure cases rather than random additions.

  答题要点

  • 检索评估和生成评估是两套：前者问「找得到找不到」，后者问「答得对不对」，分开维护。
  • 评估集是 20 条左右的 query，每条人工标 1 到 3 个必须召回的 chunkId——标到块级，不是文档级。
  • recall@5 是真正关心的数（模型只看得到这几条），recall@20 是天花板，MRR 衡量排序质量。
  • 评估集一旦定下来就冻结，否则分数没有可比性；线上再配引用为空率、幻觉引用率做对照。
  • 20 条不够做显著性但够做回归；扩充时优先补失败案例，不是随机加样本。

  Key points

  • Retrieval and generation evaluation are two separate sets: 'was it found' versus 'was the answer right'.
  • Around 20 queries, each labelled with 1-3 chunk ids that must be retrieved — chunk level, not document level.
  • recall@5 is what the model actually sees, recall@20 is the ceiling, MRR measures ordering quality.
  • Freeze the set once agreed or scores stop being comparable; pair it with online empty-citation and hallucinated-citation rates.
  • Twenty cases is a regression guard, not a significance test; grow it from failure cases, not random samples.

14 天 RAG：从检索到可信回答

D8 评估先行：搭 golden set、算召回与排序指标、用模型当裁判判忠实度

• 让你从零给一个公司知识库的 RAG 系统建评估集，你会怎么做？多少题才算够用？You need to build an evaluation set from scratch for a RAG system over a company knowledge base. How would you do it, and how many questions are enough?

  国内高频海外高频进阶#evaluation#golden-set#rag

  分析过程 · 先想清楚再作答

  1. 这题的区分度在「出题方向」和「规模的理由」两处。开口就说「找几百个用户真实问题」的，多半没真做过——真实问题的答案在哪篇文档里，没人标得出来。
  2. 先给方向：从语料反向出题，打开每一篇读它能回答什么，出题的那一刻答案文档就已经确定了，标注成本几乎为零。反方向（先想问题再找答案）会得到一堆自己都不知道答案的题。
  3. 再给结构：每题记问题、答案文档列表、类型三个字段；类型至少分单文档、多跳、无答案三类，并说明多跳必须全部答案文档命中才算命中，无答案不参与召回率而是考拒答。
  4. 规模的理由要给出来，不能只报一个数字：20 题能把「完全不能用」和「基本能用」分开，够做冒烟；100 到 200 题才有资格判断「涨了两个点」是真的还是噪声。上线之后每次线上出问题就把那个问题补进集合——评估集是长出来的。
  5. 补一句成本与保鲜：出题是人力活，20 题两小时是正常量级；语料更新后要复核答案文档还在不在，否则集合会悄悄腐烂，指标下跌你会误以为是系统坏了。
  6. 可预期的追问是「怎么防止评估集被过拟合」。答案是留一份不参与调优的保留集，并且定期从线上真实问题里补充新题，只用来验收不用来调参。

  How to reason about it · think before answering

  1. The discriminator here is the direction you generate questions in, and whether you can justify a size rather than name one.
  2. Go corpus-first: read each document and write the questions it can answer. The answer document is fixed at authoring time, so labeling is nearly free. Question-first gives you items whose answers nobody can locate.
  3. Give the schema: question, answer document ids, and a type. At minimum three types - single-document, multi-hop, and unanswerable. Multi-hop counts as a hit only when every answer document makes it into the context; unanswerable items are scored on abstention, not recall.
  4. Justify the size: 20 items separate 'broken' from 'usable' and are enough for a smoke gate; 100 to 200 are needed before a two-point delta means anything. Then grow the set - every production failure becomes a new item.
  5. Mention cost and decay: roughly two hours for 20 items, and answer labels must be rechecked whenever the corpus changes, or the set rots and you misread the drop as a system regression.
  6. Expected follow-up: how do you avoid overfitting to the eval set? Keep a held-out slice that never informs tuning, and refresh it from real production questions.

  答题要点

  • 从语料反向出题，出题时答案文档就已确定，标注成本最低。
  • 每题标类型：单文档、多跳、无答案，三类缺一不可。
  • 多跳要求全部答案文档命中；无答案不算召回率，考的是拒答。
  • 20 题够冒烟，100 到 200 题才能判断小幅变化；线上故障持续补题。
  • 留一份不参与调优的保留集，防止对评估集过拟合。

  Key points

  • Author corpus-first so the answer document is known at authoring time.
  • Label every item with a type: single-document, multi-hop, unanswerable.
  • Multi-hop requires all answer documents; unanswerable items score abstention, not recall.
  • 20 items for a smoke gate, 100 to 200 to trust small deltas, and keep growing it from production failures.
  • Hold out a slice that never informs tuning to avoid overfitting the set.