You need to build an evaluation set from scratch for a RAG system over a company knowledge base. How would you do it, and how many questions are enough?

Author corpus-first so the answer document is known at authoring time. Label every item with a type: single-document, multi-hop, unanswerable. Multi-hop requires all answer documents; unanswerable items score abstention, not recall. 20 items for a smoke gate, 100 to 200 to trust small deltas, and keep growing it from production failures. Hold out a slice that never informs tuning to avoid overfitting the set.

Recall, mean reciprocal rank, and normalized discounted cumulative gain - which failure mode does each one catch first, and what do you miss by watching only one?

Recall answers 'did it make it into the context', sensitive to total misses, blind to rank shifts. MRR answers 'how high is the first hit', sensitive to ranking degradation, blind to how many hits there are. nDCG answers 'how good is the top k overall', the direct target for reranking, blind to existence. Only the combination localizes the failure to retrieval, ranking, or noise. State the hit criterion: context is packed against a token budget, not a fixed top-k.

What systematic biases does an LLM judge have when scoring RAG faithfulness, and how do you detect them and prove your judge is trustworthy?

Three biases: position, verbosity, and self-preference, each with a matching mitigation. Score pointwise rather than pairwise; decompose into claims and score a ratio to kill the length premium; never let the generator judge itself. Pin rubric anchors in the prompt and force structured output that quotes unsupported sentences. Establish trust through stratified human spot-checks and an agreement rate; below 0.8 the judge cannot gate merges. Add probes with known verdicts so a broken judge is caught on every run.

Why must a RAG evaluation set include questions the corpus cannot answer, and what does leaving them out hide?

An all-answerable eval set makes 'answers confidently when it should not' completely invisible. Unanswerable items are scored on abstention, not recall, and you check both the retrieval gate and the generation refusal. Author them with strong distractor terms, or retrieval returns nothing and you are testing the tokenizer. Keep them at roughly 15% or more of the set, alongside multi-hop items, as the coverage floor. A low abstention rate splits into two causes: a no-op retrieval score gate, or a missing refusal instruction in the prompt.

Why do hybrid retrieval systems usually use reciprocal rank fusion instead of normalizing both scores and adding them with weights? When does the weighted approach break down?

BM25 is unbounded, cosine is bounded; the two scales are not comparable, so adding them is meaningless. Per-route max normalization has a denominator that floats with the query, so the least relevant hit of an unanswerable query still normalizes to 1.0. Weights must be tuned against an eval set, the search is high-dimensional once you add routes, and swapping models invalidates it; RRF has a single constant k. RRF consumes only the ordered id list from each route, because rank is the one thing the routes share. Larger k rewards `ranked well by both routes`; ties must fall back to document id so results are reproducible.

Why is a cross-encoder more accurate than a bi-encoder? And if it is more accurate, why not just use it to search the whole corpus directly?

A bi-encoder encodes both sides separately and joins them with one dot product; a cross-encoder concatenates them so attention can align across the pair. The accuracy gap comes from compression: a bi-encoder squeezes a whole document into one vector and loses bindings; a cross-encoder does not compress. Bi-encoder document vectors can be computed offline and indexed; a cross-encoder has nothing to precompute. Reranking the full corpus means running every chunk through a model on every question, so cost scales linearly with corpus size. The standard split is recall plus rerank, with reranking applied only to the top few dozen after fusion.

You replaced pure vector retrieval with hybrid search plus reranking, and after shipping it your eval metrics went down. How do you investigate?

Run all four configurations separately for attribution, on one eval set with one context budget, before touching any parameter. Separate a recall drop from a ranking drop: the first is a candidate-pool or budget issue, the second is a fusion or rerank issue. Check recall depth: taking too many per route lets noise vote, and narrowing it can bring recall back. Confirm the eval protocol did not change; touching budget, hit rule, or candidate depth makes old and new numbers incomparable. Freeze the four-way comparison into one command plus a baseline report, and block merges on regression.

Adding a reranker costs you 200 ms of extra latency per question plus a per-search fee. How do you decide whether that spend is worth it?

Report all three ledgers together: metric gain, latency growth, cost growth; a proposal missing one is incomplete. Confirm whether the bottleneck is ordering or recall first; reranking only reorders and cannot rescue an answer that was never retrieved. Judge the latency by where it lands: it is small relative to a streaming generation, but fatal in an as-you-type search box. Rerankers bill per search rather than per token, so cost scales with question volume, not candidate count. If it is unaffordable: route only hard queries to the reranker, cache results, or self-host an open-weights cross-encoder.

Why does HyDE (hypothetical document embeddings) work, and when does it steer retrieval in the wrong direction?

It works by register alignment: a question and a policy paragraph sit in different neighbourhoods, and the fake passage moves the query into the document's. The draft's factual accuracy is irrelevant — it only supplies a direction and is never shown to the user. It misfires when the model invents over-specific details, or when the corpus has no answer and the fabrication finds plausible neighbours anyway. Two guardrails: gate on raw per-route scores, not fused ones; fuse the hypothetical document with the original query instead of replacing it. It is the most expensive query-side technique (long output plus doubled retrievals), so keep it off by default and A/B it.

How do you handle coreference in multi-turn RAG, and what is the classic failure when you skip it?

Add a short rewrite call before retrieval: last few turns plus current question in, one retrieval line out, temperature 0, answering explicitly forbidden. Do not splice the whole history into the query — it grows unbounded, dilutes IDF, and leaks the previous answer into the search. Classic failure: an unresolved pronoun means no candidate clears the gate, so the system refuses a question the corpus can answer. That false "not found" hurts more than a wrong answer, since the user just asked about the same thing. Order matters: rewrite first, then route; multi-query and step-back both build on the rewritten query.

Query rewriting adds a model call per question and doubles end-to-end latency. How do you decide whether it is worth paying?

Locate the gain first: rewriting is near-zero on single-turn traffic and pays off on follow-ups, so start from the share of multi-turn sessions. All three ledgers are mandatory: metric delta on a golden set, added latency, added calls and token cost. Try the cheap variants before deciding: conditional triggering, per-session caching, and a small model for the rewrite. Decide on gain times affected traffic share over added latency and cost, then rank it against your other optimisations. If latency is a hard constraint, fire the rewrite in parallel with the first retrieval and fuse both rankings, turning a sum into a max.

What happens when intent routing misclassifies, and how would you design the fallback?

The three routes have asymmetric error costs: sending a retrieval-worthy question to a direct answer is the worst, while routing chit-chat to single-hop only wastes one retrieval. Bias the fallback toward spending more: default to single-hop whenever the classifier is unsure, since that error is the mildest and is recoverable. Add runtime fallbacks: re-retrieve if a direct answer contains figures that need a source; escalate or abstain if no single-hop candidate clears the gate. Log every routing decision — raw question, label, whether a fallback fired — for both monitoring and the next training set. When chit-chat and multi-hop are both rare, the classification call costs more than it saves; route everything to single-hop instead.

Parent-child indexing and contextual retrieval both patch the same problem — chunks losing their context. What actually distinguishes them?

Parent-child acts on the generation side: retrieve small, swap in the parent for context. It fixes 'found but unreadable'. Contextual retrieval acts on the retrieval side: the header makes the chunk findable. It fixes 'readable but never found'. They are orthogonal and compose; keep the header in the index only, never in the context window. Parent-child costs more index entries and a larger context unit; contextual retrieval costs one call per chunk plus a permanently larger index. Pick based on the observed failure: incomplete evidence points to the former, zero retrieval to the latter.

Contextual retrieval needs one model call per chunk. How do you estimate that one-off cost, and what levers bring it down?

Split into one-off (per-chunk input/output plus re-embedding) and per-query (header read by both reranker and generator). The one-off is dominated by re-reading each document n times; caching turns that into one write plus n-1 reads. Caching is prefix-matched: the full document must come first, the chunk after, or you get zero hits. Measured on 30 docs / 134 chunks, caching cut the one-off cost by about 29%, and finer chunks save more. The lasting cost is per query: keep headers out of the context window, keep them short, and generate them selectively.

What kind of question actually requires graph retrieval? Give one concrete case where it is justified and one where it is not.

The test is not 'is it multi-hop' but 'does the second document overlap the query at all' — only zero overlap earns a graph. Justified: the approver question, where an intermediate entity is the only bridge and the second doc never enters the candidate pool. Not justified: a multi-hop question whose documents both overlap the query — hybrid retrieval collects them in one pass. Real graph cost is entity disambiguation, relation dedup, incremental subgraph recomputation and a whole extra store — not a single extraction call. Try two-pass agentic retrieval first; build the graph only when that fails.

You have built three different indexes over the same corpus. How do you decide which one a query goes to?

First challenge the premise: all five index structures landed at the same 93.8% recall in our measurement, so an index without a matching weakness is pure cost. At request time the usable signal is question shape — detail, summary, or entity-chaining — mapping to chunk, tree-summary and graph indexes. Reuse the previous day's intent router for classification and record the routing decision as request metadata. Fall back to the default index on misclassification instead of fanning out and fusing, which multiplies latency and cost. Replay the golden set periodically to check whether the classifier picks the best-scoring index.

You are exposing retrieval to a model as a tool. How do you write the tool description, and what concrete failure modes appear when you write it badly?

The description is a prompt for the model, not a code comment: scope, when to call, when not to call, query shape. Missing scope turns it into a web search; missing 'must call' produces confident answers from memory. Missing 'do not call' makes small talk trigger retrieval, paying cost and latency for nothing. Stating 'keyword phrase, no question words' fixes query pollution at the source. Optional filters need 'only set when certain' - a wrong filter silently hides the right answer. Regression-test with a negative suite and assert the tool was not invoked.

Self-reflective retrieval rewrites the query and retries. How do you guarantee it terminates instead of spinning on the same query forever?

Three brakes, none optional: max rounds, cumulative token budget, duplicate-query detection. The cumulative budget catches rounds that each pass but blow up together - the round cap cannot see that. Check for duplicates before retrieving, and normalize the query to a term set before comparing. Record stop reasons as distinct categories rather than one 'finished' bucket. Every brake needs a case that actually trips it; an untested brake is no brake. Have the assessor emit covered and missing elements so 'not enough' is auditable.

In multi-hop retrieval a wrong first hop poisons every hop after it. How would you design for that?

Classify first: never retrieved needs a new query term (a hop); retrieved-then-filtered needs a relaxed gate. The fixes are opposite. Derive the next query from a bridge phrase - a concrete noun absent from the question that also appears in another document. Keep the previous hop's material so a failed hop does not discard existing evidence. Trace every hop separately so you can pinpoint which one drifted. Allocate context budget round-robin across hops, or a successful hop still fails to change the metric. Surrender explicitly when no lead remains and hand it to the generation-side refusal.

When would you refuse to make a RAG system agentic, and what data would you use to convince your team?

Gains concentrate in multi-hop and retry cases; single-document questions gain almost nothing. Settle it with the evaluation set: multi-hop delta against the multiplier on total calls. One measured set: multi-hop recall 75% to 100%, overall 93.8% to 100%, retrieval calls 1 to 1.75 plus the same number of assessment calls. Refuse when latency-sensitive, when question patterns are fixed, or when cost is tight - variance hurts more than the mean. Route instead: a cheap check up front, and only multi-hop-looking questions enter the loop. Looping is a capability, not a default.

After a document changes, how do you recompute only the affected chunks? And how do you guarantee a deleted document really disappears from the index?

Three-way reconciliation covering adds, modifies and deletes; a modify replaces the whole document, old chunks first. Normalize line endings and trim before hashing, or cross-platform re-uploads look like edits and you are doing a full rebuild every night. Deletion is an absence, not an event: walk the index for ids the source no longer has instead of waiting on a change feed. Cascade deletes from documents to chunks to embeddings so one statement suffices; assert chunk count equals embedding count to catch orphans. On an incomplete pull, apply adds and modifies only, and soft-delete with a retention window so mistakes are reversible.

Why can't access control be applied at the generation step? What exactly leaks if you put it there?

Filtering at generation time means unauthorized documents were already retrieved, ranked, held in memory and written to logs and traces; the exposure is far wider than 'did the text reach the answer'. The second consequence is dilution: filtered-out slots are not backfilled, so users see 'nothing found' while the log shows a normal retrieval. The fix is to put the permission predicate in the same statement as ordering and LIMIT so the database prunes before ranking. Choose between row-level filtering and index isolation by the count and stability of the boundaries; a shared index lets a large tenant crowd out a small one's candidate slots. Caching is the same bug's second crime scene: the answer cache key must carry the permission scope or answers leak across users without a trace.

What can be cached in a RAG system, and what are the invalidation conditions for each?

Three layers — answer, retrieval, embedding — with lifetimes orders of magnitude apart; treating them as one thing is the mistake. The rule is that 'when must it expire' equals 'is it in the key'; anything left out of the key can never invalidate the entry. The answer key carries question, permission scope, index version, model and prompt version; the retrieval key drops the model and adds topK and the embedding backend. The embedding layer is content-addressed and keyed only on text plus backend; adding an index version turns every sync back into a full rebuild. Version-based invalidation beats targeted deletion because you cannot enumerate which questions a given change affected.

You need to switch embedding models. How do you migrate a live system without downtime and without losing recall?

Vectors from different models are not comparable, so a model switch is equivalent to re-embedding the entire corpus. Four steps: add a nullable second vector column, backfill in the background, canary with the golden set scored on both columns, then cut over once the numbers hold. The whole value lies in rollback cost: cutover is a config change, so reverting takes a second instead of another full rebuild. Build the ANN index on the new column after the backfill; make the backfill resumable and rate-limited so it does not exhaust the embedding quota and stall live queries. Validate with an A/B on one golden set reporting recall, faithfulness, latency and cost; a model switch is also the only time the embedding cache truly must be invalidated.

You are handed a knowledge base of five million documents that must answer in about a second, with accuracy as the top priority. How would you design it?

Translate constraints into numbers first: twenty million chunks means the index will not fit one machine, and a one-second budget leaves retrieval two to three hundred milliseconds. Dedicated store or partitions plus half precision; keep keyword and vector routes with RRF, and rerank only the top twenty after fusion. Name the two things you will not ship: agentic only for a routed multi-hop minority, and chunk headers only after measuring on your own embeddings. Turn accuracy-first into a hundred-plus question golden set, abstention rate as its own column, and code-verified citations. First index build is a one-off large expense: batch it, make it resumable, add incremental sync on day one, and dual-write columns for model swaps.

Looking back at the RAG project you built, which decision would you change now, and why?

Structure the answer in four beats: the choice, the evidence then, the evidence later, the call now. Pick a decision that was defensible at the time and later overturned by data, not one you knew was a shortcut. Worked example: 'headers hurt keyword retrieval' was corrected to 'headers hurt when chunk boundaries misalign with structure', because the chunker premise changed. Record the premises bound to every number; when a premise changes you owe a re-run rather than a reinterpretation. Classify before asserting: measured by this experiment, or forced by the implementation's structure — the former needs its boundaries stated.

How do you convince a non-technical stakeholder that your retrieval system actually got better?

Externally report three numbers they can adjudicate: correct, wrong, declined — summing to one hundred. Keep wrong and declined separate; a single accuracy number is gamed by a system that learns to decline everything. Pair it with ten before-and-after real questions, every claim carrying a citation they can open and verify. Volunteer the saturation caveat: a column of perfect scores means a broken ruler, and the fix is harder questions. Let stakeholders contribute questions; append every production miss to the golden set so it grows over time.

Users report that your live RAG system 'answers inaccurately'. What is your triage order?

Classify the complaint into four failures first — off-topic, partial, misaligned citation, stale — because their fixes do not transfer. Read right to left, fix left to right: dump the candidate pool and final context and find which layer the answer document stalls at. The four rules in order: never retrieved, retrieved but below the gate, admitted but squeezed out of the budget, packed but unused by the model. For multi-hop, diagnose only the missing documents; judging by 'any one retrieved' mislabels a never-retrieved case as a budget problem. For stale content, check reconciliation and the cache key: what must expire is exactly what the key must contain.

If you could only fund three changes to improve an existing RAG system, which three would you pick and why those three?

First, build evaluation: without a scale the other two changes are unverifiable, and the retrieval metrics are cheap enough to gate every commit. The golden set must include unanswerable questions, or a system that only ever guesses scores perfectly on your report. Second, move abstention from the prompt into code: citation numbers are a closed set, and a substantive-overlap check catches real-number-fake-content forgeries. Third is chosen by the failure cases, not by a list of techniques — failure cases first, index structure or retrieval trick second. Toggle each change individually and report three ledgers: metric, latency, cost. A proposal reporting only the first should not be approved.

Interview Bank

328 questions total; 29 shown with current filters.

CourseAll From Frontend Engineer to Agent Engineer in 30 Days Prompt Engineering From Scratch in 5 Days Mastering Claude: From Conversation to Claude Code in 5 Days Mastering Codex and the OpenAI Agents SDK in 5 Days MCP in 7 Days: Wire Tools Into Any Agent Agent Skills in 7 Days: Turn Experience Into Reusable Capability Context Engineering in 5 Days RAG in 14 Days: From Retrieval to Trustworthy Answers Build an AI Short-Drama Production Pipeline With Agents in 14 Days

WeekAll W1 W2

MarketAll Common in China Common overseas

DifficultyAll Basic Intermediate Deep dive

Tag

All #evaluation8 #agentic-rag4 #abstention3 #cost3 #chunking2 #contextual-retrieval2 #cost-tradeoff2 #golden-set2 #hybrid-search2 #multi-hop2 #observability2 #query-rewriting2

49 more tags

RAG in 14 Days: From Retrieval to Trustworthy Answers

D8 Evaluation First: Building a Golden Set, Computing Recall and Ranking Metrics, Using a Model as Judge for Faithfulness

You need to build an evaluation set from scratch for a RAG system over a company knowledge base. How would you do it, and how many questions are enough?让你从零给一个公司知识库的 RAG 系统建评估集，你会怎么做？多少题才算够用？
Common in ChinaCommon overseasIntermediate#evaluation#golden-set#rag
How to reason about it · think before answering
The discriminator here is the direction you generate questions in, and whether you can justify a size rather than name one.
Go corpus-first: read each document and write the questions it can answer. The answer document is fixed at authoring time, so labeling is nearly free. Question-first gives you items whose answers nobody can locate.
Give the schema: question, answer document ids, and a type. At minimum three types - single-document, multi-hop, and unanswerable. Multi-hop counts as a hit only when every answer document makes it into the context; unanswerable items are scored on abstention, not recall.
Justify the size: 20 items separate 'broken' from 'usable' and are enough for a smoke gate; 100 to 200 are needed before a two-point delta means anything. Then grow the set - every production failure becomes a new item.
Mention cost and decay: roughly two hours for 20 items, and answer labels must be rechecked whenever the corpus changes, or the set rots and you misread the drop as a system regression.
Expected follow-up: how do you avoid overfitting to the eval set? Keep a held-out slice that never informs tuning, and refresh it from real production questions.
分析过程 · 先想清楚再作答
这题的区分度在「出题方向」和「规模的理由」两处。开口就说「找几百个用户真实问题」的，多半没真做过——真实问题的答案在哪篇文档里，没人标得出来。
先给方向：从语料反向出题，打开每一篇读它能回答什么，出题的那一刻答案文档就已经确定了，标注成本几乎为零。反方向（先想问题再找答案）会得到一堆自己都不知道答案的题。
再给结构：每题记问题、答案文档列表、类型三个字段；类型至少分单文档、多跳、无答案三类，并说明多跳必须全部答案文档命中才算命中，无答案不参与召回率而是考拒答。
规模的理由要给出来，不能只报一个数字：20 题能把「完全不能用」和「基本能用」分开，够做冒烟；100 到 200 题才有资格判断「涨了两个点」是真的还是噪声。上线之后每次线上出问题就把那个问题补进集合——评估集是长出来的。
补一句成本与保鲜：出题是人力活，20 题两小时是正常量级；语料更新后要复核答案文档还在不在，否则集合会悄悄腐烂，指标下跌你会误以为是系统坏了。
可预期的追问是「怎么防止评估集被过拟合」。答案是留一份不参与调优的保留集，并且定期从线上真实问题里补充新题，只用来验收不用来调参。
Key points
Author corpus-first so the answer document is known at authoring time.
Label every item with a type: single-document, multi-hop, unanswerable.
Multi-hop requires all answer documents; unanswerable items score abstention, not recall.
20 items for a smoke gate, 100 to 200 to trust small deltas, and keep growing it from production failures.
Hold out a slice that never informs tuning to avoid overfitting the set.
答题要点
从语料反向出题，出题时答案文档就已确定，标注成本最低。
每题标类型：单文档、多跳、无答案，三类缺一不可。
多跳要求全部答案文档命中；无答案不算召回率，考的是拒答。
20 题够冒烟，100 到 200 题才能判断小幅变化；线上故障持续补题。
留一份不参与调优的保留集，防止对评估集过拟合。
Recall, mean reciprocal rank, and normalized discounted cumulative gain - which failure mode does each one catch first, and what do you miss by watching only one?召回率、平均倒数排名、归一化折损累计增益，这三个检索指标分别在什么故障下会先掉下来？只盯一个会漏掉什么？
Common in ChinaCommon overseasIntermediate#retrieval-metrics#evaluation#ranking
How to reason about it · think before answering
This tests whether you know each metric's blind spot, not whether you can recite definitions. Layer them as 'did it show up / how high / how good overall' and you are halfway there.
Recall is boolean: is the answer document in the final context. It catches 'never retrieved', but it does not move when the answer slips from rank 1 to rank 8, as long as it still fits the budget.
MRR looks only at the rank of the first relevant hit, so ranking degradation shows up immediately. Its blind spot: one relevant item in the top ten scores exactly the same as five.
nDCG discounts every relevant hit in the top k by its position, so it tracks overall ranking quality and is the direct optimization target for reranking. Its blind spot is existence - it is zero both when nothing was retrieved and when ranking is terrible.
Conclusion: together they localize the failure. Recall drops means retrieval or chunking; recall flat but MRR down means ranking degraded, reach for a reranker; both stable but nDCG down means more noise crept into the top results.
Expected follow-up: what if a metric saturates? Make the questions harder - a saturated metric means the eval set lost its discriminative power, and further tuning is blind.
分析过程 · 先想清楚再作答
这题考的是「知不知道指标之间的盲区」，不是背定义。能把三者按「有没有 / 靠不靠前 / 整体好不好」分层的，基本就答对了一半。
推导链是这样的：召回率是布尔的——答案文档在不在最终上下文里。它对「压根没捞到」最敏感，但答案从第 1 名掉到第 8 名它一动不动，只要还在预算内。
倒数排名只看第一条相关结果的名次，所以「答案还在但被挤到后面」它立刻掉。反过来它有个盲区：前十条里有一条命中还是五条命中，它给的分完全一样。
归一化折损累计增益把前 k 名里每一条相关结果都按名次折算再累加，所以它对「整体排序质量」敏感，是重排最直接的优化目标。它的盲区是不告诉你「有没有」——召回率为零时它也是零，看不出是没捞到还是排得差。
结论：三个一起看才能定位故障层。召回率掉说明检索或切块出了问题，要动召回策略；召回率不动而倒数排名掉，说明排序退化，该上重排；两者都稳而 nDCG 掉，说明前几名里混进了更多噪声。
可预期的追问是「指标顶格了怎么办」。真实答案是把题目做难：指标撞天花板说明评估集失去区分度，这时候继续优化系统是在瞎调。
Key points
Recall answers 'did it make it into the context', sensitive to total misses, blind to rank shifts.
MRR answers 'how high is the first hit', sensitive to ranking degradation, blind to how many hits there are.
nDCG answers 'how good is the top k overall', the direct target for reranking, blind to existence.
Only the combination localizes the failure to retrieval, ranking, or noise.
State the hit criterion: context is packed against a token budget, not a fixed top-k.
答题要点
召回率管「有没有进上下文」，对完全没捞到最敏感，对名次变化不敏感。
平均倒数排名管「第一条排第几」，对排序退化最敏感，但分不清命中一条还是五条。
归一化折损累计增益管「前 k 名整体质量」，是重排的直接优化目标，但看不出有没有。
三者组合才能定位故障在召回层、排序层还是噪声层。
命中口径要说清：按 token 预算装上下文，不是按固定条数取前 k。
What systematic biases does an LLM judge have when scoring RAG faithfulness, and how do you detect them and prove your judge is trustworthy?用模型当裁判来评 RAG 的忠实度，有哪些系统性偏差？你怎么发现它们、又怎么证明你的裁判可信？
Common in ChinaCommon overseasDeep dive#llm-as-judge#evaluation#faithfulness
How to reason about it · think before answering
The second half of the question is the discriminator. Plenty of people can name position, length, and self-preference bias; few can say how they prove the judge is trustworthy.
Pair each bias with its mitigation: position bias - score pointwise instead of pairwise, and if you must compare, swap the order and call disagreement a tie; length bias - decompose into claims and score a ratio, so a longer answer grows its own denominator; self-preference - judge with a different vendor or tier than the generator.
Add two prompt-level requirements: fixed rubric anchors (spell out what 1.0, 0.6 and 0.3 mean, or the same input scores differently on different days) and forced structured output that quotes the unsupported sentences verbatim, which is what makes human review possible.
Proving trust has exactly one route: human spot-checks and an agreement rate. Stratify ten to thirty items across types, hits and misses, high and low judge scores; answer one binary question only - is anything here not in the material - and compare. Below 0.8 the judge's scores cannot gate a merge.
A detail that scores points: a very high agreement rate may mean your spot-check was too easy. If all ten sampled answers copy the material verbatim, agreeing is trivial and 100% says nothing about the judge.
Expected follow-up: can the judge itself break? Add probes - fixed inputs with known verdicts, one faithful and one obviously fabricated, checked on every run. An evaluation system fails silently: the numbers keep coming, they just stop meaning anything.
分析过程 · 先想清楚再作答
这题的题眼在后半句。能背出「位置偏好、长度偏好、自我偏好」三个名词的人很多，能说出「怎么证明可信」的很少——面试官要的是后者。
先把三个偏差和各自的缓解手段一一对应：位置偏好用逐条独立打分代替两两比较，非要比较就交换顺序跑两遍、结论不一致判平局；长度偏好用逐句判定加比例计分，写得越长分母越大，长度红利自动消失；自我偏好用跨供应商或跨档位的模型评判，生成和评判不同源。
再补两条提示词层面的：给死评分锚点，1.0 / 0.6 / 0.3 各自是什么必须写明，否则同一份输入不同天给的分都不一样；强制结构化输出并要求把没支撑的句子原样列出，这是人工复核的抓手。
证明可信只有一条路：人工抽检算一致率。分层抽十到三十条——各类型都要有、命中和没命中都要有、裁判给高分和低分都要有，只判一个二元问题（有没有材料外的内容），跟裁判的结论比对。低于 0.8 就不能拿它的分数做拦合并这类决策。
一个能加分的细节：一致率很高不一定是好消息。如果抽的十条都是「答案原样抄自材料」的简单题，判对是理所当然的，这时候 100% 说明的是抽检没难度，不是裁判可靠。
可预期的追问是「裁判本身会不会坏」。答案是给裁判写探针：喂几组已知正确答案的输入（照抄材料的、明显编造的），每次跑评估都验一遍——评估系统坏掉的方式最阴险，分数照常输出，只是不再有意义。
Key points
Three biases: position, verbosity, and self-preference, each with a matching mitigation.
Score pointwise rather than pairwise; decompose into claims and score a ratio to kill the length premium; never let the generator judge itself.
Pin rubric anchors in the prompt and force structured output that quotes unsupported sentences.
Establish trust through stratified human spot-checks and an agreement rate; below 0.8 the judge cannot gate merges.
Add probes with known verdicts so a broken judge is caught on every run.
答题要点
三个偏差：位置偏好、偏爱长答案、自己评自己，各自有对应的缓解手段。
逐条独立打分代替两两比较；逐句判定按比例计分抵消长度红利；生成与评判不同源。
提示词要给死评分锚点，并强制结构化输出、列出没支撑的句子。
可信度靠人工分层抽检算一致率，低于 0.8 不能用它做拦合并的决策。
给裁判本身写探针，每次跑评估都验一遍它有没有坏。
Why must a RAG evaluation set include questions the corpus cannot answer, and what does leaving them out hide?RAG 的评估集里为什么一定要放语料里没有答案的问题？不放会掩盖什么？
Common in ChinaCommon overseasBasic#evaluation#abstention#golden-set
How to reason about it · think before answering
It looks easy but really asks whether you have considered that the eval set itself can lie. 'To test the refusal path' is a pass; 'without them the worst failure is invisible in the report' is a full mark.
The derivation is one step: a system that always answers scores well on a set of answerable questions only. It stuffs context in, the model writes something, and the set has no column for 'should have refused'. The most dangerous failure simply does not appear.
Conclusion: unanswerable questions are the only thing that makes fabrication visible. They are excluded from recall and scored on abstention instead - did retrieval gate out every weak candidate, and did generation actually say the material does not cover this.
One authoring detail worth stating: unanswerable questions need strong distractor terms. Ask which browsers the web client supports when the corpus only says 'attach your browser and version when filing a ticket'. Without distractors retrieval returns nothing and you are testing your tokenizer, not your system.
Expected follow-up: what if the abstention rate is low? Check two layers - whether the retrieval score gate is effectively a no-op, and whether the generation prompt carries an explicit refusal instruction. You need both; a prompt alone is not a reliable gate.
分析过程 · 先想清楚再作答
这题看着简单，实际是在问「你有没有想过评估集本身也会说谎」。答成「为了测试拒答功能」只算及格，答出「不放会让某个故障在报表上完全不可见」才是满分。
推导只有一步：一个只会硬答的系统，在只有可答问题的评估集上能拿到很高的分——它每次都塞材料给模型，模型每次都编一段话，而评估集根本没有「应该拒答」这一栏。于是最危险的故障在报表上是不存在的。
结论：无答案问题是唯一能让「乱编」显形的东西。它不参与召回率，它的指标是拒答率——检索侧有没有把不够格的候选全挡下来，生成侧有没有真的说出「资料里没有」。
出题上有个必须说的细节：无答案问题必须留强干扰词，比如问「网页端支持哪些浏览器」而语料里恰好有一句「提交工单请附上浏览器与版本」。没有干扰词的无答案题检索器一条都捞不到，你测出来的是分词器不是系统。
可预期的追问是「拒答率低怎么办」。分两层查：先看检索侧的门槛是不是形同虚设（分数阈值定得太低，不相干的块也过关），再看生成侧的提示词有没有明确的拒答指令，两层都要有，只靠提示词兜是不牢的。
Key points
An all-answerable eval set makes 'answers confidently when it should not' completely invisible.
Unanswerable items are scored on abstention, not recall, and you check both the retrieval gate and the generation refusal.
Author them with strong distractor terms, or retrieval returns nothing and you are testing the tokenizer.
Keep them at roughly 15% or more of the set, alongside multi-hop items, as the coverage floor.
A low abstention rate splits into two causes: a no-op retrieval score gate, or a missing refusal instruction in the prompt.
答题要点
只有可答问题的评估集，会让「不知道也硬答」这个故障完全不可见。
无答案问题不算召回率，它的指标是拒答率，检索侧和生成侧各看一层。
出题必须留强干扰词，否则检索器一条都捞不到，测的是分词器。
建议无答案题占比不低于评估集的一成五，跟多跳题一起构成覆盖度底线。
拒答率低要分两层查：检索门槛是否形同虚设，生成提示词有没有拒答指令。

D9 Hybrid Search and Reranking: Two-Path Retrieval, Reciprocal Rank Fusion, Then Re-Ranking the Top Results With a Cross-Encoder

Why do hybrid retrieval systems usually use reciprocal rank fusion instead of normalizing both scores and adding them with weights? When does the weighted approach break down?混合检索为什么普遍用倒数排名融合，而不是把两路分数归一化之后加权相加？加权那条路在什么情况下会失控？
Common in ChinaCommon overseasIntermediate#hybrid-search#rank-fusion
How to reason about it · think before answering
The hinge word is `scores`. Answering `RRF is simpler` is reciting a concept; the interviewer wants to hear that you know why the two scores are not comparable in the first place.
Start with scale: BM25 is an unbounded sum of log terms, and on one index the top hit can range from 5 to 50 depending on the query; cosine is pinned between -1 and 1. Adding those two readings is meaningless.
Then name the silent failure of normalization: dividing by the per-route maximum makes the denominator float with the query. For a question with no answer in the corpus, the vector route's best hit may score 0.09 and still normalize to a perfect 1.0, entering the fusion at full weight. You think you are comparing relevance; you are comparing `tallest among the short`.
Then the maintenance cost of weights: a 1-to-0.6 ratio has to be tuned against an eval set, tuning two routes is a 2-D search, adding multi-query retrieval makes it 4-D or 5-D, and swapping the embedding model invalidates all of it. RRF has a single k, and the default of 60 rarely needs touching.
Conclusion: rank is the only thing the two routes share. RRF throws the scores away on purpose so that an incomparable quantity cannot mislead it.
Expected follow-up: what does k do? It flattens — the larger k is, the smaller the gap between the top few ranks, so `ranked well by both routes` outweighs `ranked first by one route`, which is exactly the cross-validation effect hybrid retrieval is after. A second follow-up on ties: you must fall back to sorting by document id, or ranks drift between runs and every eval number wobbles with them.
分析过程 · 先想清楚再作答
这题的题眼在「分数」两个字。只答「RRF 更简单」是背概念，面试官想听的是你知道分数为什么不可比。
先给量纲差异：BM25 是一堆对数项累加，没有上界，同一套索引里不同查询的第一名可以从 5 分到 50 分；余弦被钉死在负一到正一。两个读数相加没有意义。
再点出归一化的静默失败：除以本路最高分之后，分母随查询浮动。一个语料里根本没有答案的问题，向量那一路最高分只有 0.09，归一化之后照样是满分 1.0 带权重进融合——你以为在比相关性，其实在比「本路矮子里有多高」。
然后是权重的维护成本：1 比 0.6 这个配比要靠跑评估调出来，两路是二维搜索，加上多路查询就是四维五维，而且换一个 embedding 模型全部作废。RRF 只有一个 k，而且 60 这个默认值几乎不用动。
结论：名次是两路唯一可比的东西。RRF 主动扔掉分数，是为了不被不可比的量误导。
可预期的追问：那 k 是干什么的？答 k 是压平器——k 越大，头几名之间的差距越小，于是「两路都排进前列」比「一路排第一」更有分量，这正是混合检索想要的交叉验证效果。再追问同分怎么办，答必须按文档 id 兜底排序，否则跨次运行名次会飘、评估数字跟着抖。
Key points
BM25 is unbounded, cosine is bounded; the two scales are not comparable, so adding them is meaningless.
Per-route max normalization has a denominator that floats with the query, so the least relevant hit of an unanswerable query still normalizes to 1.0.
Weights must be tuned against an eval set, the search is high-dimensional once you add routes, and swapping models invalidates it; RRF has a single constant k.
RRF consumes only the ordered id list from each route, because rank is the one thing the routes share.
Larger k rewards `ranked well by both routes`; ties must fall back to document id so results are reproducible.
答题要点
BM25 无上界、余弦有界，两个量纲不可比，直接相加没有意义。
按本路最高分归一化的分母随查询浮动，无答案的查询里最不相干的结果也能拿到满分。
权重要跑评估调，路数一多就是高维搜索，换模型还得重来；RRF 只有一个常数 k。
RRF 只吃每一路的有序 id 列表，名次是两路唯一可比的东西。
k 越大越奖励「两路都排进前列」；同分必须按 id 兜底排序才可复现。
Why is a cross-encoder more accurate than a bi-encoder? And if it is more accurate, why not just use it to search the whole corpus directly?交叉编码器为什么比双编码器准？既然更准，为什么不干脆拿它直接检索全库？
Common in ChinaCommon overseasBasic#cross-encoder#bi-encoder
How to reason about it · think before answering
This is a giveaway question, but the discriminating half is the second part. Saying `cross-encoders are slow` is not enough; you have to point at the structural reason.
Start with the structure: a bi-encoder encodes query and document **separately** into vectors that never meet until a single dot product at the end; a cross-encoder concatenates query and document into one sequence, so every attention layer lets query tokens attend to document tokens.
That yields the accuracy gap: a bi-encoder must compress a document into one fixed-length vector, and compression loses information — the binding between `Zhou Min` and `platform team lead` may not survive. A cross-encoder does not compress; it aligns them on the spot.
The answer to the second half hides in the same structure: bi-encoder document vectors can be computed **offline** and indexed, so query time is just a vector search. A cross-encoder has nothing to precompute — N documents means N forward passes. Reranking a 100k-chunk corpus means pushing the entire corpus through a model on every question.
So the engineering split is a division of labor: recall pulls a small batch out of the whole corpus (cheap, indexable), reranking fixes the order of that batch (expensive, accurate). The default is to rerank only the top 20 after fusion.
Expected follow-up: is there a middle path? Yes — late interaction, where token-level document representations are precomputed and the interaction happens at query time. Accuracy and cost land between the two, at the price of a much larger index.
分析过程 · 先想清楚再作答
这是一道送分题，但送分题的区分度在第二问。只答「交叉编码器慢」是不够的，要说清慢在结构上的哪一处。
先给结构差异：双编码器把查询和文档**各自**编码成向量，两者从头到尾没有见过面，最后只靠一次内积凑到一起；交叉编码器把查询和文档拼成一段文本一起过模型，每一层注意力都能让查询的词去看文档的词。
由此推出准确率差异的来源：双编码器要把一篇文档压成一个固定长度的向量，压缩必然丢信息，「周敏是平台组组长」里两个词的绑定关系未必留得下来；交叉编码器不压缩，它当场对齐。
第二问的答案就藏在同一个结构里：双编码器的文档向量**可以离线算好**，查询时只做向量检索；交叉编码器没有任何东西能预先算好，N 篇文档就要跑 N 次前向。十万块的语料重排一遍，等于每次提问都把整个库过一遍模型。
所以工程上的定位是分工：召回负责在全库里捞出一小批（便宜、可索引），重排负责把这一小批的顺序改对（贵、准）。默认只重排融合后的前 20 条。
可预期的追问：有没有中间路线？答有——后期交互（late interaction）那一类，文档侧提前算好词级表示、查询侧当场做交互，精度和成本都在两者之间，代价是索引体积大得多。
Key points
A bi-encoder encodes both sides separately and joins them with one dot product; a cross-encoder concatenates them so attention can align across the pair.
The accuracy gap comes from compression: a bi-encoder squeezes a whole document into one vector and loses bindings; a cross-encoder does not compress.
Bi-encoder document vectors can be computed offline and indexed; a cross-encoder has nothing to precompute.
Reranking the full corpus means running every chunk through a model on every question, so cost scales linearly with corpus size.
The standard split is recall plus rerank, with reranking applied only to the top few dozen after fusion.
答题要点
双编码器各自编码、最后一次内积；交叉编码器把查询和文档拼在一起过模型，注意力可以跨两者对齐。
准确率差异来自压缩：双编码器把整篇文档压成一个向量，绑定关系会丢；交叉编码器不压缩。
双编码器的文档向量能离线算好并建索引，交叉编码器没有任何东西可以预先算好。
全库重排等于每次提问把整个语料过一遍模型，成本随语料规模线性增长。
标准分工是召回加重排，重排只作用于融合后的前几十条。
You replaced pure vector retrieval with hybrid search plus reranking, and after shipping it your eval metrics went down. How do you investigate?你把纯向量检索换成了混合检索加重排，上线之后评估指标反而掉了。你会怎么排查？
Common in ChinaCommon overseasDeep dive#hybrid-search#evaluation
How to reason about it · think before answering
This question tests whether you have actually done stage-by-stage attribution. Answering `I would tune the weights and see` loses — that is guessing, not investigating.
Step one is to run the stages apart, not to change code: pure keyword, pure vector, hybrid, and hybrid plus rerank, all on the **same eval set with the same context budget**. Whichever stage the drop appears in is where you look, and this alone separates `fusion is broken` from `reranking is broken`.
Step two asks a specific question: did recall drop, or did the ranking metrics drop? A recall drop means the answer never entered the context at all — a candidate-pool or budget problem. Ranking metrics dropping while recall holds means the answer is still there but pushed down — a fusion-weight or rerank-model problem. The two failures have completely different fixes.
A third common root cause is recall depth. This knob runs against intuition: going deeper is not safer, it lets noise vote too. On a 134-chunk corpus I measured that narrowing each route from 50 to 5 took hybrid recall from 87.5% back to 93.8% and multi-hop from 50% to 75%, while nDCG fell by almost 0.1. The metrics fight each other, so decide which one the product needs first.
A fourth root cause is that the eval protocol quietly changed. Touch the context budget, the hit rule, or the candidate depth, and the old and new numbers stop being comparable — in which case the `drop` may not be a drop at all.
Expected follow-up: how do you avoid this next time? Make the four-way comparison a single command, store the previous report as a baseline, and fail the build with a non-zero exit code on regression. That is precisely why evaluation comes before optimization.
分析过程 · 先想清楚再作答
这题考的是你有没有真的做过分阶段归因。答「调一下权重再看看」就输了——那是在猜，不是在查。
第一步是拆档跑，不是改代码：纯关键词、纯向量、混合、混合加重排四档在**同一份评估集、同一个上下文预算**下各跑一遍。指标掉在哪一档就在哪一档找原因，这一步能立刻区分「融合坏了」和「重排坏了」。
第二步问一个具体问题：掉的是召回率还是排序指标？召回率掉说明答案根本没进上下文，是候选池或者预算的问题；排序指标掉而召回率没动，说明答案还在、只是被挤到了后面，那是融合权重或重排模型的问题。这两类故障的解法完全不同。
第三个常见根因是召回深度。每路取多少条这个旋钮方向反直觉：取深了不是更保险，是把噪声也一起投了票。我在一份 134 块的语料上实测过，每路从取 50 收到取 5，混合那一档的召回率从 87.5% 回到 93.8%、多跳档从 50% 回到 75%，而 nDCG 反而掉了近 0.1——两个指标会打架，先想清楚业务要哪个。
第四个根因是评估口径被悄悄改了。上下文预算、命中判定、候选池深度只要动过一个，新旧数字就不可比，这时候「掉了」可能根本不是真的掉了。
可预期的追问：怎么防止下次再踩？答把四档对照做成一条命令、把上一版报告存成基线、指标退步就以非 0 退出码拦住合并——这就是评估要先于优化的原因。
Key points
Run all four configurations separately for attribution, on one eval set with one context budget, before touching any parameter.
Separate a recall drop from a ranking drop: the first is a candidate-pool or budget issue, the second is a fusion or rerank issue.
Check recall depth: taking too many per route lets noise vote, and narrowing it can bring recall back.
Confirm the eval protocol did not change; touching budget, hit rule, or candidate depth makes old and new numbers incomparable.
Freeze the four-way comparison into one command plus a baseline report, and block merges on regression.
答题要点
先拆档跑四种配置，在同一份评估集和同一个上下文预算下归因，不要一上来就调参。
区分召回率掉与排序指标掉：前者是候选池或预算问题，后者是融合或重排问题。
查召回深度：每路取太深会把噪声也投进融合，收窄反而可能救回召回率。
确认评估口径没被改：预算、命中判定、候选池深度动过一个，新旧数字就不可比。
把四档对照固化成一条命令加一份基线报告，指标退步直接拦住合并。
Adding a reranker costs you 200 ms of extra latency per question plus a per-search fee. How do you decide whether that spend is worth it?加上重排之后每次提问多了两百毫秒延迟，还多了一笔按次计费的开销。你怎么判断这笔钱该不该付？
Common in ChinaCommon overseasDeep dive#rerank#cost-tradeoff
How to reason about it · think before answering
This question tests whether you can translate a technical choice into a business judgment. Answering `check whether the metrics went up` covers only a third of it.
Split it into three ledgers: how much the metrics moved, how much latency grew, and how much money it costs. All three must be reported together; a proposal with only the first will not survive review.
For the first ledger, be specific about **which** metric reranking improves. Reranking changes the order, not the candidate set — it cannot fix `the answer was never retrieved`. If your recall is the bottleneck, add a retrieval route or adjust recall depth first; the 200 ms buys nothing.
For the second, ask where those 200 ms land. They sit synchronously between retrieval and generation, with the user waiting; but if a streaming generation follows and time-to-first-token is already a second or two, the relative cost is small. In an as-you-type search box, 200 ms is fatal.
For the third, note the billing unit: rerankers usually charge per search rather than per token, so sending a few more candidates barely changes the bill — what is expensive is the number of questions. That points optimization at reducing query volume (caching, intent routing) rather than at trimming the candidate list.
Expected follow-up: what if you simply cannot afford it? Three paths — rerank only queries classified as hard (intent routing), cache results, or self-host an open-weights cross-encoder to convert per-call fees into fixed compute cost.
分析过程 · 先想清楚再作答
这题考的是你会不会把技术选择翻译成业务判断。只答「看指标涨没涨」只答了三分之一。
先把账拆成三笔：指标涨了多少、延迟涨了多少、钱涨了多少。三笔必须一起报，只报第一笔的方案在评审会上过不去。
第一笔要问清楚重排改善的是**哪个**指标。重排改的是顺序，不是候选集合——它救不了「答案压根没被召回」这种故障。如果你的召回率本来就不够，先去加召回路数或者调召回深度，重排这两百毫秒是白花的。
第二笔要看这两百毫秒落在哪。它是同步卡在检索之后、生成之前的，用户全程在等；但如果后面接的是一个流式生成、首字节本来就要一两秒，这两百毫秒的相对占比就小得多。反过来，如果这是一个自动补全式的即时搜索框，两百毫秒就是致命的。
第三笔要注意计价单位：重排普遍按检索次数计价而不是按 token，所以「多送几条给它排」几乎不涨钱，真正贵的是提问次数本身。这直接决定了优化方向是压提问量（缓存、意图路由）而不是压候选数。
可预期的追问：如果就是付不起怎么办？答三条路——只对判定为复杂的查询走重排（意图路由）、把结果缓存起来、或者换成自部署的开源交叉编码器把按次付费变成固定的算力成本。
Key points
Report all three ledgers together: metric gain, latency growth, cost growth; a proposal missing one is incomplete.
Confirm whether the bottleneck is ordering or recall first; reranking only reorders and cannot rescue an answer that was never retrieved.
Judge the latency by where it lands: it is small relative to a streaming generation, but fatal in an as-you-type search box.
Rerankers bill per search rather than per token, so cost scales with question volume, not candidate count.
If it is unaffordable: route only hard queries to the reranker, cache results, or self-host an open-weights cross-encoder.
答题要点
三笔账一起报：指标增量、延迟增量、成本增量，缺一笔方案就不完整。
先确认瓶颈是排序还是召回：重排只改顺序，救不了没被召回的答案。
延迟要看落在哪：流式生成场景下相对占比小，即时搜索框里两百毫秒就是致命的。
重排按检索次数计价而不是按 token，涨钱的是提问量而不是候选条数。
付不起时的三条路：意图路由只对难查询重排、结果缓存、换自部署的开源交叉编码器。

D10 Query-Side Optimization: Rewriting, Hypothetical Document Embeddings, Multi-Query, Step-Back Prompting, and Intent Routing

Why does HyDE (hypothetical document embeddings) work, and when does it steer retrieval in the wrong direction?假设文档嵌入（HyDE）为什么有效？它在什么情况下会把检索带偏？
Common in ChinaCommon overseasIntermediate#hyde#query-transformation#retrieval-quality
How to reason about it · think before answering
The tell is in the second half. Anyone can recite why HyDE works; only someone who has run it on real data can say when it hurts.
Give the mechanism first: dense retrieval compares semantic similarity, but a user's question and a policy paragraph differ in register, syntax and vocabulary. HyDE has the model draft a fake passage that looks like the target document, then retrieves with that vector — effectively moving the query into the documents' register.
Then kill the common misreading: the factual accuracy of the draft does not matter, because it is never shown to the user. It only contributes a direction in embedding space.
Two failure modes. The model invents an over-specific field or process name that does not exist in the corpus, and the vector chases something imaginary. Or the corpus genuinely has no answer, and the fabricated passage finds plausible-looking neighbours anyway — abstention rate drops and hallucination rate climbs.
Pair the risk with a mitigation: gate admission on each retriever's raw score, never on the fused score (fused scores are relative, so even the worst batch tops out at 1.0); and treat the hypothetical document as a second query fused with the original rather than a replacement, so a bad draft can only dilute the signal, not erase it.
Expect the follow-up on cost. The draft runs to a hundred-plus output tokens, an order of magnitude more than a rewrite, and it doubles retrieval calls. That is why it belongs in an A/B queue, not in the default config.
分析过程 · 先想清楚再作答
这题的题眼在后半句。前半句网上到处都能抄到，能不能说清「什么时候不该用」才是区分度所在——只答前半句的人，多半没在真实语料上跑过。
先给机制：向量检索比的是语义相似度，而用户的疑问句和文档里的制度条文在文体、句式、用词上都不同类。HyDE 先让模型编一段「长得像目标文档」的假文本，用它的向量去找邻居，等于把查询搬进了文档所在的那个语域。
紧接着点破一个常见误解：这段假文本的**事实对不对根本不重要**，因为它不给用户看，只贡献一个向量方向。理解到这一层，才算真懂它为什么不怕模型瞎编。
带偏有两种典型情况。一是模型编得太具体，给出语料里根本不存在的字段名或流程名，向量朝着一个不存在的方向去了；二是语料里压根没有答案，本该拒答的问题被编出来的假文档匹配到几个「看起来挺像」的邻居，拒答率掉下去、瞎编率涨上来。
说完风险要给对策，这一步最见工程经验：门槛卡在**每一路检索器的原始分**上而不是融合分上（融合分是相对的，最不相干的一批也能拿最高分）；以及把假设文档当成**第二个检索式与原问题融合**，而不是直接替换原问题——替换在模型编歪时会把原问题的信号一起丢掉。
可预期的追问是「它多花多少钱」。答：假设文档要写上百字，输出 token 是查询改写的十几倍，是查询侧四种手法里最贵的一次调用，而且检索次数翻倍。所以它通常不该默认打开，应该进 A/B 队列。
Key points
It works by register alignment: a question and a policy paragraph sit in different neighbourhoods, and the fake passage moves the query into the document's.
The draft's factual accuracy is irrelevant — it only supplies a direction and is never shown to the user.
It misfires when the model invents over-specific details, or when the corpus has no answer and the fabrication finds plausible neighbours anyway.
Two guardrails: gate on raw per-route scores, not fused ones; fuse the hypothetical document with the original query instead of replacing it.
It is the most expensive query-side technique (long output plus doubled retrievals), so keep it off by default and A/B it.
答题要点
有效的原因是语域对齐：疑问句和制度条文本来不在一个语义邻域，假设文档把查询搬到了文档那一侧。
假文本的事实对错不重要，它只贡献一个向量方向，不展示给用户。
带偏的两种情况：编得太具体，追一个语料里不存在的方向；本该拒答的问题被假文档匹配上，拒答率下降。
两条护栏：门槛卡原始分不卡融合分；把假设文档当第二个检索式融合，而不是替换原问题。
成本上它是查询侧最贵的一项（长输出加检索次数翻倍），默认关闭、按场景 A/B。
How do you handle coreference in multi-turn RAG, and what is the classic failure when you skip it?多轮对话里怎么处理指代？不做指代消解最典型的翻车场景是什么？
Common in ChinaCommon overseasBasic#coreference#multi-turn#query-rewriting
How to reason about it · think before answering
This is a warm-up question, but there is still a gap between answers. Saying "just concatenate the history into the query" invites a follow-up about growing histories that most candidates cannot handle.
State the mechanism: insert a short rewrite call before retrieval that takes the last few turns plus the current question and returns one retrieval-ready line. Set temperature to 0 so the same input always yields the same query, and forbid the model from answering the question in the prompt.
Explain why concatenation is worse: history grows without bound, filler words dilute inverse document frequency, and the previous answer leaks in — you end up retrieving an answer with an answer. The rewriter emits one sentence, not a transcript.
Make the failure concrete. Turn one: "who must sign off on this operation?" Answer: "the platform team lead." Turn two: "what is that person's name?" Retrieved unresolved, not a single candidate clears the admission gate and the system refuses — even though the corpus contains the answer. The failure is not a wrong answer, it is a false "not found" right after the user's own question.
Add the ordering trap: rewrite before intent routing. A pronoun is a classic multi-hop signal, so an unresolved query gets routed to the expensive path for nothing; after rewriting it is an ordinary single-hop question. Multi-query and step-back must also sit downstream of the rewrite, or one unresolved pronoun becomes three.
Expect "how do you decide when to rewrite?" Trigger on short queries, pronouns and elliptical follow-ups; skip on a clearly new topic. The check is nearly free and removes most of the calls.
分析过程 · 先想清楚再作答
这是一道送分题，但送分题也有高下之分：只说「把历史拼进查询里」的答案，会被追问一句「历史越拼越长怎么办」就卡住。
先把做法说清：在检索之前加一次很短的改写调用，输入是最近几轮对话加本轮问题，输出是一行可以直接检索的检索式；温度设 0 保证同一句话每次改成同一个结果，并在提示词里明确禁止模型顺手回答问题。
为什么不是「把历史整个拼进查询」：历史越拼越长，噪声词把逆文档频率摊薄，检索反而更差；而且历史里包含上一轮的答案，等于拿答案去检索答案。改写的产出是一句话，不是一段历史。
最典型的翻车场景要举实例：上一轮问「这个操作必须由谁审批」，答「必须由某某组组长审批」；这一轮问「这个人叫什么名字」。不消解直接检索这七个字，实测是**一条候选都过不了门槛，系统只能拒答**。注意失败方式不是答错，是「明明语料里有答案却说找不到」，用户体验是崩塌式的。
补一条顺序上的坑：改写必须在意图路由**之前**。「这个人」是典型的多跳信号词，路由看到它会判成多跳、白跑一轮；改写之后它只是个普通单跳问题。同理，多路查询、后退提问也都要建立在改写后的那句话上，否则错误被放大好几倍。
可预期的追问是「怎么知道要不要改写」。答：短问题、含指代词、含省略（「那审计日志呢」）时才触发，纯新话题跳过——这一步很便宜，但能省掉一大半调用。
Key points
Add a short rewrite call before retrieval: last few turns plus current question in, one retrieval line out, temperature 0, answering explicitly forbidden.
Do not splice the whole history into the query — it grows unbounded, dilutes IDF, and leaks the previous answer into the search.
Classic failure: an unresolved pronoun means no candidate clears the gate, so the system refuses a question the corpus can answer.
That false "not found" hurts more than a wrong answer, since the user just asked about the same thing.
Order matters: rewrite first, then route; multi-query and step-back both build on the rewritten query.
答题要点
在检索前加一次短改写调用，输入最近几轮加本轮问题，输出一行检索式，温度 0，禁止模型回答问题。
不要把历史整段拼进查询：越拼越长、噪声稀释逆文档频率，还会拿上一轮的答案去检索。
典型翻车：上一轮的「这个人 / 他 / 那个」不消解，检索一条都过不了门槛，系统在有答案的情况下拒答。
失败方式是「假的查不到」，比答错更伤体验，因为用户刚刚才问过同一件事。
顺序：先改写、再路由，多路查询与后退提问都建立在改写后的查询上。
Query rewriting adds a model call per question and doubles end-to-end latency. How do you decide whether it is worth paying?上线查询改写之后每问多了一次模型调用，端到端延迟涨了一倍，你怎么判断这笔开销值不值？
Common in ChinaCommon overseasDeep dive#cost-tradeoff#latency#query-rewriting
How to reason about it · think before answering
This question is about turning an engineering judgement into numbers. "Rewriting obviously helps quality" is a fail — the candidate never measured the gain.
Start with one question that nearly settles it: which slice of traffic does the gain land on? Query rewriting buys almost nothing on single-turn questions (we measured identical metrics with it on and off across 20 single-turn items); the entire payoff is in follow-up turns. So step one is to pull the share of multi-turn sessions from production logs.
Step two is to lay out all three ledgers, because one alone cannot support a decision: how much the metrics moved on a fixed golden set, how much latency grew (a rewrite is a short-output task, so a cheap fast model often costs a few hundred milliseconds rather than doubling anything), and how many extra calls were added — one model call for rewriting versus one call plus several retrievals for multi-query is a completely different cost shape.
Step three is to price the cheaper variants before deciding: rewrite only when a trigger fires (short query, pronoun, ellipsis), cache rewrites per session, and run a small model instead of the main one. These usually remove most of the cost while keeping the gain.
Land on a usable rule: gain times affected traffic share, divided by added latency and cost, ranked against your other candidate optimisations. Rewriting usually ranks high because its failure mode is a false "not found" immediately after the user's own question — an abandonment-grade experience bug, not a few metric points.
Expect "what if the latency genuinely is unacceptable?" Fire the rewrite and the first retrieval in parallel: search with the raw query immediately, search again when the rewrite returns, and fuse both rankings. You pay a max instead of a sum, at the cost of one extra retrieval.
分析过程 · 先想清楚再作答
这题考的是「能不能把工程判断落到数字上」。凡是回答「改写当然要做，能提升效果」的，一律判为没做过——他连收益是多少都没量。
先问自己一句：**收益出现在哪一类流量上**。这一条几乎决定了答案。查询改写在单轮问答上的收益接近零（我们在 20 道单轮题上实测开关它指标一模一样），收益全在多轮追问。所以第一步是去线上日志里查多轮会话占比，占比很低的话这笔钱不该花在全量流量上。
第二步是把三笔账摆齐，缺一笔就不能下判断：指标涨了多少（用固定的标准答案集跑，不要用感觉）、延迟涨了多少（改写是短输出任务，可以换便宜快的那一档模型，往往只多两三百毫秒而不是翻倍）、多了几次调用（改写是一次，多路查询是一次调用加几次检索，成本结构完全不同，别混着算）。
第三步是找**便宜的替代路径**再比一次：只在命中触发条件时才改写（短问题、含指代词、含省略），纯新话题直接跳过；改写结果按会话缓存；用小模型跑改写而不是主模型。这三招通常能把这笔开销压掉一大半，而收益几乎不掉。
结论要落成一条可执行的判据：**收益乘以受影响流量占比，除以增加的延迟与成本**，跟你手上其他候选优化排个序。改写通常能排到很前面，因为它的失败方式是「用户明明追问同一件事却被告知查不到」，那是会直接导致弃用的体验故障，不只是指标掉几个点。
可预期的追问是「延迟真的不能接受怎么办」。答：把改写和第一次检索**并行发**，用原查询先检索一路，改写回来后再补一路，两路用倒数排名融合合起来——延迟只多一个 max 而不是一个加法，代价是多一次检索。
Key points
Locate the gain first: rewriting is near-zero on single-turn traffic and pays off on follow-ups, so start from the share of multi-turn sessions.
All three ledgers are mandatory: metric delta on a golden set, added latency, added calls and token cost.
Try the cheap variants before deciding: conditional triggering, per-session caching, and a small model for the rewrite.
Decide on gain times affected traffic share over added latency and cost, then rank it against your other optimisations.
If latency is a hard constraint, fire the rewrite in parallel with the first retrieval and fuse both rankings, turning a sum into a max.
答题要点
先定位收益落在哪一类流量：改写在单轮上接近零收益，价值全在多轮追问，先查多轮会话占比。
三笔账缺一不可：标准答案集上的指标变化、增加的延迟、增加的调用次数与 token 成本。
先试便宜的替代路径：条件触发、按会话缓存、用小模型跑改写，通常能压掉大半开销。
判据是「收益 × 受影响流量占比 ÷ 增加的延迟与成本」，再和其他候选优化排序。
延迟真的卡死时，把改写与首次检索并行发，两路名次用倒数排名融合，延迟从加法变成取最大值。
What happens when intent routing misclassifies, and how would you design the fallback?意图路由判错了会怎样？你会怎么设计兜底？
Common in ChinaCommon overseasIntermediate#intent-routing#fallback#observability
How to reason about it · think before answering
This tests whether you have thought about the direction of the error. A router is a classifier and classifiers misfire; "add more training data" is not a fallback design.
Break the errors down by direction — that is the backbone of the answer. Across three routes (direct answer, single-hop, multi-hop) the six confusions carry wildly asymmetric costs. Routing a retrieval-worthy question to a direct answer leaves the model with no material at all, so it fabricates: the most expensive error. Routing chit-chat to single-hop merely wastes one retrieval. Routing multi-hop to single-hop just yields an incomplete answer.
The conclusion follows: bias the fallback toward spending a little more, and default to single-hop retrieval whenever the classifier is unsure. Single-hop is the cheapest error to make, and it is recoverable — with partial material the model can still say it only found half the answer; with no material it can only invent one.
Add a runtime fallback, which beats better up-front classification: after a direct-answer routing, if the draft reply contains figures, amounts or dates that need a source, fall back to retrieval and answer again; after a single-hop routing, if no candidate clears the admission gate, escalate to multi-hop or abstain. Correcting the earlier decision with the later observation is the single most useful pattern in routing systems.
Mention observability: log every routing decision with the raw question, the label, and whether a fallback fired. Without that log you know neither how accurate the router is nor what to train the next version on.
Expect "when should you skip routing entirely?" When chit-chat is a small share of traffic and multi-hop questions are rare, the classification call costs more than it saves. In our 30-document lab the real gain from routing was not saved retrievals but the ability to give recognised multi-hop questions a larger context budget.
分析过程 · 先想清楚再作答
这题在考「有没有想过错误的方向」。路由是分类器，分类器一定会错；只答「多加训练数据提高准确率」的，等于没回答兜底怎么设计。
先把错误按方向拆开，这一步是整题的骨架：三条路（直接回答、单跳检索、多跳检索）两两误判，代价完全不对称。把该检索的判成直接回答，模型手里一点材料都没有，只能编，这是最贵的一种错；把闲聊判成单跳，只是白花一次检索；把多跳判成单跳，只是少查一轮、答得不全。
结论顺势就出来了：**兜底方向要偏向「多花一点钱」，判不出来一律退回单跳检索。** 单跳是三条路里错得最轻的一条，而且它的错误是可恢复的——材料不全模型还能说「资料里只查到一半」，材料为空它就只能编。
再补一层运行时兜底，比事前分类更管用：分类成直接回答之后，如果模型的回答里出现了具体数字、金额、日期这类需要出处的内容，就回退去检索一次再答；分类成单跳之后，如果检索侧一条都没过门槛，就升级走多跳或直接拒答。**用后一步的观测结果纠正前一步的判断**，这是路由系统最实用的一条设计。
还要提一句可观测性：路由的每一次判定都要落日志，带上原始问题、判定结果、后续是否发生了兜底升级。没有这份日志，你既不知道路由准不准，也没法攒出下一版的训练集。
可预期的追问是「什么时候干脆别做路由」。答：流量里闲聊占比很低、且多跳问题很少时，路由省下的钱还不够付分类调用的钱，这时候直接全部走单跳更划算——我们在 30 篇语料的实验里就看到，路由真正的收益并不在省检索，而在于认出多跳之后给它更高的上下文预算。
Key points
The three routes have asymmetric error costs: sending a retrieval-worthy question to a direct answer is the worst, while routing chit-chat to single-hop only wastes one retrieval.
Bias the fallback toward spending more: default to single-hop whenever the classifier is unsure, since that error is the mildest and is recoverable.
Add runtime fallbacks: re-retrieve if a direct answer contains figures that need a source; escalate or abstain if no single-hop candidate clears the gate.
Log every routing decision — raw question, label, whether a fallback fired — for both monitoring and the next training set.
When chit-chat and multi-hop are both rare, the classification call costs more than it saves; route everything to single-hop instead.
答题要点
三条路的误判代价不对称：把该检索的判成直接回答最贵（模型没材料只能编），把闲聊判成单跳只是白花一次检索。
兜底方向偏向多花钱：判不出来一律退回单跳检索，它是错得最轻且可恢复的一条路。
加运行时兜底：直接回答里出现需要出处的数字就补一次检索；单跳检索一条都没过门槛就升级或拒答。
每一次路由判定都落日志（原始问题、判定结果、是否触发兜底），既用于监控也用于攒下一版训练集。
闲聊与多跳占比都很低时，路由省的钱付不起分类调用，直接全走单跳更划算。

D11 Advanced Indexing: Parent-Child Documents, Summary Indexes, Contextual Retrieval, and the Trade-Offs of Tree Aggregation vs. Graph Retrieval

Parent-child indexing and contextual retrieval both patch the same problem — chunks losing their context. What actually distinguishes them?父子索引和上下文检索都在补『块被切碎』这个问题，它们的差别到底在哪？
Common in ChinaCommon overseasIntermediate#indexing#contextual-retrieval#chunking
How to reason about it · think before answering
The hinge is which half of the pipeline each one fixes. Answering 'one is a chunking trick, the other adds a prompt' just describes implementations; the interviewer wants to know where each acts.
Split the pipeline in two and ask separately: what does the retriever see, and what does the generator see. Parent-child changes the generation side — retrieval still runs on small chunks, but a hit is swapped for its parent. Contextual retrieval changes the retrieval side — the header exists so the chunk can be found at all, and the generator does not need it.
Conclusion: parent-child fixes 'found it but can't read it'; contextual retrieval fixes 'readable but never found'. Neither changes what the other changes, so they compose.
That difference also dictates which metric can see each one. Contextual retrieval moves rank, so recall and nDCG catch it. Parent-child moves 'is the evidence sufficient to answer', which a binary recall metric cannot see. Our 20-question set is already saturated at 100% on single-document questions, so parent-child comes out level with the baseline — that is the ruler failing, not the technique.
That difference yields a free optimization: since the header only serves retrieval, keep it out of the context window. Leaving it in pays rent on every single query. Flipping that one switch in our lab freed 36 tokens inside a 600-token budget with every metric unchanged.
The costs differ too. Parent-child costs index entries and a bigger context unit. Contextual retrieval costs one model call per chunk up front plus a permanently larger index. One is space; the other is time and space.
Expect the follow-up 'why not both'. Look at the failure logs first: are you mostly seeing incomplete evidence, or nothing retrieved at all? Without the matching failure mode, neither is worth its price.
分析过程 · 先想清楚再作答
这题的题眼是『补的是哪一半』。答成『一个是切块技巧、一个是加提示词』就是在描述实现，面试官想听的是它们各自作用在检索管道的哪一段。
拆的办法是把管道分成两段问：检索时看到什么、生成时看到什么。父子索引改的是**生成侧**——检索单位还是小块，只是命中之后把上下文单位换成大块；上下文检索改的是**检索侧**——块头拼进去是为了让这一块能被检索到，模型生成时并不需要它。
结论：父子索引解决『找到了但看不全』，上下文检索解决『看得全但找不到』。前者不改变谁被检索到，后者不改变模型看到多少。它们正交，可以叠加。
这个差别还决定了它们各自要用什么指标去量：上下文检索动的是名次，用召回率和 nDCG 量得到；父子索引动的是『材料够不够答』，召回率这种二值指标量不出来。我们那份 20 题评估集单文档档已经 100% 饱和，父子索引在表里跟基线持平——那不是它没用，是尺子量不了它。
顺着这条差异能推出一个立刻能用的优化：既然块头只服务检索，就不该进上下文。它进了上下文就是在每一次查询里白占预算，而且这笔钱是长期的。我们的实验里把这个开关一改，五列指标一个不变，600 token 的预算里多装进了 36 个 token。
代价也不同：父子索引的代价是索引条目变多、每次装进上下文的东西变大；上下文检索的代价是一次性要给每块调一次模型，加上索引 token 永久变大。前者是空间，后者是时间加空间。
可预期的追问是『那我全都上』。答案是先看失败案例：日志里是『材料不完整』多，还是『压根没检索到』多。没有对应的失败模式就不该上，这两个手法都不是免费的。
Key points
Parent-child acts on the generation side: retrieve small, swap in the parent for context. It fixes 'found but unreadable'.
Contextual retrieval acts on the retrieval side: the header makes the chunk findable. It fixes 'readable but never found'.
They are orthogonal and compose; keep the header in the index only, never in the context window.
Parent-child costs more index entries and a larger context unit; contextual retrieval costs one call per chunk plus a permanently larger index.
Pick based on the observed failure: incomplete evidence points to the former, zero retrieval to the latter.
答题要点
父子索引作用在生成侧：检索单位是小块，上下文单位换成父块，解决『找到了但看不全』。
上下文检索作用在检索侧：块头让块能被检索到，解决『看得全但找不到』。
两者正交可叠加；块头只该进索引不该进上下文，否则每次查询都在为它付钱。
父子索引的代价是索引条目与上下文单位变大；上下文检索的代价是一次性建索引调用加永久变大的索引。
选哪个看失败案例：材料不完整选前者，压根没检索到选后者。
Contextual retrieval needs one model call per chunk. How do you estimate that one-off cost, and what levers bring it down?上下文检索要给每个块调一次模型，这笔一次性成本怎么估？有哪些办法能压下来？
Common in ChinaCommon overseasDeep dive#contextual-retrieval#prompt-caching#cost
How to reason about it · think before answering
This checks whether you have actually done the arithmetic. Saying 'prompt caching makes it cheap' without knowing which line item it touches is a tell.
Split the bill first: one-off = per-chunk input + output + full re-embedding; per-query = the header read twice, once by the reranker and once in the context. Keep them separate, because they scale with completely different things.
The dominant term on the one-off side is how many times the same document is re-read. A doc split into n chunks is read n times. Prompt caching attacks exactly that: put the whole document first and mark it cacheable, pay a cache write once, then cache reads for the remaining n-1, typically an order of magnitude cheaper than input.
Order matters. Caching is prefix-matched, so the document must come first and the chunk after. Put the varying part first and the prefix changes every call — zero cache hits. This is the most common way people get it wrong.
Our measurement: 30 docs, 134 chunks. Without caching, 103017 input tokens; with caching, 17340 written plus 60137 read, cutting the one-off cost by roughly 29%. The finer the chunks, the bigger the saving, because re-reads multiply.
The counter-intuitive part is the useful part: the one-off cost amortizes below 10% of per-query cost after about 217 queries. The lasting bill is the extra tokens every query carries (we measured +12.3%). So the first lever is not cheaper index building — it is keeping the header out of the context, keeping it short, and not generating it for the whole corpus indiscriminately.
A bonus point: before spending any of it, confirm your evaluation setup can actually detect the benefit. In our offline harness the vector route contributed exactly zero unique answer documents, so it cannot answer whether headers help embeddings at all — an A/B run there hands you a wrong conclusion that looks numerically supported.
分析过程 · 先想清楚再作答
这题考的是你有没有真的算过账。只会说『用提示词缓存就便宜了』属于听过没做过——面试官会追问缓存到底省在哪一项上。
先把成本拆开：一次性 = 每块的输入 + 输出 + 全量 embedding；每次查询 = 块头在重排和上下文里各被读一遍。**这两笔要分开记**，因为它们随业务量的增长方式完全不同。
一次性那笔的主项是『同一篇文档被重复读了多少遍』。一篇切成 n 块就要读 n 遍，这是成本的大头。提示词缓存省的正是这一项：把整篇放在提示词最前面并标记为可缓存，第一块付一次缓存写入，后面 n-1 块只付缓存读取，而读取价通常比输入价低一个数量级。
顺序不能反：缓存按前缀匹配，整篇必须在前、块内容在后。把变化的块放前面，前缀次次都变，缓存一次都不会命中——这是最常见的翻车点。
我们的实测：30 篇、134 块，不开缓存输入 103017 token，开缓存后拆成写入 17340 加读取 60137，一次性成本降约 29%。**块切得越碎这个比例越高**，因为重复读的次数更多。
结论反直觉但很实用：一次性那笔是小钱，摊到 217 次查询就降到每次查询成本的一成以下；真正的长期账是每次查询多出来的那几十个 token（我们量到 +12.3%）。所以压成本的第一优先级不是压建索引，而是让块头别进上下文、别过长、别对全库无差别地生成。
最后一条是加分项：花这笔钱之前先确认你的评估环境**测得出**收益。我们的离线环境里向量路对召回的独立贡献实测为 0，所以它根本没法回答『块头对向量侧有没有用』——在这种环境里做的 A/B 会给你一个看起来有数字支撑的错误结论。
Key points
Split into one-off (per-chunk input/output plus re-embedding) and per-query (header read by both reranker and generator).
The one-off is dominated by re-reading each document n times; caching turns that into one write plus n-1 reads.
Caching is prefix-matched: the full document must come first, the chunk after, or you get zero hits.
Measured on 30 docs / 134 chunks, caching cut the one-off cost by about 29%, and finer chunks save more.
The lasting cost is per query: keep headers out of the context window, keep them short, and generate them selectively.
答题要点
把账拆成一次性（每块的输入输出 + 全量 embedding）和每次查询（块头在重排与上下文里各读一遍）两笔。
一次性的大头是同一篇被重复读 n 遍；提示词缓存把它压成一次写入加 n-1 次读取。
缓存按前缀匹配，整篇必须放在提示词最前面，块内容在后，顺序反了一次都不会命中。
实测 30 篇 134 块，一次性成本降约 29%，块越碎省得越多。
长期账在每次查询：块头别进上下文、控制长度、只对真正需要的文档生成。
What kind of question actually requires graph retrieval? Give one concrete case where it is justified and one where it is not.什么样的问题必须上图检索？给一个该上的具体例子和一个不该上的例子。
Common in ChinaCommon overseasDeep dive#graph-rag#multi-hop#cost
How to reason about it · think before answering
This one tests whether you reach for tools you don't need. If the answer is 'multi-hop questions need a graph', the interviewer knows you haven't shipped one — multi-hop is necessary, nowhere near sufficient.
Anchor the criterion on something observable: does the second required document share any lexical or semantic overlap with the query? If it does, ordinary hybrid retrieval will surface it and the hop is illusory. If it shares nothing, only a relation edge gets you there — that is graph territory.
Justified case: 'who must sign off on a production failover, and what is that person's name?' One doc says the platform lead must approve; another says who the platform lead is. The second shares not one term with the query. Across all five index structures we tested, it never once appeared in a 20-item candidate pool — rechunking, headers and parent backfill all failed.
Unjustified case: 'which process covers a capacity change, and how many working days ahead must the ticket be filed?' Also two documents, but both overlap the query lexically; hybrid retrieval ranked them second each, and one pass collected both. Building a graph for this buys a solved problem at several times the cost.
Then state the cost, which is what makes the answer sound operational: graph building is not one extraction call. Entities need disambiguation, relations need dedup, updates force recomputing affected subgraphs, and you now run a graph store and its update pipeline.
Expect 'what else could you do instead'. Hand multi-hop to agentic retrieval: let the model retrieve the intermediate entity first, then issue a second query with it. Near-zero build cost, paid back in latency and call count per query. Try that before you build a graph.
分析过程 · 先想清楚再作答
这题在考你会不会为了用而用。只要答案里出现『多跳问题就要上图检索』，面试官基本就知道你没落地过——多跳只是必要条件，远不是充分条件。
判据要落在一个可观察的现象上：**答案的第二篇文档和查询之间，有没有字面或语义上的重合**。有重合，普通的混合检索就能捞到它，多跳是假的；完全没有重合，只能靠一条关系边走过去，这才是图检索的领地。
该上的例子：问『生产库主备切换必须谁书面审批、这个人叫什么』。一篇写着须平台组组长审批，另一篇写着平台组组长是某人。第二篇跟查询一个词都不重合，我们在五种索引结构下测了一遍，它在 20 条候选池里一次都没出现过——换切法、加块头、父子回填全都无效。
不该上的例子：问『扩容要走哪个流程、最晚提前几个工作日提单』。同样跨两篇文档，但两篇都跟查询有明显字面重合，混合检索把它们分别排在第 2 名，一次检索就凑齐了。为它建图是拿几倍成本买一个已经解决的问题。
然后说代价，这一段决定了你像不像做过：建图不止一次抽取调用，实体要消歧、关系要去重、文档更新时受影响的子图要重算，还要多维护一套图存储和一套更新链路。
可预期的追问是『不上图检索还有什么办法』。答案是把多跳交给 Agentic 检索：让模型先查出中间实体，再拿这个实体发起第二次检索。它的一次性成本几乎为零，代价换成了每次查询的延迟与调用次数——先试这条，试不通再考虑建图。
Key points
The test is not 'is it multi-hop' but 'does the second document overlap the query at all' — only zero overlap earns a graph.
Justified: the approver question, where an intermediate entity is the only bridge and the second doc never enters the candidate pool.
Not justified: a multi-hop question whose documents both overlap the query — hybrid retrieval collects them in one pass.
Real graph cost is entity disambiguation, relation dedup, incremental subgraph recomputation and a whole extra store — not a single extraction call.
Try two-pass agentic retrieval first; build the graph only when that fails.
答题要点
判据不是『是不是多跳』，而是『第二篇文档跟查询有没有字面或语义重合』——没有重合才轮得到图检索。
该上：审批人那类问题，中间实体是唯一的桥，第二篇文档在候选池里一次都不出现。
不该上：两篇都跟查询有重合的多跳题，混合检索一次就能凑齐。
建图的真实成本是实体消歧、关系去重、增量重算和一套额外的图存储，不是一次抽取调用。
先试 Agentic 检索的两次查询，走不通再考虑建图。
You have built three different indexes over the same corpus. How do you decide which one a query goes to?同一份语料建了三套索引，检索时你怎么决定走哪一套？
Common in ChinaCommon overseasIntermediate#index-routing#evaluation#architecture
How to reason about it · think before answering
Whether this is an easy point or a lost one depends on whether you first ask 'do we actually need three?'. Jumping straight to routing accepts an unverified premise.
Step one is admitting the answer is usually 'none of them — use the default'. Across 30 documents we measured five index structures and every one landed at 93.8% recall, none beating the baseline. The only metric that moved was nDCG@10, which headers lifted from 0.6438 to 0.7218, while the two-stage summary index fell to 87.5%. Each structure patches one specific weakness; without that weakness it is pure overhead.
Step two is routing, and the criterion is not 'which index is more accurate' — that is an offline evaluation question, not something you know at request time. What you do have at request time is the shape of the question: detail-seeking, summarizing, or entity-chaining. Those map onto the chunk index, the tree-summary index and the graph index.
Implementation is a lightweight intent classifier — the same one from the previous day's intent routing, no need to invent another. Carry the decision as request metadata so you can replay it later.
Spell out the fallback: on a misclassification, fall back to the default index rather than fanning out across all three and fusing. Fan-out looks safe but multiplies latency and cost by the number of indexes, and the extra routes usually never make it into the context budget anyway.
Expect 'how do you know the classifier is right'. Log every routing decision and replay the golden set periodically: run each question through all three indexes and check whether the classifier picked the best-scoring one. It is a standing offline job that needs no human labelling.
分析过程 · 先想清楚再作答
这题是送分还是丢分，取决于你有没有先反问一句『真的需要三套吗』。上来就答路由策略的人，默认了一个没被验证的前提。
第一步是承认多数情况下答案是『都不走，走默认那套』。我们在 30 篇语料上把五种索引结构各测一遍，**召回率全部停在 93.8%，没有一种跑赢基线**；唯一动了的是 nDCG@10（块头把它从 0.6438 抬到 0.7218），而两段式的摘要索引还掉到了 87.5%。每种结构补的都是一个特定短板，你没有那个短板时它只带来成本。
第二步才是路由，而判据不是『哪套准』——那是离线评估该回答的问题，不是运行时能知道的。运行时能拿到的只有**问题的形状**：细节型（答案落在某一段）、概括型（要全库的一个概括）、多跳型（要跨实体串联）。按形状分流，正好对应块级索引、树状聚合索引、图索引。
实现上就是一个轻量意图分类器，跟前一天的意图路由是同一套东西，不必再造一个。分类结果作为元数据带进请求，方便事后拿评估集回看分错了多少。
兜底策略要说清楚：分类错了**回落到默认那一套**，不要并行全查一遍再融合。并行看着稳，实际上把延迟和成本按索引套数翻倍，而多出来的那两路大概率一条都进不了上下文预算。
可预期的追问是『怎么知道分类器分对了』。答案是把路由决策记进日志，定期拿标准答案集回放：对每个问题分别走三套索引，看分类器选的那套是不是指标最好的那套。这是一个能持续跑的离线作业，不需要人工标注。
Key points
First challenge the premise: all five index structures landed at the same 93.8% recall in our measurement, so an index without a matching weakness is pure cost.
At request time the usable signal is question shape — detail, summary, or entity-chaining — mapping to chunk, tree-summary and graph indexes.
Reuse the previous day's intent router for classification and record the routing decision as request metadata.
Fall back to the default index on misclassification instead of fanning out and fusing, which multiplies latency and cost.
Replay the golden set periodically to check whether the classifier picks the best-scoring index.
答题要点
先反问是否真需要三套：实测五种索引结构召回率全部持平在 93.8%，没有对应短板就是纯成本。
运行时的判据是问题的形状——细节型、概括型、多跳型，分别对应块级、树状摘要、图索引。
复用前一天的意图路由做分类，把路由决策记进请求元数据。
分类错了回落到默认索引，不要并行全查再融合——延迟和成本按套数翻倍。
用标准答案集定期回放，检验分类器选的那套是不是指标最好的那套。

D12 Agentic RAG: Turning Retrieval Into a Tool So the Model Decides Whether to Search, How Many Times, and Whether to Start Over

You are exposing retrieval to a model as a tool. How do you write the tool description, and what concrete failure modes appear when you write it badly?把检索包成一个工具交给模型，这个工具的描述该怎么写？写不好会导致哪些具体的错误行为？
Common in ChinaCommon overseasBasic#tool-design#agentic-rag#prompting
How to reason about it · think before answering
The discriminator is whether you can name concrete failure modes. Reciting 'the description should be clear' signals you have never shipped one.
Give the structure first: a usable description answers four things - what is and is not in the corpus, when the tool must be called, when it must not be called, and what shape the query string should take.
Attach a failure to each: no scope and the model treats it as a web search; no 'must call' and it answers policy questions from memory, convincingly; no 'must not call' and greetings or translations each burn a retrieval; no query shape and the model pastes the raw user sentence in, dragging interrogative words into the index.
The query-shape line is the cheapest win: one sentence saying 'keyword phrase, no question words' beats ten heuristics for query cleaning on the retrieval side.
Production angle: optional filter parameters such as department need an explicit 'only set this when you are certain'. Models like to fill optional fields, and a wrong filter hides the correct answer while the logs only show 'no results'.
Expected follow-up: how do you verify the description works? Run a negative suite - small talk, translation, arithmetic, follow-ups already answered in the conversation - and assert the tool was not called. That regression is automatable.
分析过程 · 先想清楚再作答
这题的题眼是「具体的错误行为」。只会背「描述要写清楚工具的用途」的，一句话就暴露了没上过线——面试官想听的是描述里少一句话，线上就多一类工单。
先给结构：一段合格的工具描述要回答四件事——库里有什么和没有什么、什么时候必须用、什么时候不要用、查询串写成什么形状。四条各对应一类事故，逐条挂钩着说最有说服力。
逐条挂钩：不写范围，模型拿它当搜索引擎，问天气也去查；不写「必须用」，涉及公司制度的问题被模型凭记忆编答案，而且编得非常像真的；不写「不要用」，闲聊和翻译都触发一次无谓检索，成本和延迟白涨；不写查询形状，模型把用户整句问话塞进 query，「叫什么名字」这种疑问词进了检索，纯噪声。
最后一条最值钱也最容易漏：在描述里加一句「写成关键词短语，不要带疑问词」，比在检索侧做十种查询清洗都管用——问题在源头，就在源头修。
补一个生产视角：参数里的过滤字段（比如部门）要写明「只在确定时才填」。模型倾向于把可选参数填满，填错一个部门就把正确答案挡在库外，而这种错误在日志里看不出来，表现是「检索没结果」。
可预期的追问是「怎么验证描述写对了」。答案是拿一批负样本跑：闲聊、翻译、算术、以及答案已在对话里的追问，看模型有没有多调一次工具；这类回归是能自动化的。
Key points
The description is a prompt for the model, not a code comment: scope, when to call, when not to call, query shape.
Missing scope turns it into a web search; missing 'must call' produces confident answers from memory.
Missing 'do not call' makes small talk trigger retrieval, paying cost and latency for nothing.
Stating 'keyword phrase, no question words' fixes query pollution at the source.
Optional filters need 'only set when certain' - a wrong filter silently hides the right answer.
Regression-test with a negative suite and assert the tool was not invoked.
答题要点
描述是写给模型看的提示词，不是注释；四段式：范围、什么时候用、什么时候不用、查询写成什么形状。
不写范围会被当成搜索引擎；不写「必须用」会导致凭记忆编答案。
不写「不要用」会让闲聊也触发检索，成本和延迟白涨。
写明查询要用关键词短语、不带疑问词，比在检索侧清洗查询更根本。
可选过滤参数要写「只在确定时才填」，填错会静默地把正确答案挡在外面。
用一批负样本（闲聊、翻译、算术）做回归，断言工具没有被调用。
Self-reflective retrieval rewrites the query and retries. How do you guarantee it terminates instead of spinning on the same query forever?自反思式检索会反复改写查询重试。你怎么保证它一定会停下来，而不是在同一个查询上原地打转？
Common in ChinaCommon overseasIntermediate#agentic-rag#self-reflection#reliability
How to reason about it · think before answering
This checks whether you have actually run such a loop. 'Set a max iteration count' is half an answer: it stops one failure mode and lets two others through.
Split runaway behaviour into three shapes and give each its own brake. Progress that never completes is capped by max rounds. Per-round budgets that pass individually but blow up in aggregate need a cumulative token budget - four rounds of 600 tokens each never trips a per-round check yet quadruples what reaches the model. Spinning in place needs duplicate-query detection.
Two implementation details prove you have written it: the duplicate check belongs before the retrieval call, otherwise you pay for a call to learn you are looping; and queries must be normalized to a set of terms, or 'failover approval' and 'approval failover' count as two distinct queries and the loop keeps turning.
Say what happens after it stops: stop reasons must be recorded as distinct categories - satisfied, gave up, hit round cap, hit token budget, duplicate query. Collapsing them into 'loop finished' hides how often the system simply surrendered.
An easy miss: installing a brake is not testing it. If the default token budget sits far above real usage it never fires, which is the same as not having one. Every brake needs a case that trips it.
Expected follow-up: what if the model says 'not enough' when it actually is? Make the assessment structured - which elements are covered, which are missing - and treat an empty missing list as sufficient, so the decision is auditable rather than a bare boolean.
分析过程 · 先想清楚再作答
这题在考「有没有真让循环跑过」。只答「设一个最大轮数」的能拿一半分，因为最大轮数只拦住了一类失控，剩下两类照样漏出去。
怎么拆：把失控分成三种形态，每种配一道闸。一是「每轮都在推进但永远推进不完」，用最大轮数拦；二是「每轮都不超标但累计爆掉」，用累计 token 预算拦——四轮各读 600 token 没有一轮超标，可送进模型的材料已经是单轮的四倍；三是「原地打转」，用重复查询检测拦。
重复查询检测有两个实现细节，答出来就说明真写过：一是要放在检索之前，否则要白花一次调用才发现自己在转圈；二是判重要对查询做归一化，只看词的集合，否则「主备切换审批」和「审批主备切换」会被当成两个不同的查询，圈照转不误。
还要说清停下来之后怎么办：停止原因必须分类记录，「查够了」「主动认输」「撞到轮数」「撞到预算」「原地打转」是五种不同的结局。把它们混成一个「循环结束」，你就永远看不见系统在多大比例的问题上其实是放弃了。
一个容易被忽略的点：闸门装了不等于验过。默认预算如果比实际用量高一大截，跑多少遍都踩不响它，等于没装。每一道闸都要构造一个用例把它踩响，这是验收的一部分。
可预期的追问是「模型自己说不够，但其实已经够了怎么办」。答案是自评要给结构化输出（覆盖了哪些要素、缺哪些），缺失项为空却仍判不够时按「够了」处理——让判断可审计，而不是信一个布尔值。
Key points
Three brakes, none optional: max rounds, cumulative token budget, duplicate-query detection.
The cumulative budget catches rounds that each pass but blow up together - the round cap cannot see that.
Check for duplicates before retrieving, and normalize the query to a term set before comparing.
Record stop reasons as distinct categories rather than one 'finished' bucket.
Every brake needs a case that actually trips it; an untested brake is no brake.
Have the assessor emit covered and missing elements so 'not enough' is auditable.
答题要点
三道闸缺一不可：最大轮数、累计 token 预算、重复查询检测。
累计预算拦的是「每轮都不超但加起来爆掉」，轮数闸看不见这件事。
重复查询检测要放在检索之前，且查询要归一化成词的集合再判重。
停止原因分类记录：查够了、主动认输、撞轮数、撞预算、原地打转是五种结局。
每一道闸都要构造用例踩响，装了没验过等于没装。
自评输出结构化的覆盖与缺失项，让「不够」这个判断可审计。
In multi-hop retrieval a wrong first hop poisons every hop after it. How would you design for that?多跳检索里第一跳查错了，后面全跟着错。你会怎么设计容错？
Common in ChinaCommon overseasDeep dive#multi-hop#error-propagation#agentic-rag
How to reason about it · think before answering
This is about error propagation. 'Add a retry' is not enough - retries help when one path fails, but the multi-hop problem is walking confidently down the wrong path.
Separate two failure kinds first, because the fixes are opposite. Either the answer document never entered the candidate pool - no amount of loosening helps, only a new query term does, which is the multi-hop path - or it was retrieved and then dropped by your own admission threshold, where extra hops are useless and only relaxing the gate recovers it. Coverage plus the answer slot tells them apart: low coverage means wrong direction (broaden), high coverage with an empty slot means halfway there (hop).
Then give the mechanism. Do not let the model freestyle the next query: pick a bridge phrase from the sentence that best matches the question - a concrete noun the question never mentioned that also appears in another document. 'Not in the question' makes it new information; 'appears elsewhere' guarantees there is somewhere to hop to.
Fault tolerance has three layers: keep the earlier hop's material, so a bad second hop does not destroy the evidence you already had; trace each hop separately so you can locate where it went wrong; and surrender explicitly when there is no lead left, handing 'insufficient evidence' to the generation-side refusal.
The overlooked trap is worth points: the hop succeeds but the metric does not move. The second hop really did retrieve the target document, yet if you merge both hops' candidates and pack by score, the first hop's higher lexical overlap fills the budget and the target never enters the context. Allocate the context budget round-robin across hops - that is where multi-hop gains are actually realized.
Expected follow-up: how do you know the first hop was wrong? From the structured self-assessment, not from the final answer. By the time the answer is wrong the chain is three hops deep and much more expensive to debug.
分析过程 · 先想清楚再作答
这题考的是错误传播意识。只答「加个重试」是不够的——重试只在「同一条路走不通」时有用，而多跳的问题是走上了错误的路还越走越远。
先把两类失败分开，这是整题的骨架：一类是根本没捞到（答案文档在候选池里一次都没出现，放宽门槛毫无用处，只能靠新的查询词重查，也就是多跳），一类是捞到了却被自己的过滤器扔了（排在第二名但没过准入门槛，这一类跳多少跳都没用，只能降级放宽门槛重判）。判据是要素覆盖率加答案槽位：覆盖率低是方向错了走放宽，覆盖率高但槽位空是只查到半路走多跳。判错类型，容错就完全用反了。
然后给具体机制。判断下一跳查什么，不能凭模型自由发挥，要有可解释的判据：从最贴题的那一句里挑出问题没提过、且在别的文档里也出现过的具体名词当作桥接短语。「问题没提过」保证它是新信息，「别的文档里也有」保证真的有下一跳可跳——只在这一篇里出现的短语，查了只会把同一篇再捞回来。
结论层面，容错有三层：不要丢掉上一跳的材料（第二跳查错了，第一跳的证据还在）；每一跳独立记录轨迹，事后能定位是哪一跳歪的；追不动时主动认输，把「材料不足」交给生成侧的拒答，而不是硬凑一个答案。
有一个非常容易被忽略的坑，说出来会加分：跳成功了，命中却没变。第二跳确实把目标文档检索回来了，但如果把两跳的候选混在一起按名次装上下文，第一跳的材料字面重合度更高，会把预算占满，目标文档根本挤不进去。上下文预算必须按跳轮转分配——多跳的收益是在这一步兑现的，不是在检索那一步。
可预期的追问是「怎么知道第一跳错了」。答案是靠自评的结构化输出，而不是靠最终答案对不对；等到答案错了再回头找，链路已经断了三跳，定位成本高得多。
Key points
Classify first: never retrieved needs a new query term (a hop); retrieved-then-filtered needs a relaxed gate. The fixes are opposite.
Derive the next query from a bridge phrase - a concrete noun absent from the question that also appears in another document.
Keep the previous hop's material so a failed hop does not discard existing evidence.
Trace every hop separately so you can pinpoint which one drifted.
Allocate context budget round-robin across hops, or a successful hop still fails to change the metric.
Surrender explicitly when no lead remains and hand it to the generation-side refusal.
答题要点
先分类：根本没捞到只能靠多跳换查询词，捞到了被门槛扔了只能靠降级放宽，两者修法相反。
下一跳的查询用桥接短语：问题没提过、且别的文档里也出现过的具体名词。
保留上一跳的材料，第二跳失败时第一跳的证据仍在。
每一跳独立记轨迹，能定位是哪一跳歪的。
上下文预算按跳轮转分配，否则跳成功了命中也不会变。
追不动时主动认输，把材料不足交给生成侧拒答，不硬凑答案。
When would you refuse to make a RAG system agentic, and what data would you use to convince your team?什么情况下你会拒绝把一个 RAG 系统做成 Agentic 的？拿什么数据说服你的团队？
Common in ChinaCommon overseasIntermediate#agentic-rag#cost#engineering-judgement
How to reason about it · think before answering
This tests engineering judgement and whether you can do arithmetic. Anyone who says 'agentic is more advanced so we should ship it' is out. The interviewer wants you to name the cost and draw the boundary with numbers.
Decompose it: identify which question types actually benefit, then check how much of your traffic they represent. Agentic gains concentrate in multi-hop questions and retrieval retries; single-document questions are answered by one lookup and every extra round is waste.
So the criterion is the evaluation set, not intuition. On a 20-item set we measured multi-hop recall going from 75% to 100% while overall answerable recall moved only from 93.8% to 100%, at the cost of average retrieval calls going from 1 to 1.75 plus the same number of assessment calls - you pay for 100% of traffic so that 5% of it improves.
Three clear refusals: latency-sensitive surfaces, where each round adds a retrieval plus a model round trip and roughly doubles time to first token; fixed question patterns, where nine in ten questions are single-document and the gain is near zero; and tight cost budgets, where a real model is less disciplined than an offline stand-in and the variance, not the mean, is what breaks your capacity plan.
Finish with the alternative: route. Use one cheap check to decide whether a question looks multi-hop, and only then enter the loop. Nine tenths take a single retrieval, one tenth loops, and the economics change completely. Looping is a capability, not a default.
Expected follow-up: how do you know which questions look multi-hop? Mine the eval set and production logs for patterns - two facts requested in one sentence, or a question about the person behind a role - start with rules, and reach for a small classifier only when rules stop working.
分析过程 · 先想清楚再作答
这题在考工程判断力，也在考你会不会算账。凡是答「Agentic 更先进所以要上」的，直接出局；面试官想听的是你能主动说出它的代价，并且用数字划出适用边界。
怎么拆：先承认收益来自哪一类问题，再看这类问题在你的流量里占多大比例。Agentic 的收益几乎全部集中在多跳和检索失败重试上，单文档可答的问题一次检索就够了，多查一轮纯属浪费。
所以判据不是感觉，是评估集：跑一遍，看 multi 那一档占多少题、涨了多少个点，再对照总调用次数涨了多少倍。在一份 20 题的集合上，我们量到的是多跳召回从 75% 涨到 100%，可答题整体只从 93.8% 涨到 100%，代价是平均检索调用从 1 次涨到 1.75 次、外加同样次数的自评调用——为 100% 的问题付钱，只有 5% 的问题拿到好处。
三类明确不上：延迟敏感（每多一轮就是一次检索加一次模型往返，首字延迟拉长一到两倍）；问题模式固定（九成是单文档可答，收益接近零）；成本吃紧（真实模型不像离线替身那样老实，成本方差比均值更难受，按均值做的容量规划会在长尾上被打穿）。
给出替代方案才算完整：分流。先用一次便宜的判断看这一问像不像多跳，像才进循环，不像走固定流程。九成走一次检索、一成走循环，账完全不一样。这也说明循环是一种能力，不是默认值。
可预期的追问是「那你怎么知道哪些问题像多跳」。答案是从评估集和线上日志里找模式（问句里同时问了两个事实、问的是某个角色背后的人），先用规则跑，跑不动再上小模型分类——顺序不要反。
Key points
Gains concentrate in multi-hop and retry cases; single-document questions gain almost nothing.
Settle it with the evaluation set: multi-hop delta against the multiplier on total calls.
One measured set: multi-hop recall 75% to 100%, overall 93.8% to 100%, retrieval calls 1 to 1.75 plus the same number of assessment calls.
Refuse when latency-sensitive, when question patterns are fixed, or when cost is tight - variance hurts more than the mean.
Route instead: a cheap check up front, and only multi-hop-looking questions enter the loop.
Looping is a capability, not a default.
答题要点
收益集中在多跳与检索失败重试，单文档可答的问题上收益接近零。
用评估集算账：multi 档涨了多少点，对照总调用次数涨了多少倍。
实测过的一组数字：多跳召回 75% 到 100%，整体 93.8% 到 100%，检索调用 1 次到 1.75 次外加等量自评调用。
三类不上：延迟敏感、问题模式固定、成本吃紧（方差比均值更难受）。
替代方案是分流：便宜的判断先过滤，像多跳才进循环。
循环是一种能力，不是默认值。

D13 Going to Production: Incremental Sync and Deduplication, Permission-Based Filtering, Cache Layering, Tracing, and the Cost-Latency Ledger

After a document changes, how do you recompute only the affected chunks? And how do you guarantee a deleted document really disappears from the index?文档更新之后，你怎么做到只重算受影响的块？被删掉的文档又怎么保证一定从索引里消失？
Common in ChinaCommon overseasIntermediate#incremental-sync#content-hash#index-maintenance
How to reason about it · think before answering
There are two halves here and the second one separates candidates. Almost everyone can say 'hash it and compare'; the score comes from bringing up deletion yourself, because it is the one asymmetric case in the whole mechanism.
Give the skeleton first: a three-way reconciliation between the full set from the source and the full set in the index. In source but not indexed is an add; in both but with different content hashes is a modify; indexed but absent from the source is a delete. A modify must replace the document wholesale, deleting old chunks before writing new ones, otherwise a shortened document leaves a tail behind in the index.
Then the fingerprint itself, which is where points are won: sha256 truncated, but normalize line endings and trim before hashing. The same file uploaded from Windows and from macOS differs byte-wise but not in content; skip normalization and every re-upload counts as a change, which is a full rebuild in disguise. It never raises an error, it only shows up on the bill.
The key insight in the second half: a deletion is not an event, it is an absence. Change feeds tell you what changed; nobody ever sends 'I no longer exist'. So deletion detection has to run in the opposite direction — walk the index and find ids the source no longer has. A synchronizer that only listens to change events will wait forever.
At the storage layer, cascade the foreign keys across documents, chunks and embeddings so deleting a document is a single statement and the database does the rest. Hand-written three-step deletes eventually miss one, and the one they miss is a ghost in the index. Close with a verifiable invariant: chunk count must equal embedding count, and a mismatch means orphans.
Expected follow-up: what if the source system itself is unreliable and a pull comes back incomplete? Make pull completeness a precondition for deletion: on a partial pull, apply adds and modifies only, or one failed fetch wipes half your index. Also soft-delete with a retention window so a mistake is recoverable.
分析过程 · 先想清楚再作答
这题有两半，区分度全在后半。前半几乎人人答得出「算个哈希比一比」，能不能拿到分取决于你有没有主动讲删除——那是同一套机制里唯一不对称的一种变更。
先给增量的骨架：拿来源的全集和索引的全集做三向对账。来源有、索引没有是新增；两边都有但内容指纹不同是修改；索引有、来源没有是删除。修改的处理是整篇替换，先删旧块再写新块，不能只追加——不然改短了的文档会在索引里留下一截尾巴。
接着讲指纹本身，这是给分点：sha256 取前若干位，但**算之前必须先做换行归一化再去首尾空白**。同一份文件从 Windows 传一次、从 Mac 传一次，字节不同内容相同，不归一化就每次都判成变了，等于天天在做全量重建。这个 bug 不报错，只体现在账单上。
然后是删除这一半的关键判断：**删除不是一个事件，是一个缺席**。文件变动类的通知只告诉你哪些东西变了，永远不会有人发一条「我不存在了」。所以删除检测必须反着来——遍历索引，找出来源里已经没有的 id。只监听变更事件的同步器永远等不到这条消息。
落到存储上：文档、块、向量三张表用外键级联删除，删文档只写一条语句，剩下的交给数据库。手写三条删除的版本迟早会漏掉一条，而漏掉的那条就是索引里的幽灵。收尾时报一个可验证的指标：块数与向量数必须相等，不等就说明有孤儿。
可预期的追问：来源系统本身就不可靠、拉不全怎么办？那就把「本次拉取是否完整」当成删除检测的前置条件——拉取不完整时只做新增和修改，不做删除，否则一次拉取失败会把半个索引清空。另外给删除加软删标记和保留期，误删还能回滚。
Key points
Three-way reconciliation covering adds, modifies and deletes; a modify replaces the whole document, old chunks first.
Normalize line endings and trim before hashing, or cross-platform re-uploads look like edits and you are doing a full rebuild every night.
Deletion is an absence, not an event: walk the index for ids the source no longer has instead of waiting on a change feed.
Cascade deletes from documents to chunks to embeddings so one statement suffices; assert chunk count equals embedding count to catch orphans.
On an incomplete pull, apply adds and modifies only, and soft-delete with a retention window so mistakes are reversible.
答题要点
三向对账：新增、修改、删除，缺一不可；修改是整篇替换，先删旧块再写新块。
内容指纹算之前必须先做换行归一化再 trim，否则跨系统重传会被误判为修改，等于天天全量重建。
删除是缺席不是事件，必须反过来遍历索引找出来源里已消失的 id，不能只监听变更通知。
文档、块、向量用外键级联删除，删文档只写一条语句；用「块数等于向量数」当可验证的收尾指标。
来源拉取不完整时只做新增与修改、跳过删除，并给删除加软删与保留期以便回滚。
Why can't access control be applied at the generation step? What exactly leaks if you put it there?为什么权限过滤不能放在生成阶段做？放在那里会泄露什么？
Common in ChinaCommon overseasDeep dive#access-control#filter-pushdown#multi-tenancy
How to reason about it · think before answering
This checks whether you think about RAG as a system. 'Because it's insecure' scores nothing; the interviewer wants what specifically leaks, and what else goes wrong besides the leak.
Anchor the position with an image: the archivist spreads every file on the table, you pick nine, and only then does he pull three back saying you may not read those. You have already seen the titles. Filtering at generation time is that gesture.
Then split the consequences, and note the second one is what shows engineering experience. First, exposure: the unauthorized documents were retrieved, ranked, read into process memory, and almost certainly written to retrieval logs and traces, even if none of their text reaches the answer. Second, dilution: you take the top 8, three are off-limits, the user gets five, and the legitimate results ranked ninth and tenth never get promoted. The user experiences 'it can't find anything' while your logs show a perfectly normal retrieval.
State the fix: put the permission predicate in the same query as the ordering and the LIMIT, so the database prunes rows before ranking and unauthorized vectors are never compared. Cover both shapes: row-level filtering is one index plus a predicate; index isolation is a separate index per boundary.
Give the selection criterion: the number and stability of the isolation boundaries. A handful of departments that rarely change makes isolation worthwhile; tens of thousands of per-user private document sets leave you with row-level filtering, because that many indexes is unmanageable. Add the shared-index side effect: a large tenant degrades everyone else's retrieval quality because candidate slots are shared.
Expected follow-up: what about caching? It is the same bug's second crime scene. The answer cache key must include the permission scope, or one user's answer will be served to another, and that leak leaves no trace in the retrieval log at all.
分析过程 · 先想清楚再作答
这题在考你有没有真的把 RAG 当系统看。答成「因为不安全」拿不到分，面试官要的是「具体泄露了什么」和「除了泄露还有什么后果」两件事。
先用一个画面把位置说清楚：档案管理员先把全部档案摊在桌上让你挑，你挑完他再抽走三份说这些不能看——你已经看见标题了。在生成阶段过滤就是这个动作。
然后拆后果，两条，第二条更能显出做过工程：一是**泄露面**，越权文档已经进过检索、参与过排序、被进程读进过内存、大概率写进了检索日志和链路追踪，哪怕最终答案里没有它的内容；二是**结果被稀释**，取前 8 条里有 3 条不该看，筛掉只剩 5 条，而本该补位的第 9、10 名合法结果永远没机会上来——用户体感是「查不到」，你的日志里却是一次正常检索。
给正确做法：把权限谓词和排序、LIMIT 写进同一条查询，数据库先裁行再排序取前 k，越权的行一次都没被比较过。两种落法要都讲：行级过滤是一份索引加一个谓词，索引隔离是按边界各建各的索引。
选型判据要给出来：看隔离边界的数量和稳定性。部门这种个位数且几乎不变的边界，隔离划算；几万个用户各自的私有文档就只能行级过滤，否则运维扛不住。补一句共用索引的副作用——数据量大的租户会拖慢别人的检索质量，因为候选名额是共享的。
可预期的追问：缓存怎么办？这是同一个问题的第二现场——答案缓存的 key 里必须带上权限范围，否则一个用户的答案会被另一个用户命中，而且这条泄露路径连检索日志都不会留下痕迹。
Key points
Filtering at generation time means unauthorized documents were already retrieved, ranked, held in memory and written to logs and traces; the exposure is far wider than 'did the text reach the answer'.
The second consequence is dilution: filtered-out slots are not backfilled, so users see 'nothing found' while the log shows a normal retrieval.
The fix is to put the permission predicate in the same statement as ordering and LIMIT so the database prunes before ranking.
Choose between row-level filtering and index isolation by the count and stability of the boundaries; a shared index lets a large tenant crowd out a small one's candidate slots.
Caching is the same bug's second crime scene: the answer cache key must carry the permission scope or answers leak across users without a trace.
答题要点
在生成阶段过滤时，越权文档已经被检索、排序、读进内存并写进日志与追踪，泄露面比「答案里有没有」大得多。
第二个后果是结果被稀释：筛掉之后名额空着不补，用户体感是查不到，日志里却是一次正常检索。
正确做法是把权限谓词和排序、LIMIT 写进同一条查询，让数据库先裁行再排序取前 k。
行级过滤与索引隔离的选型判据是隔离边界的数量与稳定性；共用索引时大租户会挤占小租户的候选名额。
缓存是同一个漏洞的第二现场：答案缓存的 key 必须包含权限范围，否则会跨用户串答案且不留痕迹。
What can be cached in a RAG system, and what are the invalidation conditions for each?RAG 系统里有哪些东西可以缓存？各自的失效条件是什么？
Common in ChinaCommon overseasIntermediate#caching#invalidation#cost-optimization
How to reason about it · think before answering
This looks like a giveaway and is actually a filter. 'Cache the question and answer' earns a third of the credit; the interviewer is waiting for the layering and the per-layer invalidation rules.
Lead with a transferable rule: 'when must this be invalidated' is the same question as 'is that thing part of the key'. Leave something out of the key and changes to it will never invalidate the entry. With that rule the three layers derive themselves.
Then go layer by layer. The answer layer maps a question to a final answer; its key needs the question, the permission scope, the index version, and the model plus prompt version. The retrieval layer maps a query to a hit list; its key needs the question, scope, topK, index version and embedding backend, but not the generation model. The embedding layer maps text to a vector; its key is just the text and the backend.
Emphasize the counterintuitive part of the embedding layer: it is content-addressed, so the index version must not be in its key. Put it there and a single sync invalidates tens of thousands of vectors, which is exactly the full rebuild you added caching to avoid. This is the one layer that can live a long time, even on disk.
Offer a concrete invalidation mechanism: version numbers rather than targeted deletion. Bump an index version whenever a sync actually changes something and old keys simply stop being computed. Targeted deletion would require enumerating which questions a change affected, and that list cannot be produced.
Expected follow-up: can you give a real 'should have expired but didn't' case? Yes: an answer cache keyed only on the question. A document's limit changes from 200 MB to 500 MB, the index is updated, and the same question still returns 200 MB. Nothing errors; the log shows a clean cache hit. The same key also serves one department's answer to a user from another.
分析过程 · 先想清楚再作答
这题看起来是送分题，实际是筛人题。答成「把问答结果缓存起来」只拿到三分之一，面试官等着听的是「分几层」和「各自什么时候失效」。
先给一条能迁移到别的题上的判断依据：**「什么时候必须失效」这个问题，等价于「key 里有没有把那样东西算进去」。** key 少放一样，那样东西变了缓存就不会失效。有了这条，三层的答案自己就长出来了。
然后逐层给：答案层缓存问题到最终答案，key 要有问题、权限范围、索引版本、模型与提示词版本；检索层缓存检索式到命中块列表，key 要有问题、权限范围、topK、索引版本、向量后端，但不需要模型；向量层缓存文本到向量，key 只有文本和向量后端。
重点讲向量层的反直觉之处：它是**内容寻址**的，文本没变、模型没变，向量就不会变，所以**不能把索引版本放进它的 key**。放进去的话一次同步就作废几万条向量，正好绕回全量重建——你加缓存想省的那笔钱又花回去了。这一层可以放很久甚至持久化。
给一个具体的失效手法：用**索引版本号**而不是精确删除。同步只要真的改动了索引就把版本号加一，旧 key 再也算不出来，自然没人读得到。精确删除要求你能列出「这次改动影响了哪些问题」，而那是列不出来的。
可预期的追问：能举一个「该失效却没失效」的真实例子吗？答：答案缓存的 key 只放了问题本身，文档里的上限从 200 MB 改成 500 MB、索引已经更新，再问同一个问题仍然返回 200 MB。它不报错，日志上是一次漂亮的缓存命中；同一个 key 还会让另一个部门的用户直接命中别人的答案。
Key points
Three layers — answer, retrieval, embedding — with lifetimes orders of magnitude apart; treating them as one thing is the mistake.
The rule is that 'when must it expire' equals 'is it in the key'; anything left out of the key can never invalidate the entry.
The answer key carries question, permission scope, index version, model and prompt version; the retrieval key drops the model and adds topK and the embedding backend.
The embedding layer is content-addressed and keyed only on text plus backend; adding an index version turns every sync back into a full rebuild.
Version-based invalidation beats targeted deletion because you cannot enumerate which questions a given change affected.
答题要点
分三层：答案、检索、向量，三者的寿命差着数量级，不能当成一件事。
判断依据是「什么时候必须失效」等价于「key 里有没有算进那样东西」，key 少一样就永远失效不了。
答案层 key 要有问题、权限范围、索引版本、模型与提示词版本；检索层去掉模型、加上 topK 与向量后端。
向量层是内容寻址的，key 只有文本与后端；把索引版本放进去会让每次同步都退化成全量重建。
用索引版本号做失效比精确删除可靠，因为「这次改动影响了哪些问题」根本列不出来。
You need to switch embedding models. How do you migrate a live system without downtime and without losing recall?要换一个 embedding 模型，线上系统怎么迁移才能不停机也不掉召回？
Common in ChinaCommon overseasDeep dive#embedding-migration#zero-downtime#rollout
How to reason about it · think before answering
The crux is why you cannot swap in place. Jumping straight to the steps without establishing that reads like reciting a runbook.
Set up the premise: vectors from different models are not comparable. Dimensions may differ, and even at equal dimensions the coordinate spaces are unrelated, so encoding the query with the new model and comparing against documents encoded with the old one yields noise. Switching models therefore means re-embedding the entire corpus.
Then the four steps: add a nullable second vector column; backfill it with a background job while the old column is untouched and still serves live traffic; canary a slice of traffic onto the new column while running the golden set against both columns to compare recall and faithfulness; cut over fully once the numbers hold, and drop the old column only after a week or two of observation.
Name the payoff explicitly, because this is where the points are: the value of the whole procedure is the rollback cost. Cutover is a config change naming which column to read, so reverting takes a second rather than re-running an eight-hour rebuild. A migration plan with no rollback path is not a plan.
Add two engineering details: build the approximate-nearest-neighbour index on the new column after the backfill, not during it, since concurrent building is slow and prone to locking; and make the backfill resumable and rate-limited, or it will exhaust the embedding API quota and drag live queries down with it.
Expected follow-up: how do you prove the new model is actually better? Not from an offline metric alone — run an A/B on the same golden set with identical retrieval parameters and report four numbers: recall, faithfulness, latency and cost. A conclusion resting on the first number only does not hold. Note also that switching models is the one moment when the embedding cache genuinely must be invalidated.
分析过程 · 先想清楚再作答
这题的题眼是「为什么不能就地换」。没有先说清这一点就直接讲步骤，会显得是在背流程。
先给前提：不同模型的向量之间**没有可比性**。维度可能不同，即使维度相同坐标系也完全不是一回事，用新模型编码问题去和旧模型编码的文档比距离，算出来的相似度是纯噪声。所以「换模型」实质上等于「把整个知识库重新向量化一遍」。
然后给四步：加一列新向量、允许为空；后台任务慢慢回填新列，旧列一个字节不动，线上仍走旧列；小流量灰度到新列，同时用标准答案集在两列上各跑一遍比召回率与忠实度；数字站得住再全量切换，旧列观察一两周后才删。
把这套流程的价值点破，这是给分点：**它的价值全在回滚成本上**。切换只是改一个配置项「走哪一列」，出问题时切回去是一秒钟的事，而不是重跑一遍八小时的重建任务。凡是拿不出回滚路径的迁移方案都不算方案。
补两个工程细节：新列的近似最近邻索引要在回填完之后再建，边写边建又慢又容易锁表；回填要能断点续传并限速，否则会把 embedding 接口的配额打满，把线上查询一起拖垮。
可预期的追问：怎么证明新模型确实更好？答：不能只看离线指标涨没涨，要在同一份标准答案集、同一套检索参数下跑 A/B，报召回率、忠实度、延迟、花费四笔账；只报第一笔的结论不成立。另外注意换模型会让缓存里的向量全部作废，那是这次迁移唯一该作废向量缓存的时刻。
Key points
Vectors from different models are not comparable, so a model switch is equivalent to re-embedding the entire corpus.
Four steps: add a nullable second vector column, backfill in the background, canary with the golden set scored on both columns, then cut over once the numbers hold.
The whole value lies in rollback cost: cutover is a config change, so reverting takes a second instead of another full rebuild.
Build the ANN index on the new column after the backfill; make the backfill resumable and rate-limited so it does not exhaust the embedding quota and stall live queries.
Validate with an A/B on one golden set reporting recall, faithfulness, latency and cost; a model switch is also the only time the embedding cache truly must be invalidated.
答题要点
不同模型的向量之间没有可比性，所以换模型等价于把整个知识库重新向量化一遍。
四步：加一列可空的新向量、后台回填、小流量灰度并用标准答案集在两列上对比、数字站得住再全量切换。
这套流程的价值全在回滚成本上：切换是改一个配置项，回滚是一秒钟的事而不是重跑一次重建。
新列的近似最近邻索引在回填完成后再建；回填要可断点续传并限速，别把接口配额打满拖垮线上查询。
验证要在同一份标准答案集上跑 A/B，同时报召回率、忠实度、延迟与花费四笔账；换模型也是唯一该作废向量缓存的时刻。

D14 Capstone Project and Retrospective: A Multi-Tenant Enterprise Knowledge-Base Q&A, a RAG Decision Map, and an Interview Deep Dive

You are handed a knowledge base of five million documents that must answer in about a second, with accuracy as the top priority. How would you design it?给你一个五百万文档、要求秒级响应、准确率优先的知识库场景，你会怎么设计这套系统？
Common in ChinaCommon overseasDeep dive#system-design#scaling#latency-budget
How to reason about it · think before answering
The real subject here is not which technologies you know, it is whether you have a repeatable way to derive a configuration from constraints. Opening with an architecture diagram reads as a memorized answer; the way to score is to turn each constraint into a number first, then let every choice be forced by one of those numbers.
Quantify the three constraints. Five million documents at roughly four or five chunks each is over twenty million chunks; at 1536 float dimensions that is hundreds of gigabytes, so the index does not fit in one machine's memory — that alone settles storage. A one-second budget to first token, with generation typically eating seven or eight hundred milliseconds, leaves only two or three hundred for retrieval. Accuracy first means you may trade latency and money for metrics, but only within that remaining budget.
Now derive each knob from one of those numbers: a dedicated vector store or partitioning, plus half precision (its recall loss usually sits inside run-to-run noise while the index shrinks by about forty percent — essentially free); keep both keyword and vector routes with reciprocal rank fusion, because exact matches on document ids, error codes and names are a permanent blind spot for embeddings; rerank only the top twenty after fusion, since it buys ranking quality at the cost of one synchronous round trip, and a one-second budget affords exactly one.
Then state two things you deliberately do not build, which is the part that reads as field experience. Agentic retrieval is not the default path: its gains concentrate on multi-hop questions while its cost is spread over every question, and it blows a one-second budget outright — the right move is a cheap classifier that routes only the multi-hop minority into the loop. Contextual chunk headers and similar tricks also wait, because they dilute the keyword route while helping the vector route; the directions are opposite, so measure on your own embeddings before committing.
Accuracy first has to become something you can sign off on. That means a golden set of at least a hundred questions with multi-hop and unanswerable each above ten percent, recall and ranking quality read separately, abstention rate on unanswerable questions as its own column, and citations verified by code rather than trusted from the model. Reporting the ugliest column alongside the headline number is far more credible than reporting a single score.
Expected follow-up: how do you build the first index over five million documents? It is a one-off large expense, so batch it, make it resumable, and put content-hash incremental sync in from day one, or every config change means buying the whole corpus again. Push further and you get to rollout: dual-write the new embeddings into a second column, evaluate both columns on the same golden set, then shift traffic, so rollback is a config flip rather than an eight-hour rebuild.
分析过程 · 先想清楚再作答
这题的题眼不在「你会用什么技术」，而在「你有没有一套从约束推配置的方法」。开口就报架构图和技术栈的答案会被判成背方案；拿到分的答法是先把约束翻译成数字，再让每个选择被某个数字逼出来。
先把三个约束量化：五百万文档按一篇四五块估，是两千多万块，单精度 1536 维就是上百 GB，**索引塞不进单机内存**，这一条直接决定了存储选型；秒级响应意味着从收到问题到第一个字的预算大约一秒，而生成本身通常就吃掉七八百毫秒，检索侧只剩两三百毫秒；准确率优先意味着可以拿延迟和钱换指标，但只能换到那两三百毫秒为止。
然后逐项落地，每一项都挂在上面某个数字上：存储上专用向量库或分区加半精度量化（半精度的召回损失通常落在重跑噪声里，索引却小四成，这是白捡的）；检索保留关键词与向量两路加倒数排名融合，因为精确匹配的文档号、错误码、人名是向量的固定盲区；重排只作用于融合后的前二十条——它买的是排序质量，一次同步往返，秒级预算里放得下一次，放不下两次。
接着讲两个「不上」的决定，这一段比上面更能显出做过工程：**Agentic 检索不作为默认路径**，它的收益集中在多跳题上而代价摊给全部问题，秒级预算下更是直接超支——正确做法是先用一次便宜的分类把多跳分流出来，只让那一小部分进循环；**上下文块头之类的手法先不上**，因为它对关键词一路是稀释、对向量一路才是补位，方向相反，得在自己的真实 embedding 上测过再说。
准确率优先必须落成可验收的东西，否则是空话：一份不少于一百题的标准答案集（其中多跳与无答案各占一成以上）、召回率与排序质量分开看、无答案题的拒答率单独一栏、引用由代码回查而不是靠提示词自觉。**报数字时把最难看的那一栏也报出来**，比只报总分可信得多。
可预期的追问：五百万文档怎么建第一版索引？答案是这笔钱是一次性大额支出，要按批做、可断点续跑，并且从第一天就上基于内容指纹的增量同步——否则每次改配置都等于把整个知识库重买一遍。再追问就谈灰度：新旧两套向量双写在两列上，用同一份标准答案集在两列上各跑一遍再切流量，回滚只是改一个配置项。
Key points
Translate constraints into numbers first: twenty million chunks means the index will not fit one machine, and a one-second budget leaves retrieval two to three hundred milliseconds.
Dedicated store or partitions plus half precision; keep keyword and vector routes with RRF, and rerank only the top twenty after fusion.
Name the two things you will not ship: agentic only for a routed multi-hop minority, and chunk headers only after measuring on your own embeddings.
Turn accuracy-first into a hundred-plus question golden set, abstention rate as its own column, and code-verified citations.
First index build is a one-off large expense: batch it, make it resumable, add incremental sync on day one, and dual-write columns for model swaps.
答题要点
先把约束翻译成数字：两千多万块决定索引塞不进单机内存，一秒预算里检索侧只剩两三百毫秒。
存储用专用库或分区加半精度；检索保留关键词与向量两路加倒数排名融合，重排只作用于前二十条。
明确说出「不上」的两项：Agentic 只对分流出来的多跳开，块头这类方向相反的手法先测再说。
准确率优先要落成一百题以上的标准答案集、拒答率单独一栏、引用由代码回查。
第一版建索引是一次性大额支出：分批可续跑，并从第一天就上增量同步；换模型走双写切列。
Looking back at the RAG project you built, which decision would you change now, and why?你做过的这个 RAG 项目里，哪个决定你现在会改？为什么？
Common in ChinaCommon overseasDeep dive#retrospective#evidence#chunking
How to reason about it · think before answering
This looks like a soft question but it separates people sharply. Saying 'nothing yet' admits you never ran a retrospective; a long list of self-criticism reads as poor judgement. What the interviewer is listening for is whether you can chain four things together: the decision, the evidence you had then, the evidence you got later, and your current call.
How to pick: choose a decision that was justified at the time and later overturned by data, not one you always knew was a shortcut. The first proves you measure; the second only proves you were behind schedule. So the answer has a fixed four-part shape — what you chose, on what basis, what you measured later, and what you now believe.
This course supplies a ready example. One day measured that prepending a heading-path header to every chunk left hit rate unchanged, grew index tokens by about ten percent, and pushed the answer document's mean rank from 2.88 to 3.25 — hence 'headers hurt keyword retrieval'. A later day re-ran the same comparison under structure-aware chunking and the rank regression did not reproduce. The reason was the chunker: with fixed-length cuts, chunk boundaries do not line up with section boundaries, so the header injects heading terms that do not belong to that chunk; with structure-aware cuts, each chunk already sits inside one section and the header largely restates what is already there. The correct statement is therefore not 'headers hurt' but 'headers hurt when chunk boundaries are misaligned with document structure'.
The value of the chain is that it demonstrates a reusable habit: attach the premises to every conclusion. Change a premise and you owe a re-run; you may not pair new settings with an old conclusion. The same reasoning yields a second example: an earlier claim that 'the vector route is clearly a net gain' collapsed once the two routes were counted separately before fusion — the vector-only candidates were a small share and contained the answer document zero times, so the improvement was never semantic at all.
Expected follow-up: how will you avoid this class of error in future? Give two concrete practices. Write the bound premises next to every number — corpus, question set, budget, chunker. And before publishing any conclusion, ask whether it was measured by this experiment or forced by the structure of the implementation; the first needs its boundaries stated, only the second can be asserted flatly.
分析过程 · 先想清楚再作答
这题看着是软性问题，其实区分度极高。答「暂时没有」等于承认没做过复盘；答成一长串自我批评又会显得没有判断力。面试官真正在听的是：你能不能把一个决定、它当时的依据、后来的证据、以及新的判断，四样东西串成一条链子说清楚。
怎么拆：挑一个**当时有理由、后来被数据推翻**的决定，而不是一个「当时就知道是凑合」的决定。前者证明你有量化的习惯，后者只证明你赶过工期。所以答案的骨架固定是四段——当时选了什么、依据是什么、后来量到了什么、现在的判断是什么。
本课里有一个现成的样本：某一天先量到「给每个块拼上标题块头之后，命中率不变、索引 token 涨一成、答案文档平均名次从 2.88 退到 3.25」，据此写下「块头对关键词检索是负收益」。后来换成按文档结构切块再复核，这条名次退化**没有复现**。原因是切法变了：固定长度硬切时块边界跟小节边界不对齐，块头会把不属于这一块的标题词塞进来；按结构切时块本身就落在一个小节里，块头补的信息跟块里已有的高度重合。所以正确的表述不是「块头有害」，而是「**块头在块边界与结构不对齐时才有害**」。
这条链子的价值在于它演示了一个可复用的动作：**给每个结论标出它绑定的前提**。前提变了就要重跑，不能拿新配置去配旧结论。顺着这个思路还能给出第二个例子：曾经写过「向量侧确实是正收益」，后来把融合前的两路拆开数了一遍才发现，向量路独有的候选只占很小一部分，其中含答案文档的次数是零——那个「涨」根本不是语义检索带来的，于是这条结论被自己推翻。
可预期的追问：那你以后怎么避免这类错误？答两条具体的：一是每个数字旁边写清它绑定了哪几个前提（语料、题集、预算、切法），二是报结论前先问自己一句「这是这次实验测出来的，还是这个结构必然导致的」——前者要标边界，后者才能直接讲。
Key points
Structure the answer in four beats: the choice, the evidence then, the evidence later, the call now.
Pick a decision that was defensible at the time and later overturned by data, not one you knew was a shortcut.
Worked example: 'headers hurt keyword retrieval' was corrected to 'headers hurt when chunk boundaries misalign with structure', because the chunker premise changed.
Record the premises bound to every number; when a premise changes you owe a re-run rather than a reinterpretation.
Classify before asserting: measured by this experiment, or forced by the implementation's structure — the former needs its boundaries stated.
答题要点
答案要串成四段：当时选了什么、依据是什么、后来量到了什么、现在的判断是什么。
挑一个当时有理由、后来被数据推翻的决定，而不是一个当时就知道在凑合的决定。
样本：块头从「对关键词检索有害」修正成「块边界与结构不对齐时才有害」，因为切法这个前提变了。
每个数字旁边写清它绑定的前提；前提变了就必须重跑，不能新配置配旧结论。
报结论前先分类：这是实验测出来的，还是实现结构必然导致的——前者要标边界。
How do you convince a non-technical stakeholder that your retrieval system actually got better?怎么向不懂技术的业务方证明你的检索系统真的变好了？
Common in ChinaCommon overseasBasic#evaluation#stakeholder-communication#abstention
How to reason about it · think before answering
This is a communication question whose scoring hinges on technical judgement: which numbers you choose to show reveals whether you understand the metrics yourself. Dumping recall, nDCG and MRR on a business stakeholder reads as tone-deaf; saying 'user feedback improved' reads as unmeasured.
Start from a principle: show them something they can adjudicate themselves. They cannot judge normalized discounted cumulative gain, but they can absolutely judge 'out of these hundred real questions, how many did it answer correctly, how many wrongly, and how many did it honestly decline'. So the external framing is three numbers — correct, wrong, declined — and they sum to one hundred.
The crucial move is separating wrong from declined, and it is the fastest way to earn trust: saying 'not found' is a correct output, not a failure; the failure is inventing an answer when nothing was found. Teams that report a single 'accuracy' number can be gamed by a system that learns to decline everything, which is why all three must appear side by side.
Then supply checkable evidence rather than only numbers: take ten real questions and show before-and-after answers with clickable citations on every claim. A stakeholder who opens the source and verifies one claim is more convinced than by any percentage, and the exercise doubles as the human spot-check you need anyway to calibrate whether your model judge is trustworthy.
There is a lesson from this course worth volunteering: a column of perfect scores means the ruler is broken. Our questions were written backwards from the corpus, lexical overlap is unusually high, and mean reciprocal rank sits at exactly 1.0000. Showing that to a stakeholder only invites the misreading that you are already perfect, when in fact the metric has saturated. When a metric hits the ceiling, the response is to make the questions harder.
Expected follow-up: how do you get the business side involved? One very practical answer: let them supply questions. Every production miss gets appended to the golden set, so the evaluation set grows rather than being built once. Then each release can point at 'the question you raised last month now answers correctly', which lands better than any status report.
分析过程 · 先想清楚再作答
这题在考沟通，但拿分点在技术判断上：你选哪几个数字给业务方看，暴露了你自己有没有看懂这些指标。把召回率、nDCG、MRR 一股脑摊出去的答法会被判成不懂受众；只说「用户反馈变好了」又会被判成没有度量。
先立一条原则：**给业务方看的必须是他们能自己判断对错的东西**。归一化折损累计增益他们没法判断，而「这一百个真实问题里，系统答对了多少、答错了多少、老老实实说查不到了多少」他们一眼就能判断。所以对外的口径应该是三个数：答对率、答错率、拒答率，而且三个加起来是一百。
关键是把**答错和拒答分开**。这一条最能建立信任：查不到就说查不到不是故障，是正确输出；真正的故障是查不到还编一段。很多团队只报「准确率」，结果一个学会了一直拒答的系统能刷出满分——所以这三个数必须并排出现，缺一个都能被骗。
然后给可核对的证据，而不是只给数字：**挑十条真实问题做前后对照**，各贴出改动前和改动后的回答，每句结论后面挂着可点开的引用。业务方点开原文核对一遍，比看任何百分比都有说服力，而且这个动作顺带完成了一次人工抽检——你自己也需要它来校准模型裁判靠不靠谱。
本课里有一条要主动说的教训：**一列全是满分说明尺子坏了**。我们的题目是从语料反向出的，字面重合度过高，平均倒数排名恒为 1.0000。这个数字拿给业务方看，只会换来一次「那你们已经完美了」的误会，而它其实是指标饱和。指标撞天花板时该做的是把题目出难一点。
可预期的追问：那怎么让业务方参与进来？答一条很实用的：让他们提供题目。把线上答错的问题一条条补进标准答案集，评估集是长出来的，而不是一次性造好的；这样每一次改进都能指着「你上次提的那个问题现在答对了」，比任何汇报都直接。
Key points
Externally report three numbers they can adjudicate: correct, wrong, declined — summing to one hundred.
Keep wrong and declined separate; a single accuracy number is gamed by a system that learns to decline everything.
Pair it with ten before-and-after real questions, every claim carrying a citation they can open and verify.
Volunteer the saturation caveat: a column of perfect scores means a broken ruler, and the fix is harder questions.
Let stakeholders contribute questions; append every production miss to the golden set so it grows over time.
答题要点
对外只用三个他们能自己判断的数：答对率、答错率、拒答率，三者相加为一百。
答错和拒答必须分开——查不到就说查不到是正确输出，只报一个准确率会被「一直拒答」刷满分。
配十条真实问题的前后对照，每句结论挂可点开的引用，让他们自己核对原文。
主动说明指标饱和：某一列恒为满分是尺子坏了，不是系统完美，该做的是把题目出难一点。
让业务方提供题目，把线上答错的问题补进标准答案集——评估集是长出来的。
Users report that your live RAG system 'answers inaccurately'. What is your triage order?RAG 系统上线后用户反馈「答得不准」，你的排查顺序是什么？
Common in ChinaCommon overseasIntermediate#debugging#failure-modes#observability
How to reason about it · think before answering
This one is almost guaranteed to be asked, and most people answer with a flat list of possibilities: maybe chunking, maybe the prompt, maybe the model. A list is not triage. Triage means an order, a decision rule at each step, and each step eliminating half the search space.
First decompose the complaint. 'Inaccurate' hides at least four distinct failures whose fixes do not transfer: off-topic answers, partial answers, misaligned citations, and stale content. So the first action is not to change a setting, it is to obtain the specific question and answer and classify it into one of those four.
Then give the order along with its justification: read the pipeline right to left, fix it left to right. Right to left because the generated answer is what you see first; left to right because upstream errors are amplified downstream — no prompt can recover a document retrieval never fetched. Concretely: dump the candidate pool and the final context for that question, and check whether the answer document is in the pool at all. Absent means a retrieval debt; present but below the admission gate means a gate debt; admitted but never packed into the context budget means chunks too large or budget too small; all present and still unused means it is finally a generation problem.
One detail worth volunteering because it is easy to get wrong: for multi-hop questions, diagnose the documents that are missing, not whether any one of them was retrieved. In our experiment one question needed two documents; the first ranked first every time and the second never entered the candidate pool at all. Judging by 'any of them' labels it a budget problem, and you can spend a full day tuning budgets to no effect. This distinction only occurs to someone who has actually triaged question by question.
The fourth class, stale content, happens outside the question path and has its own rule: first check whether reconciliation even noticed the edit (was the content hash computed after line-ending normalization?), then check whether the cache key includes the index version and the permission scope. 'When must this expire' is equivalent to 'is that thing part of the key' — leave something out of the key and changes to it will never invalidate the entry.
Expected follow-up: how do you stop relying on manual triage? Build the classification into the evaluation panel so every missed question is automatically labelled with one of the four classes, and report it per tenant. A global average dilutes one customer's collapse across the whole population, and that customer is exactly the one who will file the complaint.
分析过程 · 先想清楚再作答
这题几乎是必考题，而绝大多数人答成一堆并列的可能性：可能是切块问题、可能是提示词问题、可能是模型不行。并列不是排查，排查的意思是**有顺序、有判据、每一步能把可能性砍掉一半**。
先把「答得不准」这四个字拆开——它至少塞了四种病，而且修法互不通用：答非所问、只答得出片段、引用错位、更新不生效。所以第一个动作不是改配置，是**拿到具体的问题和回答，把它归到这四类里的一类**。
然后给顺序，而且要说清顺序的理由：**排查从右往左看、修复从左往右修**。从右往左是因为你最先看到的是生成结果；从左往右是因为上游的错会被下游放大——检索没捞到的东西，再好的提示词也救不回来。具体走法是：打印这一问的候选池和最终上下文，先看答案文档在不在候选池里。不在，是检索的债；在候选池但没过准入门槛，是门槛的债；过了门槛却没装进上下文预算，是块太大或预算太小；都进了而模型没用上，才轮到生成侧。
这里有一个容易写错的细节值得主动讲：**多跳题的诊断对象是缺的那几篇，不是「有没有捞到任意一篇」**。我们实验里有一道题要同时命中两篇，第一篇稳稳排第一、第二篇一次都没进候选池；用「任意一篇」去判会把它归成预算问题，然后你去调预算，调一整天也没用。这一条区分度很高，因为它只有真的按题排查过才想得到。
第四类「更新不生效」发生在问答之外，判据是另一条：先看对账认没认出这篇改了（内容指纹算之前有没有做换行归一化），再看缓存的 key 里有没有把索引版本和权限范围算进去。「什么时候必须失效」等价于「key 里有没有把那样东西算进去」，key 少放一样，那样东西变了缓存就不会失效。
可预期的追问：怎么让这套排查不靠人肉？答案是把分类做进评估面板——每一道没中的题自动标出它属于四类中的哪一类，并按租户分开统计。全局平均会把单个客户的塌方按人头摊薄，而线上会投诉的恰恰是那个客户。
Key points
Classify the complaint into four failures first — off-topic, partial, misaligned citation, stale — because their fixes do not transfer.
Read right to left, fix left to right: dump the candidate pool and final context and find which layer the answer document stalls at.
The four rules in order: never retrieved, retrieved but below the gate, admitted but squeezed out of the budget, packed but unused by the model.
For multi-hop, diagnose only the missing documents; judging by 'any one retrieved' mislabels a never-retrieved case as a budget problem.
For stale content, check reconciliation and the cache key: what must expire is exactly what the key must contain.
答题要点
先把「答得不准」归类成四种病：答非所问、只答得出片段、引用错位、更新不生效——修法互不通用。
排查从右往左看、修复从左往右修：先打印候选池与最终上下文，看答案文档卡在哪一层。
四层判据依次是：没进候选池、进了没过门槛、过了没装进预算、都进了模型没用上。
多跳题只诊断缺的那几篇；用「有没有捞到任意一篇」会把「根本没捞到」误判成预算问题。
「更新不生效」查对账与缓存 key：什么时候必须失效，等价于 key 里有没有算进那样东西。
If you could only fund three changes to improve an existing RAG system, which three would you pick and why those three?如果预算只够做三件事来提升一个已有 RAG 系统的效果，你选哪三件？为什么是这三件？
Common in ChinaCommon overseasIntermediate#prioritization#evaluation#abstention
How to reason about it · think before answering
This tests prioritisation, not breadth. Answering with a list of techniques — add reranking, add hybrid retrieval, add query rewriting — almost always loses points, because it skips a prerequisite: how do you know those three help your system? That is precisely the sentence the interviewer is waiting for.
So the first item has to be building evaluation, with a reason specific enough to be unarguable: without a scale, you cannot tell whether the other two helped or hurt; with one, every subsequent spend has a measurable return. It is also cheap — the three retrieval metrics are pure local computation, run in seconds, cost nothing, and can gate every commit; the only real effort is labelling answer documents once. Include the composition rule: multi-hop and unanswerable each above ten percent, because without the unanswerable class a system that only ever guesses scores perfectly on your report.
Second, move abstention out of the prompt and into code — usually the best return per unit of effort, and the item most often skipped. Writing 'say you don't know' ten times in a prompt buys almost nothing. Citation numbers are a closed set, so checking existence is one line, and adding a substantive-overlap check catches the harder forgery where the number is real but the content is not. Our baseline abstention rate was 0.0 percent: four questions with no answer in the corpus, zero of them declined — a defect that is completely invisible on a report that only shows recall.
Third, look at the failure cases before deciding, which is the actual answer to this question. After reading the panel you land on one of a few branches: a high share of multi-hop means bridging retrieval or a different index structure; queries that miss when phrased differently mean you need the vector route or hybrid retrieval; answers retrieved but never packed into context means reranking or budget. Failure cases first, technique second — we tried five advanced index structures and not one beat the baseline, because our system simply did not have the weakness they address.
Why not the flashier options: agentic retrieval concentrates its gains on multi-hop while spreading cost across every question, and in our measurements turning on every query-side technique produced exactly the same recall as the default configuration while using 2.5 times the model calls and 4.3 times the retrievals. Stacking techniques is easy; explaining why you switched several off is the skill.
Expected follow-up: once the three are done, how do you prove the money was well spent? Toggle each one individually and report three ledgers — how much the metric moved, how much latency moved, how much cost moved. A proposal that reports only the first should not be approved, including your own.
分析过程 · 先想清楚再作答
这题在考优先级判断，而不是知识面。答成「上重排、上混合检索、上查询改写」这类手法清单几乎必然掉分——因为它跳过了一个前提：**你凭什么知道这三件对你的系统有用？** 面试官等的就是这句话。
所以第一件必须是**建评估**，而且理由要具体到不可反驳：没有秤，剩下两件做完你也说不清是变好还是变坏；有了秤，后面每一笔钱都能算回报。而且它便宜——检索侧三个指标是纯本地计算、几秒钟、零成本，能挂进每次提交；花时间的只是给题目标答案文档那一次。顺带说清评估集的配比：多跳与无答案各占一成以上，缺了无答案那一类，一个只会硬答的系统在报表上就是满分。
第二件是**把拒答从提示词搬进代码**，这一件的性价比通常最高而最容易被跳过。提示词里写十遍「找不到就说找不到」增益接近于零；而引用编号是一个闭集，判它存不存在只要一行代码，再加一道「这句话与被引块的实质重合度」就能拦住「编号是真的、内容是假的」那一类。我们实验里的基线拒答率是 0.0%——四道语料里根本没有答案的题一道都没闭嘴，这类缺陷在只报召回率的报表上完全不可见。
第三件要**先看失败案例再决定**，这才是这道题真正的答案。看完面板你会落到其中之一：多跳题占比高就补桥接检索或改索引结构；换个说法就捞不到，说明该上向量那一路或混合检索；答案捞到了却排不进上下文，那是重排或者预算的活。**先有失败案例，再有手法**——我们试过五种高级索引结构，没有一种跑赢基线，因为我们的系统压根没有那些结构要补的短板。
为什么不选那些看起来更亮的：Agentic 检索的收益集中在多跳题上而代价摊给全部问题；「全开」所有查询侧手法在我们的实测里召回率和默认配置一模一样，模型调用却是 2.5 倍、检索次数 4.3 倍。**堆手法很容易，说清楚为什么关掉某几项才是本事。**
可预期的追问：三件做完怎么证明钱花对了？答：每一项单独开关各跑一遍，报三笔账——指标涨了多少、延迟涨了多少、钱涨了多少。只报第一笔的提案不该被批准，包括你自己的。
Key points
First, build evaluation: without a scale the other two changes are unverifiable, and the retrieval metrics are cheap enough to gate every commit.
The golden set must include unanswerable questions, or a system that only ever guesses scores perfectly on your report.
Second, move abstention from the prompt into code: citation numbers are a closed set, and a substantive-overlap check catches real-number-fake-content forgeries.
Third is chosen by the failure cases, not by a list of techniques — failure cases first, index structure or retrieval trick second.
Toggle each change individually and report three ledgers: metric, latency, cost. A proposal reporting only the first should not be approved.
答题要点
第一件是建评估：没有秤，另外两件做完也说不清变好还是变坏；检索侧指标零成本可挂进每次提交。
评估集必须含无答案那一类，否则一个只会硬答的系统在报表上就是满分。
第二件是把拒答从提示词搬进代码：编号是闭集，再加实质重合度就能拦住「编号真、内容假」。
第三件由失败案例决定，不由手法清单决定——先有失败案例，再有索引结构或检索手法。
每一项单独开关跑一遍并报三笔账：指标、延迟、钱。只报第一笔的提案不该被批准。