Interview Bank

From Frontend Engineer to Agent Engineer in 30 Days

D21 Evaluation and Observability: a Golden Set, LLM-as-Judge, Tracing, a Failure-Rate/Cost Dashboard; Pi vs. LangGraph Summary; Week Three Retrospective

• How do you evaluate an agent's quality, and how does it differ from testing a conventional backend service?怎么评估一个 Agent 的效果？和传统后端服务的测试有什么不同？

  Common in ChinaCommon overseasIntermediate#evaluation#testing#agent-quality

  How to reason about it · think before answering

  1. The hinge is differ. Answering build a test set and measure accuracy is the textbook ML answer and misses the point; the interviewer wants to know whether you can articulate what makes agents special here.
  2. The root difference is one sentence: the same input does not guarantee the same output. Conventional tests assert equality, but an agent's output has no single correct answer, only good enough. Once the assertion changes from equality to scoring, the whole methodology changes with it.
  3. That difference cascades into three consequences, and covering all three secures the question. First, whether it ran tells you nothing about quality — the flow not throwing does not mean the reply stated the refund conclusion. Second, the blast radius of a change is diffuse: a one-word prompt edit may affect only one class of request, and hand-checking five samples that happen to miss that class yields no impact, then you ship. Third, multi-agent adds a layer: one request passes routing, planning, parallel execution, review and aggregation, and any one of them going wrong surfaces as that last paragraph seems off — without measuring each stage you cannot tell which to fix.
  4. So frame it: evaluation is not testing. Evaluation establishes a comparable baseline for a stochastic system. Its output is not pass or fail but a number you can compare against last time — and to compare, the sample set must be frozen.
  5. Then get concrete: a small stable golden set (15 items here), each declaring its expected route and a checklist of facts the reply must contain; an LLM-as-judge scoring against that checklist; and evaluation results joined to tracing on one dashboard. The checklist is the key move — it converts is this a good answer, which cannot be verified, into were these facts stated, which can.
  6. Expect: do you still need unit tests? Yes, with a clean split — deterministic parts (tool functions, state transitions, reducers) keep asserting equality in unit tests, while evaluation covers only the model-generated segment. Merge the two and you get a suite that fails randomly, after which everyone starts ignoring CI.

  分析过程 · 先想清楚再作答

  1. 题眼是「不同」。只答「建一个测试集跑准确率」拿不到分——那是机器学习的标准答案，面试官想看你能不能说清 Agent 这个场景特殊在哪。
  2. 根子上的差别只有一句：**同样的输入，Agent 不保证给同样的输出**。传统测试的断言是「等于」，而 Agent 的产出没有唯一正确答案，只有「够不够好」。断言从等值变成了判分，整套方法论跟着变。
  3. 这条差别连锁出三个后果，说全了这题就稳了：一是**跑没跑通判断不了质量**——流程没抛错，不等于回复里写明了退款结论；二是**改动的影响是弥散的**，改一个字的提示词可能只影响一类请求，人肉抽查五条恰好没覆盖到，你会得出「没影响」然后上线；三是**多 Agent 又难一层**，一次请求走路由、拆分、并行执行、评审、汇总五道工序，任何一道歪了都表现成「最后那段话不太对」，不分开量就不知道该改哪块。
  4. 所以给出定位：**评估不是测试，评估是给一个随机系统建立一条可比较的基线。** 它的产物不是「通过」或「不通过」，而是一个能和上一次比的数字。既然要比，样本集就必须固定。
  5. 然后落到具体做法：一个小而稳的 golden set（本课 15 条），每条写清期望走哪条路由和一份必备信息清单；用 LLM-as-judge 对照清单打分；把评估结果和链路追踪接到同一块面板上。**清单是关键**——它把「这答得好吗」这种没法验的问题，换成了「这几件事写没写」这种能验的问题。
  6. 可以预期的追问：那还需要单元测试吗？需要，而且分工很清楚——工具函数、状态迁移、reducer 这些确定性的部分照旧用单元测试断言等值，评估只负责模型产出那一段。把两者混成一套，你会得到一堆随机失败的测试，然后所有人开始无视 CI。

  Key points

  • The root difference: identical input does not guarantee identical output, so the assertion shifts from equality to good enough
  • Three consequences: running is not quality, change impact is diffuse (sampling misses it), and in multi-agent any of five stages failing looks like the same symptom
  • Framing: evaluation is not testing — it establishes a comparable baseline for a stochastic system, yielding a number rather than pass/fail
  • Method: a small stable golden set, a required-facts checklist per item, an LLM-as-judge, and a dashboard sharing tracing's data source
  • The checklist is the key move: it converts is this good into were these facts stated — unverifiable into verifiable
  • Unit tests remain for deterministic parts; merging the two makes CI fail randomly until everyone ignores it

  答题要点

  • 根本差别：同样的输入 Agent 不保证同样的输出，断言从「等于」变成「够不够好」
  • 三个后果：跑通不等于质量合格、改动影响弥散（抽查会漏）、多 Agent 里五道工序任一歪了都表现成同一个症状
  • 定位：评估不是测试，是给随机系统建一条可比较的基线，产物是能和上次比的数字而不是通过与否
  • 做法：小而稳的 golden set + 每条的必备信息清单 + LLM-as-judge 打分 + 与 tracing 同源的面板
  • 清单是关键，它把「答得好吗」换成「这几件事写没写」，从没法验变成能验
  • 单元测试仍然需要，负责确定性部分；两者混在一起会让 CI 随机变红，最后被所有人无视

Prompt Engineering From Scratch in 5 Days

D4 Iteration and Evaluation: Small Test Sets, A/B Testing, Version Control, Common Anti-Patterns

• How do you version prompts? How does it differ from versioning code, and what is your first move when production misbehaves?提示词版本化怎么做？它和代码版本管理有什么不同？线上出问题你先做什么？

  Common in ChinaCommon overseasIntermediate#prompt-versioning#evaluation

  How to reason about it · think before answering

  1. One of the most frequent prompt-engineering interview questions abroad; it tests whether you have managed prompts as production assets. 'Put it in git' is the floor; the interviewer wants the changelog contents and why prompts differ from code.
  2. Shape first: prompt text in its own file, an id per version, decoupled from business code. Then the changelog's four items: what changed, why (which test cases failed), the measured pass rate, known regressions. The pass rate must come from the script.
  3. The difference is the differentiator: code changes are usually local, prompt changes are global — one added sentence can shift behavior on every input, so 'known regressions' is mandatory where commit messages have no such field. Also rollback is nearly free, just swap a string.
  4. Conclusion: when production misbehaves, roll back to the previous version first, then add the triggering input to the test set and investigate — which only works if you have version ids and a test set.
  5. Follow-ups: bind prompt versions to model versions? Yes — pass rates shift across model versions, so record which model was used. And canarying: route by version id and compare live metrics, same as code.

  分析过程 · 先想清楚再作答

  1. 这题是海外面试里提示词工程方向出现频率最高的一道，考的是「有没有把提示词当成生产资产管理过」。答「放进 git」是最低分，面试官要听的是变更记录里写什么、以及为什么和代码不一样。
  2. 拆法：先说形态——提示词正文独立成文件、每版有 id、与业务代码解耦；再说变更记录四件事——改了什么、为什么改（对应哪几条测试失败）、跑出来的通过率、已知回退。通过率必须是脚本跑出来的数字。
  3. 不同点是区分度所在：代码改动通常是局部的，提示词改动是全局的——加一句话可能改变所有输入的行为，所以「已知回退」是必填项而代码提交信息里没有这一栏；另一点是回滚成本几乎为零，只是换一个字符串。
  4. 结论：线上出问题第一步是回滚到上一版，再拿触发问题的输入补进测试集慢慢查——前提是你有版本号可回、有测试集可跑。
  5. 追问：提示词版本要不要和模型版本绑定？要——同一份提示词在不同模型版本上通过率会变，记录里要写清是在哪个模型上测的；另一个追问是多环境怎么灰度，答案是按版本 id 分流并对比两版的线上指标，跟代码灰度一样。

  Key points

  • Prompt text lives in its own file with a version id; the changelog records change, reason, measured pass rate, known regressions
  • Unlike code, prompt changes are global, so known regressions are mandatory; rollback is nearly free
  • On a production issue, roll back first, then add the triggering input to the test set
  • Record which model version was tested, since pass rates shift across models

  答题要点

  • 提示词正文独立成文件、每版有 id，变更记录写改动、原因、脚本跑出的通过率、已知回退
  • 与代码的不同：改动是全局的，所以「已知回退」必填；回滚成本几乎为零
  • 线上出问题先回滚上一版，再把触发输入加进测试集查
  • 版本要记录在哪个模型上测的，换模型版本通过率会变
• Is using a model to grade another model's output reliable? When is it acceptable, and when must a human look?用模型给模型打分靠谱吗？什么时候可以用，什么时候必须人工看？

  Common in ChinaCommon overseasIntermediate#evaluation#llm-as-judge

  How to reason about it · think before answering

  1. This tests whether you know the judge is fallible too. 'Use a stronger model as the judge' means you never calibrated one; 'prefer field comparison whenever possible' shows judgment.
  2. Split by task: structured output gets field-by-field code comparison, no judge needed; free text (summaries, emails, explanations) has no fields, and a judge is the only scalable option.
  3. Judge biases: longer and prettier answers score higher, stylistic similarity gets rewarded, vague rubrics produce noisy scores, factual errors are under-penalized. So the rubric must be concrete — list the information points, one point each — not 'rate this summary 1 to 10'.
  4. Conclusion: usable once calibrated against ten human-scored samples with an agreement rate you accept; spot-check regularly; anything involving facts, safety or money still gets human review.
  5. Follow-ups: same vendor for judge and judged? Expect self-preference, so switch vendor or at least version. And cost — every judgment is a full call, so let code handle whatever it can first.

  分析过程 · 先想清楚再作答

  1. 这题在考「知道裁判也会错」。答「用更强的模型当裁判就行」的人没校准过裁判；答出「能用字段比对就不用裁判」才说明有判断力。
  2. 拆法：先分任务。输出是固定字段就用代码逐字段比对，不需要裁判；输出是自由文本（摘要、邮件、解释）才没有字段可比，这时裁判是唯一能规模化的办法。
  3. 再说裁判的偏差：偏向长的、格式漂亮的、和自己风格接近的回答；评分标准含糊时打分随意；对事实性错误不敏感。所以评分标准要像便签一样具体——列出信息点、每点一分——而不是「给这段摘要打 1 到 10 分」。
  4. 结论：裁判可以用，前提是先拿十条人工打过分的样本校准它，看它和人的一致率；上线后定期抽样复核；对涉及事实、安全、金额的输出必须人工看。
  5. 追问方向：裁判和被评的模型是同一家会怎样？会有自我偏好，尽量换一家或至少换一个版本；另一个追问是「裁判的成本」，每条评估都是一次完整调用，测试集大了要算钱，所以能用代码判的部分先用代码判掉。

  Key points

  • Prefer field comparison; reserve the judge for free text with nothing to compare
  • Judges favor long, well-formatted answers and score noisily on vague rubrics, so rubrics must list concrete points
  • Calibrate against ten human-scored samples first, then spot-check regularly
  • Facts, safety and money always get human review; use a different vendor or version to avoid self-preference

  答题要点

  • 能用字段比对就不用裁判；裁判只用于没有字段可比的自由文本
  • 裁判偏向长的、格式漂亮的回答，评分标准含糊就打得随意，所以标准要具体到信息点
  • 先用十条人工打分样本校准裁判，上线后定期抽样复核
  • 涉及事实、安全、金额的输出必须人工看；裁判尽量换一家或换版本以避免自我偏好

Mastering Codex and the OpenAI Agents SDK in 5 Days

D5 Choosing and Combining Claude and Codex: A Real Side-by-Side on the Same Task, a Write-One-Review-One Mixed Workflow

• Your team must pick between two coding agents. How do you propose a comparison that teammates can both understand and verify?团队要在两家 coding agent 之间选一个，你怎么给出一套可以向团队解释、也能被验证的对比维度？

  Common in ChinaCommon overseasIntermediate#coding-agent#evaluation#decision-making

  How to reason about it · think before answering

  1. This tests methodology, not a verdict; leading with 'I prefer X' signals weak engineering judgment. Show how you make the comparison reproducible.
  2. Give the dimensions: instruction effort (prompt and instruction-file size), approvals (how many interruptions and why), verification (does it run tests unprompted, what happens on red), cost (time, tokens, money). All are measurable in your own repo.
  3. State the preconditions for comparability: same starting commit, identical requirement text, identical instruction-file content, default permissions, and 'run tests before reporting' on both sides.
  4. Then the reading order: check comparability, then structural differences (permission model, placement and wording of rules), and only then capability differences, which need several runs and a median.
  5. For the team: label every differing row as 'workflow' or 'capability'; workflow gaps are closed by configuration, capability gaps drive the choice.
  6. Expect the follow-up: why not benchmarks? They score standard problems with one number, while teams change legacy repos and care about four dimensions.

  分析过程 · 先想清楚再作答

  1. 这题考的是方法论而不是结论。上来就说「我觉得 X 好」会被判为没有工程判断；面试官想听的是你怎么让比较可复现。
  2. 先给维度：交代（写多少需求、准备多少说明文件）、审批（中断几次、为了什么）、验证（是否主动跑测试、红了怎么办）、成本（时间、token、钱）。这四项都能在自己的仓库里量出来。
  3. 再给可比性的前置条件：同一个起点 commit、同一段需求文字、说明文件同内容、默认权限、都要求跑完测试再汇报；有一项不同，差异就说不清来源。
  4. 然后是读数的顺序：先查可比性，再看结构性差异（权限模型、说明文件的位置与措辞导致的行为差别），最后才看能力差异，而且能力差异要多次运行取中位数。
  5. 落到团队沟通：报告里每一行差异都标「来自工作方式还是能力」，工作方式的差异靠配置弥补，能力差异才影响选型。
  6. 可预期的追问：榜单为什么不够？榜单测标准题，团队干的是有历史包袱的仓库里的改动，且榜单只给一个分数、不给四个维度。

  Key points

  • Four measurable dimensions: instruction effort, approvals, verification, cost, all measured in your own repo
  • Comparability first: same commit, same prompt, same instruction file, default permissions, tests required
  • Read in order: comparability, structural differences, then capability, with medians over several runs
  • Label each gap as workflow or capability; only capability gaps should drive the decision

  答题要点

  • 四个可量维度：交代、审批、验证、成本，全部在自己仓库里测
  • 可比性前置：同起点、同需求、同说明文件、默认权限、都要求跑测试
  • 读数顺序：可比性、结构性差异、能力差异；能力差异要多次运行取中位数
  • 每行差异标「工作方式还是能力」，前者靠配置弥补，后者才决定选型

Agent Skills in 7 Days: Turn Experience Into Reusable Capability

D3 A Design Method: Distilling From Repeated Tasks, Checklist Style vs. Reference-Manual Style, Four Anti-Patterns, and Trigger Testing

• How do you evaluate whether a skill description is good? Is trying a few prompts yourself enough?怎么测一个 skill 的 description 好不好？自己试几句话够吗？

  Common in ChinaCommon overseasIntermediate#agent-skills#evaluation

  How to reason about it · think before answering

  1. The hinge is the second half. Saying a few prompts is enough fails immediately, but saying write a test set is not enough either; the interviewer wants the design.
  2. Explain why spot checks fail: fine with one skill, useless at ten, because you cannot hold ten descriptions in your head nor tell whether an edit helped or hurt.
  3. Then give three ingredients. First, a labeled query set of about twenty, balanced positive and negative. Vary positives along phrasing, explicitness, detail and complexity; the most valuable positives are the ones where the skill applies but the wording does not say so.
  4. Negatives are where the design effort goes: unrelated sentences test nothing. Near-misses that share keywords but need something else are what matters, such as editing Excel formulas or loading CSV rows into a database for a CSV-analysis skill.
  5. Second, repeat runs for a trigger rate, since model behavior is nondeterministic: three runs per query with a 0.5 threshold. Third, a roughly sixty-forty train and validation split with the validation set untouched.
  6. Expected follow-up: how do you decide whether a query triggered? Build the catalog of names and descriptions, hand it plus the query to the model and ask which skill applies. That is exactly what a client does at discovery, run standalone.

  分析过程 · 先想清楚再作答

  1. 题眼在后半句。答「自己试几句就行」直接出局，但只答「要写测试集」也不够——面试官要看你知不知道这个测试集该怎么设计。
  2. 先说为什么抽查不够：一个 skill 时够用，装到第十个就不行了，因为你既记不住十个描述之间会不会互相抢，也没法在改完一句话后判断是改好了还是改坏了。
  3. 然后给三件东西。第一是带标注的查询集，约 20 条，正负各半。正例要在措辞、显式程度、详略、复杂度四个维度上铺开；**最有价值的正例是那些确实该用但字面看不出来的**，字面已经念了一遍功能的查询任何描述都能命中，测不出区别。
  4. 负例是设计的重点：毫无重叠的句子测不出任何东西，真正有用的是近似负例——共享关键词或概念但目标动词不同。对 CSV 分析 skill，「改 Excel 预算表的公式」和「把 CSV 每行写进数据库」都是好负例。
  5. 第二是重复跑取触发率：模型是不确定的，每条跑三次算命中比例，阈值取 0.5。第三是训练验证拆分，六比四，验证集全程不看。
  6. 可预期的追问是「怎么判断一条查询触发了没有」。答案是把所有 skill 的名字与描述拼成目录，连同这句话交给模型问它该用哪一个——这正是客户端在发现阶段做的事，只是单独拎出来跑。

  Key points

  • Spot checks work for one skill and break down once several skills compete.
  • About twenty labeled queries, balanced, with positives varied by phrasing, explicitness, detail and complexity.
  • Negatives must be near-misses that share keywords but need a different action.
  • Three runs per query for a trigger rate with a 0.5 threshold, because behavior is nondeterministic.
  • Split roughly sixty-forty; train guides revision, validation picks the winning version.

  答题要点

  • 抽查在一个 skill 时够用，多个 skill 互相干扰时完全不够。
  • 约 20 条带标注查询，正负各半，正例在措辞、显式程度、详略、复杂度四维上铺开。
  • 负例必须是近似负例：共享关键词但目标动词不同，无关句子测不出东西。
  • 每条跑三次取触发率，阈值 0.5，因为模型行为不确定。
  • 训练验证六四拆分，训练集指导改写，验证集只用来选版本。

RAG in 14 Days: From Retrieval to Trustworthy Answers

D1 Why Retrieve at All: Hallucination, Knowledge Cutoffs, and the Cost of Long Context; a Minimal Keyword-Only RAG

• A retrieval-augmented generation system gave a wrong answer. How do you determine whether retrieval or generation is at fault?一个检索增强生成系统答错了，你怎么定位是检索的锅还是生成的锅？

  Common in ChinaCommon overseasIntermediate#debugging#failure-modes#evaluation

  How to reason about it · think before answering

  1. The question asks how you localise the fault, not what the possible causes are. Listing causes loses; the interviewer wants an ordered procedure that ends in concrete actions.
  2. Give the cheapest first step: print the retrieved passages verbatim and read them. If the correct answer is not in there, retrieval is at fault. If it is in there and the model ignored it, generation is at fault. Thirty seconds, and it removes most of the guesswork.
  3. Then lay out the five stages — chunking, indexing, retrieval, context assembly, generation — with the rule: diagnose right to left, fix left to right. You see the generated answer first, but an error on the left is amplified by everything to its right.
  4. Add symptoms that pin down a stage: half-correct answers usually mean a rule was split across chunks; obviously irrelevant hits usually mean dirty parsing; the model ignoring the supplied material usually means the prompt never said it must; citation numbers that do not match their content point at generation.
  5. Land it in engineering terms: to run this procedure repeatedly you must log the retrieved hits, the passages that entered the context, and the final answer together, otherwise production issues are unreproducible. At scale this becomes a fixed question set with metrics rather than case-by-case reading.
  6. Expected follow-up: if retrieval missed the document, will prompt tuning help? No. Nothing in the prompt can conjure material that was never supplied.

  分析过程 · 先想清楚再作答

  1. 题眼在「怎么定位」，不在「有哪些原因」。答成一串可能原因的罗列就输了，面试官想听的是一个有先后顺序、能落到具体动作的排查流程。
  2. 先给最省时间的第一步：把这次检索出来的几段原文原样打印出来，自己读一遍。正确答案不在里面就是检索的锅，在里面而模型没用上才是生成的锅。这一步三十秒，能省掉大半天的瞎猜。
  3. 然后把链路展开成五个环节——切块、建索引、检索、组装上下文、生成——并给出「排查从右往左、修复从左往右」这条口径：从右往左是因为你最先看到的是生成结果，从左往右是因为左边的错会被右边放大。
  4. 补充几个能把环节钉死的症状：答案「半对」多半是切块把一条完整规则切断了；检索结果里混着一眼不相干的东西多半是解析没做干净；模型无视材料用先验知识作答，通常是提示词里少了「只能依据资料回答」；引用编号和内容对不上，那是生成侧漏读或串了行。
  5. 最后落到工程做法：这套排查要能重复做，就必须把每次请求的检索结果、进上下文的段落、最终回答一起记下来，否则线上出问题时你根本复现不了。到了要批量做的时候，就得换成一批固定问题加指标，而不是一条条人工看。
  6. 可预期的追问：如果检索确实没捞到，改提示词有没有用？答案是没用——材料里没有的东西，再好的指令也只能换一种编法。这句话最能证明你分清了两层。

  Key points

  • Always start by printing the retrieved passages and checking whether the correct answer is present at all.
  • Split the pipeline into chunking, indexing, retrieval, context assembly and generation; diagnose right to left, fix left to right.
  • Use symptoms to pin the stage: half-correct answers point at chunking, irrelevant hits at parsing, ignored material at the prompt, mismatched citations at generation.
  • If retrieval missed the document, prompt changes cannot help; the material simply is not there.
  • Log retrieved hits, the passages that entered the context, and the final answer together, or production failures are unreproducible.

  答题要点

  • 第一步永远是把检索出来的原文打印出来读一遍，判断正确答案在不在里面。
  • 把链路拆成切块、建索引、检索、组装上下文、生成五个环节，排查从右往左、修复从左往右。
  • 用症状钉环节：半对多半是切块问题，混入无关结果多半是解析问题，无视材料多半是提示词缺约束，引用与内容对不上是生成问题。
  • 检索没捞到时改提示词没有意义，材料里没有的东西模型只能编。
  • 要能重复排查就必须把检索结果、进上下文的段落和最终回答一起记录下来。

D2 Embeddings and Vector Search: Similarity, Dimensionality, and Model Choice; Storing Text in pgvector

• Why do some embedding models require different prefixes for queries and documents? What happens if you skip them, and how would you catch it before shipping?为什么有些 embedding 模型要求查询和文档加不同的前缀？不加会怎样，你怎么在上线前发现这个问题？

  Common in ChinaCommon overseasIntermediate#embeddings#model-selection#evaluation

  How to reason about it · think before answering

  1. The core of this question is silent failure. Reciting 'e5 needs query: and passage: prefixes' is the baseline; explaining why nothing errors out and how you would catch it is what shows experience.
  2. The reason: these models are trained on pairs, short questions on one side and longer passages on the other, two genuinely different distributions. The prefix is a role marker learned during training. Omit it at inference and you are off-distribution.
  3. The consequence: the model still returns vectors, distances still compute, results still have an order, quality just degrades. Nothing throws, exactly like forgetting to normalise.
  4. How to catch it: run a small labelled question set against the same corpus twice, with and without prefixes, and compare hit rate. That is the evaluation gate built on day 8, and catching silent regressions is precisely what it is for.
  5. Mention the sneakier variant: prefixing at index time but not at query time, or using the same prefix on both sides. Everything sits in one coordinate space and looks healthier, yet the query-document alignment is wrong and the loss is just as invisible. Encapsulate prefixes in the embedding call convention rather than hand-writing them everywhere.
  6. Expected follow-up: do OpenAI models need prefixes? No, they are not in that family, so this is not a universal rule but a per-model detail you re-check on the model card every time you switch.

  分析过程 · 先想清楚再作答

  1. 这题的题眼是「静默失效」。会背「e5 要加 query 和 passage 前缀」只能拿基础分，能说清它为什么不报错、以及怎么在上线前抓住它，才是做过的人。
  2. 先讲原因：这一族模型是拿成对数据训练的，一侧是短问句、一侧是长段落，两者的分布本来就不一样。前缀是训练时给模型的角色标记，告诉它这一段该按查询编码还是按文档编码。推理时不给，模型就落在了训练分布之外。
  3. 再讲后果的性质：不加前缀模型照样输出向量、照样能算距离、名次照样有先后，只是整体质量下滑。**没有任何报错**——这跟忘了归一化是同一类问题：错误不会自己浮出来。
  4. 怎么发现：唯一可靠的办法是一小份标注问题集，用同一批文档跑两遍（加前缀与不加前缀），比命中率。这就是第 8 天要做的评估闸门，它的价值恰恰在于抓这类静默错误。上线前跑一遍，比读十遍文档管用。
  5. 补一个更容易踩的变体：**建库时加了前缀、查询时忘了加**，或者两边加成同一个前缀。这种情况下所有向量都在同一个坐标系里，看起来更「正常」，但查询与文档的对齐关系是错的，掉分同样查不出来。所以前缀应该封装在 embed 的调用约定里，而不是散在各处手拼。
  6. 可预期的追问：OpenAI 的模型要不要加前缀？不需要——它不属于这一族。所以这不是一条普遍规则，而是**每换一个模型都要重新读模型卡片确认**的事。

  Key points

  • These models are trained on question-passage pairs; the prefix marks which role a text plays, and omitting it puts you off-distribution.
  • Skipping prefixes never errors, it only degrades quality, so the failure is silent.
  • The reliable detection is an A/B run over a small labelled question set, comparing hit rate.
  • A subtler bug is mismatched or identical prefixes on both sides, which looks healthier but misaligns queries and documents.
  • Keep prefixes inside the embedding call convention, and re-read the model card whenever you switch models.

  答题要点

  • 这类模型用问句与段落的成对数据训练，前缀是区分两种角色的标记，缺了就落在训练分布之外。
  • 不加前缀不会报错，只会整体掉分，属于静默失效。
  • 唯一可靠的发现方式是拿一份标注问题集跑 A/B 对比命中率。
  • 更隐蔽的错法是两边前缀不一致或用了同一个前缀，看起来更正常但对齐是错的。
  • 前缀应封装在 embed 的调用约定里；换模型必须重读模型卡片，它不是普遍规则。

D4 Chunking Strategies: Five Approaches — Fixed, Recursive, Structure-Based, Parent-Child, and Semantic — and Choosing by Evaluation, Not Intuition

• How do you decide on chunk size? Name two metrics you would look at, and one counterexample.你怎么决定切块大小？说出你会看的两个指标和一个反例。

  Common in ChinaCommon overseasIntermediate#chunking#evaluation

  How to reason about it · think before answering

  1. The question is about method, not about a number. Answering with a specific default (512 tokens, 1000 characters) already loses it — the interviewer wants to hear that you have a procedure.
  2. State the tension first: large chunks dilute the signal and cost context; small chunks lose the surrounding meaning so the model cannot use them. The two metrics you name should map onto those two failure modes.
  3. Metric one is retrieval-side hit rate: did a document that actually answers the question make it into the context. Metric two is generation-side usability, cheaply proxied by the fraction of chunks that end mid-sentence, and more seriously by faithfulness and whether citations resolve.
  4. Add the point that separates candidates: both metrics must be compared under the same token budget, never under a fixed top-k. With fixed k, bigger chunks simply buy more text and win for the wrong reason.
  5. Make the counterexample concrete: raising chunk size from 400 to 1200 characters can lift hit rate purely because whole short documents now fit in one chunk, which means retrieval stopped doing anything and you are back to stuffing full documents. The metric improved while the system got worse.
  6. Expect the follow-up: where do you start on day one. Pick the strategy from the document type first (structural splitting whenever headings exist), start around 300 to 500 characters with 10 to 20 percent overlap, then build a golden set immediately and iterate. A starting point is not a conclusion.

  分析过程 · 先想清楚再作答

  1. 这题的题眼是「怎么决定」，不是「多大合适」。答一个具体数字（512 token、1000 字符）就已经输了——面试官想看的是你有没有一套定法，而不是你记得住哪个默认值。
  2. 先把矛盾摆出来：块大则信噪比低、上下文贵，块小则单块缺语境、模型答不出所以然。切块大小就是在这两头之间找位置，所以两个指标必须分别对应这两头。
  3. 第一个指标是检索侧的命中率——答案文档有没有进上下文。第二个是生成侧的可用性，最省事的代理指标是切碎率，也就是有多少块结尾停在半句话上；再往前一步就是忠实度和引用是否可定位。
  4. 关键补一句：两个指标必须在**同一个 token 预算**下比，不能按「取前 k 块」比。k 固定时块越大塞进去的字越多，大块切法会赢在买得多而不是切得准上。这一句往往是这道题的区分点。
  5. 反例要具体。最好用的一个是：把块从 400 字调到 1200 字，命中率不降反升——但那是因为一整篇短文档被当成一块塞了进去，检索其实什么都没做，等于退化成了全文投喂。指标涨了，系统更差了。
  6. 可预期的追问是「那你第一次上手时从哪个数字起步」。答：先按文档类型选切法（有标题层级就按结构切），块长从 300 到 500 字起步、重叠取一到两成，然后立刻建一组标准问题跑评估，用两三轮迭代把它调到位。起步值是起步值，不是结论。

  Key points

  • Choose the strategy from the document type first, then tune length: split on headings whenever the structure survives parsing.
  • Watch two metrics: retrieval hit rate on one side, mid-sentence break rate (then faithfulness and citation resolvability) on the other.
  • Compare under an equal token budget, never a fixed top-k, or larger chunks win by buying more text.
  • Counterexample: hit rate rises after enlarging chunks because whole documents now fit in one chunk and retrieval has effectively stopped working.
  • Start near 300 to 500 characters with 10 to 20 percent overlap, then iterate against a fixed question set instead of guessing.

  答题要点

  • 先按文档类型选切法，再调长度：有标题层级就按结构切，没有结构才谈固定长度或语义。
  • 看两个指标：检索侧的命中率，生成侧的切碎率（进一步是忠实度与引用可定位性）。
  • 两个指标必须在同一个 token 预算下比，不能按「取前 k 块」比，否则大块只是买得更多。
  • 反例：块调大后命中率上升，但那是因为整篇被当成一块，检索退化成全文投喂。
  • 起步值 300 到 500 字、重叠一到两成，然后靠一组固定问题迭代，不靠直觉定稿。

D5 Vector Indexes and Store Selection: HNSW vs. Inverted File, Quantization to Save Memory, Filtered Queries and Multi-Tenant Isolation

• If you switch your vectors from full precision to half precision or binary quantisation, how do you verify that recall has not dropped materially?把向量从全精度换成半精度或二值量化，你会用什么方法确认召回没有明显下降？

  Common in ChinaCommon overseasIntermediate#quantization#evaluation#recall

  How to reason about it · think before answering

  1. The question looks like it is about quantisation, but it is really about whether you know how to evaluate. Answering 'try a few queries and eyeball it' fails immediately.
  2. Pin down ground truth first: it must come from an exhaustive scan with the index disabled. Using index results as ground truth is the classic self-deception, because recall then looks close to 100% no matter what you changed.
  3. Give the procedure: fix a query set of at least a few dozen covering short and long queries across topics, compute ground truth at full precision, rerun with the quantised representation, and report recall at k. Report index size, build time, and median plus p95 latency alongside it, because recall alone is not a decision.
  4. Add the judgement rule: quantisation loss depends on your vector distribution, so published numbers do not transfer. Sparse vectors suffer badly under binary quantisation because only the sign bit survives and zeros collapse together.
  5. Land on something actionable: half precision is usually near lossless and raises the indexable dimension ceiling from 2000 to 4000, so it is a safe first step. Binary quantisation loses real recall and should be used as a cheap first pass, re-ranked with the original vectors over a wider candidate window.
  6. Expected follow-up: how much loss is acceptable? It depends on what comes next. With a re-ranker downstream, a couple of points off first-stage recall is usually invisible; if retrieval feeds the prompt directly, one point means one more unanswerable question per hundred. Tie the threshold to a product metric, not to a number you made up.

  分析过程 · 先想清楚再作答

  1. 这题表面问量化，实际问的是你会不会做评估。只回答「跑几个问题看看结果对不对」的人会被直接判为没做过——面试官想听的是一套可复现的量法。
  2. 先把真值这件事说死：真值必须来自暴力全量比对，也就是把索引关掉、全表算距离取前 k。拿索引结果当真值是最常见的自欺，因为那样量出来的召回永远接近 100%，你会以为量化无损。
  3. 然后给流程：固定一批查询（几十条起步，覆盖长短查询和不同主题），先用全精度算出真值，再换量化重跑，计算召回率@k。同时记录三件事——索引大小、建索引耗时、查询延迟的中位数与 p95，只报召回是不够的。
  4. 补一条判据：量化损失有多大取决于向量分布，别人的数字不能抄。稀疏向量对二值量化尤其不友好，因为二值化只保留符号位，零和负数会被压成同一个值，信息几乎被抹平。所以换方案必须在自己的数据上重新量一次。
  5. 结论要给可操作的建议：半精度通常近乎无损，还能把建索引维度上限从 2000 提到 4000，是默认可以先上的一档；二值量化损失明显，标准用法是拿它粗筛一批候选，再用原始向量在这一小批里精排，粗筛窗口越宽召回补得越多、延迟也越高。
  6. 可预期的追问：召回掉了多少算可以接受？答这取决于下游——后面还有重排时，粗排召回掉两三个点通常无感；如果检索结果直接进提示词，掉一个点就意味着每一百次回答里多一次缺材料。要把这个判断挂到业务指标上，而不是拍一个阈值。

  Key points

  • Ground truth must come from an exhaustive scan with indexes disabled; using index output as truth pins recall near 100%.
  • Run one fixed query set before and after, report recall at k together with index size, build time and latency percentiles.
  • Quantisation loss depends on your own vector distribution, so measure it on your data instead of quoting benchmarks.
  • Half precision is usually near lossless and raises the indexable dimension limit from 2000 to 4000, making it a safe default.
  • Binary quantisation loses real recall; use it as a cheap first pass and re-rank with the original vectors over a wider window.

  答题要点

  • 真值必须来自关掉索引的暴力全量比对，拿索引结果当真值会让召回永远接近 100%。
  • 固定一批查询，量化前后跑同一批，报召回率@k，同时报索引大小、建索引耗时和延迟分位数。
  • 量化损失取决于向量分布，别人的数字不能抄，必须在自己的数据上重新量。
  • 半精度通常近乎无损，还能把索引维度上限从 2000 提到 4000，可以作为默认第一档。
  • 二值量化损失明显，正确用法是粗筛加原始向量重排，粗筛窗口越宽召回补得越多、延迟越高。

D8 Evaluation First: Building a Golden Set, Computing Recall and Ranking Metrics, Using a Model as Judge for Faithfulness

• You need to build an evaluation set from scratch for a RAG system over a company knowledge base. How would you do it, and how many questions are enough?让你从零给一个公司知识库的 RAG 系统建评估集，你会怎么做？多少题才算够用？

  Common in ChinaCommon overseasIntermediate#evaluation#golden-set#rag

  How to reason about it · think before answering

  1. The discriminator here is the direction you generate questions in, and whether you can justify a size rather than name one.
  2. Go corpus-first: read each document and write the questions it can answer. The answer document is fixed at authoring time, so labeling is nearly free. Question-first gives you items whose answers nobody can locate.
  3. Give the schema: question, answer document ids, and a type. At minimum three types - single-document, multi-hop, and unanswerable. Multi-hop counts as a hit only when every answer document makes it into the context; unanswerable items are scored on abstention, not recall.
  4. Justify the size: 20 items separate 'broken' from 'usable' and are enough for a smoke gate; 100 to 200 are needed before a two-point delta means anything. Then grow the set - every production failure becomes a new item.
  5. Mention cost and decay: roughly two hours for 20 items, and answer labels must be rechecked whenever the corpus changes, or the set rots and you misread the drop as a system regression.
  6. Expected follow-up: how do you avoid overfitting to the eval set? Keep a held-out slice that never informs tuning, and refresh it from real production questions.

  分析过程 · 先想清楚再作答

  1. 这题的区分度在「出题方向」和「规模的理由」两处。开口就说「找几百个用户真实问题」的，多半没真做过——真实问题的答案在哪篇文档里，没人标得出来。
  2. 先给方向：从语料反向出题，打开每一篇读它能回答什么，出题的那一刻答案文档就已经确定了，标注成本几乎为零。反方向（先想问题再找答案）会得到一堆自己都不知道答案的题。
  3. 再给结构：每题记问题、答案文档列表、类型三个字段；类型至少分单文档、多跳、无答案三类，并说明多跳必须全部答案文档命中才算命中，无答案不参与召回率而是考拒答。
  4. 规模的理由要给出来，不能只报一个数字：20 题能把「完全不能用」和「基本能用」分开，够做冒烟；100 到 200 题才有资格判断「涨了两个点」是真的还是噪声。上线之后每次线上出问题就把那个问题补进集合——评估集是长出来的。
  5. 补一句成本与保鲜：出题是人力活，20 题两小时是正常量级；语料更新后要复核答案文档还在不在，否则集合会悄悄腐烂，指标下跌你会误以为是系统坏了。
  6. 可预期的追问是「怎么防止评估集被过拟合」。答案是留一份不参与调优的保留集，并且定期从线上真实问题里补充新题，只用来验收不用来调参。

  Key points

  • Author corpus-first so the answer document is known at authoring time.
  • Label every item with a type: single-document, multi-hop, unanswerable.
  • Multi-hop requires all answer documents; unanswerable items score abstention, not recall.
  • 20 items for a smoke gate, 100 to 200 to trust small deltas, and keep growing it from production failures.
  • Hold out a slice that never informs tuning to avoid overfitting the set.

  答题要点

  • 从语料反向出题，出题时答案文档就已确定，标注成本最低。
  • 每题标类型：单文档、多跳、无答案，三类缺一不可。
  • 多跳要求全部答案文档命中；无答案不算召回率，考的是拒答。
  • 20 题够冒烟，100 到 200 题才能判断小幅变化；线上故障持续补题。
  • 留一份不参与调优的保留集，防止对评估集过拟合。
• Recall, mean reciprocal rank, and normalized discounted cumulative gain - which failure mode does each one catch first, and what do you miss by watching only one?召回率、平均倒数排名、归一化折损累计增益，这三个检索指标分别在什么故障下会先掉下来？只盯一个会漏掉什么？

  Common in ChinaCommon overseasIntermediate#retrieval-metrics#evaluation#ranking

  How to reason about it · think before answering

  1. This tests whether you know each metric's blind spot, not whether you can recite definitions. Layer them as 'did it show up / how high / how good overall' and you are halfway there.
  2. Recall is boolean: is the answer document in the final context. It catches 'never retrieved', but it does not move when the answer slips from rank 1 to rank 8, as long as it still fits the budget.
  3. MRR looks only at the rank of the first relevant hit, so ranking degradation shows up immediately. Its blind spot: one relevant item in the top ten scores exactly the same as five.
  4. nDCG discounts every relevant hit in the top k by its position, so it tracks overall ranking quality and is the direct optimization target for reranking. Its blind spot is existence - it is zero both when nothing was retrieved and when ranking is terrible.
  5. Conclusion: together they localize the failure. Recall drops means retrieval or chunking; recall flat but MRR down means ranking degraded, reach for a reranker; both stable but nDCG down means more noise crept into the top results.
  6. Expected follow-up: what if a metric saturates? Make the questions harder - a saturated metric means the eval set lost its discriminative power, and further tuning is blind.

  分析过程 · 先想清楚再作答

  1. 这题考的是「知不知道指标之间的盲区」，不是背定义。能把三者按「有没有 / 靠不靠前 / 整体好不好」分层的，基本就答对了一半。
  2. 推导链是这样的：召回率是布尔的——答案文档在不在最终上下文里。它对「压根没捞到」最敏感，但答案从第 1 名掉到第 8 名它一动不动，只要还在预算内。
  3. 倒数排名只看第一条相关结果的名次，所以「答案还在但被挤到后面」它立刻掉。反过来它有个盲区：前十条里有一条命中还是五条命中，它给的分完全一样。
  4. 归一化折损累计增益把前 k 名里每一条相关结果都按名次折算再累加，所以它对「整体排序质量」敏感，是重排最直接的优化目标。它的盲区是不告诉你「有没有」——召回率为零时它也是零，看不出是没捞到还是排得差。
  5. 结论：三个一起看才能定位故障层。召回率掉说明检索或切块出了问题，要动召回策略；召回率不动而倒数排名掉，说明排序退化，该上重排；两者都稳而 nDCG 掉，说明前几名里混进了更多噪声。
  6. 可预期的追问是「指标顶格了怎么办」。真实答案是把题目做难：指标撞天花板说明评估集失去区分度，这时候继续优化系统是在瞎调。

  Key points

  • Recall answers 'did it make it into the context', sensitive to total misses, blind to rank shifts.
  • MRR answers 'how high is the first hit', sensitive to ranking degradation, blind to how many hits there are.
  • nDCG answers 'how good is the top k overall', the direct target for reranking, blind to existence.
  • Only the combination localizes the failure to retrieval, ranking, or noise.
  • State the hit criterion: context is packed against a token budget, not a fixed top-k.

  答题要点

  • 召回率管「有没有进上下文」，对完全没捞到最敏感，对名次变化不敏感。
  • 平均倒数排名管「第一条排第几」，对排序退化最敏感，但分不清命中一条还是五条。
  • 归一化折损累计增益管「前 k 名整体质量」，是重排的直接优化目标，但看不出有没有。
  • 三者组合才能定位故障在召回层、排序层还是噪声层。
  • 命中口径要说清：按 token 预算装上下文，不是按固定条数取前 k。

D11 Advanced Indexing: Parent-Child Documents, Summary Indexes, Contextual Retrieval, and the Trade-Offs of Tree Aggregation vs. Graph Retrieval

• You have built three different indexes over the same corpus. How do you decide which one a query goes to?同一份语料建了三套索引，检索时你怎么决定走哪一套？

  Common in ChinaCommon overseasIntermediate#index-routing#evaluation#architecture

  How to reason about it · think before answering

  1. Whether this is an easy point or a lost one depends on whether you first ask 'do we actually need three?'. Jumping straight to routing accepts an unverified premise.
  2. Step one is admitting the answer is usually 'none of them — use the default'. Across 30 documents we measured five index structures and every one landed at 93.8% recall, none beating the baseline. The only metric that moved was nDCG@10, which headers lifted from 0.6438 to 0.7218, while the two-stage summary index fell to 87.5%. Each structure patches one specific weakness; without that weakness it is pure overhead.
  3. Step two is routing, and the criterion is not 'which index is more accurate' — that is an offline evaluation question, not something you know at request time. What you do have at request time is the shape of the question: detail-seeking, summarizing, or entity-chaining. Those map onto the chunk index, the tree-summary index and the graph index.
  4. Implementation is a lightweight intent classifier — the same one from the previous day's intent routing, no need to invent another. Carry the decision as request metadata so you can replay it later.
  5. Spell out the fallback: on a misclassification, fall back to the default index rather than fanning out across all three and fusing. Fan-out looks safe but multiplies latency and cost by the number of indexes, and the extra routes usually never make it into the context budget anyway.
  6. Expect 'how do you know the classifier is right'. Log every routing decision and replay the golden set periodically: run each question through all three indexes and check whether the classifier picked the best-scoring one. It is a standing offline job that needs no human labelling.

  分析过程 · 先想清楚再作答

  1. 这题是送分还是丢分，取决于你有没有先反问一句『真的需要三套吗』。上来就答路由策略的人，默认了一个没被验证的前提。
  2. 第一步是承认多数情况下答案是『都不走，走默认那套』。我们在 30 篇语料上把五种索引结构各测一遍，**召回率全部停在 93.8%，没有一种跑赢基线**；唯一动了的是 nDCG@10（块头把它从 0.6438 抬到 0.7218），而两段式的摘要索引还掉到了 87.5%。每种结构补的都是一个特定短板，你没有那个短板时它只带来成本。
  3. 第二步才是路由，而判据不是『哪套准』——那是离线评估该回答的问题，不是运行时能知道的。运行时能拿到的只有**问题的形状**：细节型（答案落在某一段）、概括型（要全库的一个概括）、多跳型（要跨实体串联）。按形状分流，正好对应块级索引、树状聚合索引、图索引。
  4. 实现上就是一个轻量意图分类器，跟前一天的意图路由是同一套东西，不必再造一个。分类结果作为元数据带进请求，方便事后拿评估集回看分错了多少。
  5. 兜底策略要说清楚：分类错了**回落到默认那一套**，不要并行全查一遍再融合。并行看着稳，实际上把延迟和成本按索引套数翻倍，而多出来的那两路大概率一条都进不了上下文预算。
  6. 可预期的追问是『怎么知道分类器分对了』。答案是把路由决策记进日志，定期拿标准答案集回放：对每个问题分别走三套索引，看分类器选的那套是不是指标最好的那套。这是一个能持续跑的离线作业，不需要人工标注。

  Key points

  • First challenge the premise: all five index structures landed at the same 93.8% recall in our measurement, so an index without a matching weakness is pure cost.
  • At request time the usable signal is question shape — detail, summary, or entity-chaining — mapping to chunk, tree-summary and graph indexes.
  • Reuse the previous day's intent router for classification and record the routing decision as request metadata.
  • Fall back to the default index on misclassification instead of fanning out and fusing, which multiplies latency and cost.
  • Replay the golden set periodically to check whether the classifier picks the best-scoring index.

  答题要点

  • 先反问是否真需要三套：实测五种索引结构召回率全部持平在 93.8%，没有对应短板就是纯成本。
  • 运行时的判据是问题的形状——细节型、概括型、多跳型，分别对应块级、树状摘要、图索引。
  • 复用前一天的意图路由做分类，把路由决策记进请求元数据。
  • 分类错了回落到默认索引，不要并行全查再融合——延迟和成本按套数翻倍。
  • 用标准答案集定期回放，检验分类器选的那套是不是指标最好的那套。

D14 Capstone Project and Retrospective: A Multi-Tenant Enterprise Knowledge-Base Q&A, a RAG Decision Map, and an Interview Deep Dive

• If you could only fund three changes to improve an existing RAG system, which three would you pick and why those three?如果预算只够做三件事来提升一个已有 RAG 系统的效果，你选哪三件？为什么是这三件？

  Common in ChinaCommon overseasIntermediate#prioritization#evaluation#abstention

  How to reason about it · think before answering

  1. This tests prioritisation, not breadth. Answering with a list of techniques — add reranking, add hybrid retrieval, add query rewriting — almost always loses points, because it skips a prerequisite: how do you know those three help your system? That is precisely the sentence the interviewer is waiting for.
  2. So the first item has to be building evaluation, with a reason specific enough to be unarguable: without a scale, you cannot tell whether the other two helped or hurt; with one, every subsequent spend has a measurable return. It is also cheap — the three retrieval metrics are pure local computation, run in seconds, cost nothing, and can gate every commit; the only real effort is labelling answer documents once. Include the composition rule: multi-hop and unanswerable each above ten percent, because without the unanswerable class a system that only ever guesses scores perfectly on your report.
  3. Second, move abstention out of the prompt and into code — usually the best return per unit of effort, and the item most often skipped. Writing 'say you don't know' ten times in a prompt buys almost nothing. Citation numbers are a closed set, so checking existence is one line, and adding a substantive-overlap check catches the harder forgery where the number is real but the content is not. Our baseline abstention rate was 0.0 percent: four questions with no answer in the corpus, zero of them declined — a defect that is completely invisible on a report that only shows recall.
  4. Third, look at the failure cases before deciding, which is the actual answer to this question. After reading the panel you land on one of a few branches: a high share of multi-hop means bridging retrieval or a different index structure; queries that miss when phrased differently mean you need the vector route or hybrid retrieval; answers retrieved but never packed into context means reranking or budget. Failure cases first, technique second — we tried five advanced index structures and not one beat the baseline, because our system simply did not have the weakness they address.
  5. Why not the flashier options: agentic retrieval concentrates its gains on multi-hop while spreading cost across every question, and in our measurements turning on every query-side technique produced exactly the same recall as the default configuration while using 2.5 times the model calls and 4.3 times the retrievals. Stacking techniques is easy; explaining why you switched several off is the skill.
  6. Expected follow-up: once the three are done, how do you prove the money was well spent? Toggle each one individually and report three ledgers — how much the metric moved, how much latency moved, how much cost moved. A proposal that reports only the first should not be approved, including your own.

  分析过程 · 先想清楚再作答

  1. 这题在考优先级判断，而不是知识面。答成「上重排、上混合检索、上查询改写」这类手法清单几乎必然掉分——因为它跳过了一个前提：**你凭什么知道这三件对你的系统有用？** 面试官等的就是这句话。
  2. 所以第一件必须是**建评估**，而且理由要具体到不可反驳：没有秤，剩下两件做完你也说不清是变好还是变坏；有了秤，后面每一笔钱都能算回报。而且它便宜——检索侧三个指标是纯本地计算、几秒钟、零成本，能挂进每次提交；花时间的只是给题目标答案文档那一次。顺带说清评估集的配比：多跳与无答案各占一成以上，缺了无答案那一类，一个只会硬答的系统在报表上就是满分。
  3. 第二件是**把拒答从提示词搬进代码**，这一件的性价比通常最高而最容易被跳过。提示词里写十遍「找不到就说找不到」增益接近于零；而引用编号是一个闭集，判它存不存在只要一行代码，再加一道「这句话与被引块的实质重合度」就能拦住「编号是真的、内容是假的」那一类。我们实验里的基线拒答率是 0.0%——四道语料里根本没有答案的题一道都没闭嘴，这类缺陷在只报召回率的报表上完全不可见。
  4. 第三件要**先看失败案例再决定**，这才是这道题真正的答案。看完面板你会落到其中之一：多跳题占比高就补桥接检索或改索引结构；换个说法就捞不到，说明该上向量那一路或混合检索；答案捞到了却排不进上下文，那是重排或者预算的活。**先有失败案例，再有手法**——我们试过五种高级索引结构，没有一种跑赢基线，因为我们的系统压根没有那些结构要补的短板。
  5. 为什么不选那些看起来更亮的：Agentic 检索的收益集中在多跳题上而代价摊给全部问题；「全开」所有查询侧手法在我们的实测里召回率和默认配置一模一样，模型调用却是 2.5 倍、检索次数 4.3 倍。**堆手法很容易，说清楚为什么关掉某几项才是本事。**
  6. 可预期的追问：三件做完怎么证明钱花对了？答：每一项单独开关各跑一遍，报三笔账——指标涨了多少、延迟涨了多少、钱涨了多少。只报第一笔的提案不该被批准，包括你自己的。

  Key points

  • First, build evaluation: without a scale the other two changes are unverifiable, and the retrieval metrics are cheap enough to gate every commit.
  • The golden set must include unanswerable questions, or a system that only ever guesses scores perfectly on your report.
  • Second, move abstention from the prompt into code: citation numbers are a closed set, and a substantive-overlap check catches real-number-fake-content forgeries.
  • Third is chosen by the failure cases, not by a list of techniques — failure cases first, index structure or retrieval trick second.
  • Toggle each change individually and report three ledgers: metric, latency, cost. A proposal reporting only the first should not be approved.

  答题要点

  • 第一件是建评估：没有秤，另外两件做完也说不清变好还是变坏；检索侧指标零成本可挂进每次提交。
  • 评估集必须含无答案那一类，否则一个只会硬答的系统在报表上就是满分。
  • 第二件是把拒答从提示词搬进代码：编号是闭集，再加实质重合度就能拦住「编号真、内容假」。
  • 第三件由失败案例决定，不由手法清单决定——先有失败案例，再有索引结构或检索手法。
  • 每一项单独开关跑一遍并报三笔账：指标、延迟、钱。只报第一笔的提案不该被批准。

Build an AI Short-Drama Production Pipeline With Agents in 14 Days

D11 Quality Control and Compliance: Machine Review, Content Safety, Generated-Content Labeling, and Copyright Boundaries

• How do you turn a subjective judgement like visual quality into an automatable check?怎么把画面质量这种主观判断变成可自动判定的检查？

  Common in ChinaCommon overseasIntermediate#quality-check#evaluation#multimodal

  How to reason about it · think before answering

  1. This tests decomposition. Answering just use a multimodal model to score it covers only the lazy half; the interviewer wants to see you turn a non-falsifiable statement into checkable ones.
  2. Step one is classification: split checks into locally measurable and must-be-seen-by-a-model. Resolution, audio-video duration delta, subtitle length and reading rate, and loudness all have deterministic answers from ffprobe plus arithmetic. Character consistency and visual breakdown have no reliable local proxy.
  3. The classification pays off in accounting: a failing objective check means the file really is wrong, while a failing subjective one might just mean the model misread. Merge them into one score and you cannot tell whether to fix the file or the prompt.
  4. Step two gives every check three things: what is measured, the threshold, and the corrective action. The third is the one people skip and the one that matters, because a check that fails without a prescribed fix is decoration.
  5. Step three handles model-side uncertainty: demand a structured verdict, and when it cannot be parsed mark the item as no-conclusion, needs-human, never as a pass. Conflating the model said fine with the model did not answer is the classic automated-QC incident.
  6. Expected follow-up: how to set thresholds. Backtest against human-reviewed samples and pick the threshold where machine and human verdicts agree most. Without that data you are guessing.

  分析过程 · 先想清楚再作答

  1. 这题在考拆解能力。直接答「让多模态模型打分」只答了一半，而且是偷懒的那一半——面试官想看你怎么把一个不可判真假的命题拆成可判定的。
  2. 第一步是分类：把检查项分成「本地量得出来的」和「必须让模型看图的」。分辨率、音画时长差、字幕字数与每秒字数、配音响度，这四类用 ffprobe 加几行算术就有确定答案；角色一致性、画面崩坏则本地没有可靠代理指标。
  3. 分类的价值是账算得清：客观项出问题一定是文件真有毛病，主观项出问题可能是模型看错了。混成一个总分，事故来的时候分不清该修文件还是修提示词。
  4. 第二步是给每一项配齐三样：测量对象、阈值、**修正动作**。第三样最容易漏也最关键——一项检查不合格却说不出该怎么办，它就是摆设，你只能记一行日志继续往下走。
  5. 第三步是处理模型那一侧的不确定性：要求它只返回结构化结论，并且**解析不出来时标成无结论、需人工，绝不当成通过**。把「模型说没问题」和「模型没答上来」混为一谈，是自动质检里最常见的事故。
  6. 可预期的追问是「阈值怎么定」。用人工审核攒下来的带结论的样本回测，看阈值定在几分时机器结论与人的重合度最高；没有这份数据就只能拍脑袋。

  Key points

  • Classify first: objective local measurements (resolution, av delta, subtitle density, loudness) versus model-only judgements (character consistency, visual breakdown).
  • Give every check a measurement, a threshold and a corrective action; a check with no action is decoration.
  • Require a structured verdict from the model, and treat unparseable output as needs-human, never as a pass.
  • Set thresholds by backtesting against human-reviewed samples.
  • An objective failure means the file is wrong; a subjective failure may mean the model misread. That split drives triage.

  答题要点

  • 先分类：本地量得出来的客观项（分辨率、音画差、字幕密度、响度）与必须看图的主观项（角色一致性、画面崩坏）分开记账。
  • 每一项配齐三样：测量对象、阈值、修正动作；没有修正动作的检查项是摆设。
  • 模型评审要求返回结构化结论，解析失败标成需人工，绝不默认通过。
  • 阈值靠人工审核样本回测确定，不拍脑袋。
  • 客观项失败说明文件有问题，主观项失败可能是模型看错——这个区分决定了排查方向。