Interview Bank

From Frontend Engineer to Agent Engineer in 30 Days

D4 Model Integration and System Prompts: a Multi-Provider Abstraction With Fallback, Overriding the Default Persona (dg P03/P04/M04)

• How do you pick the right model per task, balancing cost against latency?如何在成本和延迟之间给不同任务选择合适的模型？

  Common in ChinaCommon overseasIntermediate#model-routing#cost#latency

  How to reason about it · think before answering

  1. This question tests whether you have ever spent your own money. 'Use the best model' is the worst answer; 'it depends' is too vague — give actionable routing dimensions.
  2. Establish the core fact: model pricing spans 50x or more, and much of your workload does not need the strongest model. Using a flagship for intent detection is driving a sports car to fetch a parcel downstairs.
  3. Give three routing dimensions: task type (classification and extraction go cheap, long-form reasoning goes strong), latency requirement (foreground users need low latency, background batches can be slow and cheap), and input length (only some models handle very long context, and pricing rises steeply).
  4. Quantify it: 10k conversations a day at 2000 tokens each costs roughly 300 CNY/day on a flagship; routing the 60% of grunt work to a small model drops it to about 125 CNY/day, saving 60k+ CNY a year with no perceptible quality change.
  5. Volunteer the implementation trade-off: start with static tiers by task type. Dynamic routing that asks a model which model to use adds another model call, and the latency and cost may not pay for themselves — optimize once you have real data.
  6. Expect: how do you verify the cheaper tier did not hurt quality? A golden set — run both tiers over the same inputs and compare with human or LLM-as-judge scoring, so the decision rests on data rather than vibes.

  分析过程 · 先想清楚再作答

  1. 这题考的是「你有没有真的在花自己的钱」。答「用最好的模型」是最差的答案，答「按需选择」太空，要给出可执行的分档维度。
  2. 先建立核心事实：不同模型的价格能差 50 倍以上，而你的任务里很大一部分根本不需要最强的模型。用旗舰模型做意图识别，等于开跑车去楼下取快递。
  3. 然后给出三个可操作的路由维度：任务类型（分类抽取走便宜模型，长文推理走强模型）、延迟要求（前台用户在等就走低延迟，后台批处理可以慢而便宜）、输入长度（超长上下文只有部分模型支持且价格陡增）。
  4. 结论要落到数字上才有说服力：1 万轮对话每轮 2000 token，全走旗舰约 300 元一天；把六成粗活改走小模型后降到 125 元左右，一年省六万多，用户感知不到差别。
  5. 还要主动说出实现上的取舍：先按任务类型静态分档，不要一上来就做「让模型判断该用哪个模型」的动态路由——那个方案本身又要多一次模型调用，延迟和成本可能得不偿失，等有真实数据再优化。
  6. 可以预期的追问：怎么验证降档没有损失质量？答案是准备 golden set，对同一批输入跑两档模型，用人工或 LLM-as-judge 比对准确率，把降档决策建立在数据上而不是感觉上。

  Key points

  • Model pricing spans 50x or more, so a flagship doing intent detection is obvious waste
  • Three routing dimensions: task type, latency requirement, and input length
  • Add a tier parameter to the call layer, pick the starting provider statically, and reuse the fallback chain
  • Prefer static tiers first — dynamic model-picks-model routing adds a call and may not pay off
  • Validate downgrades against a golden set rather than intuition

  答题要点

  • 不同模型价格能差 50 倍以上，用旗舰模型做意图识别是明显的浪费
  • 三个路由维度：任务类型（分类抽取 vs 推理生成）、延迟要求（前台 vs 后台）、输入长度（是否需要超长上下文）
  • 实现上给调用层加 tier 参数，按任务静态分档挑起始 provider，fallback 逻辑完全复用
  • 先静态分档再考虑动态路由，让模型判断该用哪个模型本身要多一次调用，可能得不偿失
  • 用 golden set 对比两档模型的准确率，把降档决策建立在数据上

D24 RAG, Level Up: Hybrid Search, Reranking, Citations, Recall Evaluation

• How is reranking usually implemented, what problem does it solve, and what does it cost?重排（rerank）一般怎么实现？它解决了初步检索的什么问题，代价是什么？

  Common in ChinaCommon overseasIntermediate#rag#rerank#latency

  How to reason about it · think before answering

  1. The lazy answer is 'sort again, more accurately'. What the interviewer wants is why the first pass cannot rank well, and why reranking cannot run over the whole corpus.
  2. Decompose: the first pass ranks by retrieval signals — cosine distance or term statistics — which are designed to scan millions of items fast, and coarseness is the price. Reranking changes the algorithm: query and candidate go into one model together (a cross-encoder), which is far more accurate but costs one forward pass per candidate. Hence it must sit behind a wide recall stage.
  3. Distinguish two implementations. For teaching or prototypes, batch-score with an LLM (0-10 for 40 candidates in one call). Production uses a trained cross-encoder reranker. Name the cost: an extra 100-300 ms hop plus an inference box — it is not a per-token API, it consumes capacity.
  4. Framing it as a funnel is clearest: recall sets the ceiling, reranking decides whether what is under the ceiling reaches the top five. Measured: adding the keyword channel lifts recall@20 from 83% to 95%; adding reranking moves recall@20 only to 98%, but recall@5 jumps from 80% to 91% and MRR from 0.732 to 0.908.
  5. Expected follow-up 1: does reranking improve recall? No. It introduces no new candidates, so recall@20 is the wrong metric to judge it by.
  6. Expected follow-up 2: why not ship LLM scoring to production? Unpredictable latency, per-token cost, scores that drift with prompt wording, and no clean path to offline distillation.

  分析过程 · 先想清楚再作答

  1. 这题最容易答成「再排一次序，更准」。面试官想听的是**为什么第一轮不能直接排准**，以及**为什么重排不能对全库做**。
  2. 怎么拆：第一轮的排序依据是「检索信号」——余弦距离或词频统计，它们是为了能在百万条里快速筛选而设计的，代价就是粗。重排换了一种算法：把 query 和候选**拼在一起**送进同一个模型算相关度（cross-encoder），精度高得多，但复杂度是每条候选一次前向，没法对全库做。所以它必须跟在一个宽召回后面。
  3. 结论要区分两种实现：教学 / 原型可以用 LLM 批量打分（一次调用给 40 条打 0 到 10 分），生产用专门训练的 cross-encoder 重排模型。**代价说清楚：多一次 100 到 300 毫秒的调用，外加一台推理机器**——它不是按 token 计费的 API，是要占资源的。
  4. 把它放进漏斗里说最清楚：召回决定天花板，重排决定天花板上的东西能不能排到前五。实测的样子是——加了关键词那一路，recall@20 从 83% 涨到 95%（天花板抬高）；再加重排，recall@20 只到 98%，但 recall@5 从 80% 跳到 91%、MRR 从 0.732 到 0.908。
  5. 可预期的追问一：重排能不能提高召回？不能。它不引入新候选，只重排已有的那批——所以看 recall@20 判断重排效果是错的指标。
  6. 可预期的追问二：为什么不用 LLM 打分上生产？延迟不可控、成本按 token 走、分数会随提示词措辞漂移，而且没法做批量离线蒸馏。

  Key points

  • The first pass ranks by retrieval signals so it can scan a large index fast; coarseness is the trade.
  • Reranking feeds query and candidate through one model together (cross-encoder): much sharper, but one forward pass per candidate, so only tens of items.
  • Batch LLM scoring works for teaching; production uses a dedicated reranker, costing an extra 100-300 ms hop plus an inference box.
  • Reranking does not raise recall — it raises recall@5 and MRR (measured 80% to 91%, 0.732 to 0.908) while recall@20 barely moves from 95% to 98%.
  • So judge a reranker by small-k metrics, never by recall@20.

  答题要点

  • 第一轮按检索信号粗排（余弦、词频），为的是能在大库里快速筛，代价是粗。
  • 重排把 query 和候选拼在一起过同一个模型（cross-encoder），精度高但每条一次前向，只能对几十条做。
  • 教学版可用 LLM 批量打 0 到 10 分；生产用专用重排模型，代价是多一次 100 到 300 毫秒的调用加一台推理机器。
  • 重排不提高召回，它提高的是 recall@5 与 MRR——实测 80% → 91%、0.732 → 0.908，而 recall@20 只从 95% 到 98%。
  • 所以判断重排效果要看前 k 小的指标，不要看 recall@20。