Interview Bank

RAG in 14 Days: From Retrieval to Trustworthy Answers

D13 Going to Production: Incremental Sync and Deduplication, Permission-Based Filtering, Cache Layering, Tracing, and the Cost-Latency Ledger

• You need to switch embedding models. How do you migrate a live system without downtime and without losing recall?要换一个 embedding 模型，线上系统怎么迁移才能不停机也不掉召回？

  Common in ChinaCommon overseasDeep dive#embedding-migration#zero-downtime#rollout

  How to reason about it · think before answering

  1. The crux is why you cannot swap in place. Jumping straight to the steps without establishing that reads like reciting a runbook.
  2. Set up the premise: vectors from different models are not comparable. Dimensions may differ, and even at equal dimensions the coordinate spaces are unrelated, so encoding the query with the new model and comparing against documents encoded with the old one yields noise. Switching models therefore means re-embedding the entire corpus.
  3. Then the four steps: add a nullable second vector column; backfill it with a background job while the old column is untouched and still serves live traffic; canary a slice of traffic onto the new column while running the golden set against both columns to compare recall and faithfulness; cut over fully once the numbers hold, and drop the old column only after a week or two of observation.
  4. Name the payoff explicitly, because this is where the points are: the value of the whole procedure is the rollback cost. Cutover is a config change naming which column to read, so reverting takes a second rather than re-running an eight-hour rebuild. A migration plan with no rollback path is not a plan.
  5. Add two engineering details: build the approximate-nearest-neighbour index on the new column after the backfill, not during it, since concurrent building is slow and prone to locking; and make the backfill resumable and rate-limited, or it will exhaust the embedding API quota and drag live queries down with it.
  6. Expected follow-up: how do you prove the new model is actually better? Not from an offline metric alone — run an A/B on the same golden set with identical retrieval parameters and report four numbers: recall, faithfulness, latency and cost. A conclusion resting on the first number only does not hold. Note also that switching models is the one moment when the embedding cache genuinely must be invalidated.

  分析过程 · 先想清楚再作答

  1. 这题的题眼是「为什么不能就地换」。没有先说清这一点就直接讲步骤，会显得是在背流程。
  2. 先给前提：不同模型的向量之间**没有可比性**。维度可能不同，即使维度相同坐标系也完全不是一回事，用新模型编码问题去和旧模型编码的文档比距离，算出来的相似度是纯噪声。所以「换模型」实质上等于「把整个知识库重新向量化一遍」。
  3. 然后给四步：加一列新向量、允许为空；后台任务慢慢回填新列，旧列一个字节不动，线上仍走旧列；小流量灰度到新列，同时用标准答案集在两列上各跑一遍比召回率与忠实度；数字站得住再全量切换，旧列观察一两周后才删。
  4. 把这套流程的价值点破，这是给分点：**它的价值全在回滚成本上**。切换只是改一个配置项「走哪一列」，出问题时切回去是一秒钟的事，而不是重跑一遍八小时的重建任务。凡是拿不出回滚路径的迁移方案都不算方案。
  5. 补两个工程细节：新列的近似最近邻索引要在回填完之后再建，边写边建又慢又容易锁表；回填要能断点续传并限速，否则会把 embedding 接口的配额打满，把线上查询一起拖垮。
  6. 可预期的追问：怎么证明新模型确实更好？答：不能只看离线指标涨没涨，要在同一份标准答案集、同一套检索参数下跑 A/B，报召回率、忠实度、延迟、花费四笔账；只报第一笔的结论不成立。另外注意换模型会让缓存里的向量全部作废，那是这次迁移唯一该作废向量缓存的时刻。

  Key points

  • Vectors from different models are not comparable, so a model switch is equivalent to re-embedding the entire corpus.
  • Four steps: add a nullable second vector column, backfill in the background, canary with the golden set scored on both columns, then cut over once the numbers hold.
  • The whole value lies in rollback cost: cutover is a config change, so reverting takes a second instead of another full rebuild.
  • Build the ANN index on the new column after the backfill; make the backfill resumable and rate-limited so it does not exhaust the embedding quota and stall live queries.
  • Validate with an A/B on one golden set reporting recall, faithfulness, latency and cost; a model switch is also the only time the embedding cache truly must be invalidated.

  答题要点

  • 不同模型的向量之间没有可比性，所以换模型等价于把整个知识库重新向量化一遍。
  • 四步：加一列可空的新向量、后台回填、小流量灰度并用标准答案集在两列上对比、数字站得住再全量切换。
  • 这套流程的价值全在回滚成本上：切换是改一个配置项，回滚是一秒钟的事而不是重跑一次重建。
  • 新列的近似最近邻索引在回填完成后再建；回填要可断点续传并限速，别把接口配额打满拖垮线上查询。
  • 验证要在同一份标准答案集上跑 A/B，同时报召回率、忠实度、延迟与花费四笔账；换模型也是唯一该作废向量缓存的时刻。