Interview Bank

RAG in 14 Days: From Retrieval to Trustworthy Answers

D13 Going to Production: Incremental Sync and Deduplication, Permission-Based Filtering, Cache Layering, Tracing, and the Cost-Latency Ledger

• After a document changes, how do you recompute only the affected chunks? And how do you guarantee a deleted document really disappears from the index?文档更新之后，你怎么做到只重算受影响的块？被删掉的文档又怎么保证一定从索引里消失？

  Common in ChinaCommon overseasIntermediate#incremental-sync#content-hash#index-maintenance

  How to reason about it · think before answering

  1. There are two halves here and the second one separates candidates. Almost everyone can say 'hash it and compare'; the score comes from bringing up deletion yourself, because it is the one asymmetric case in the whole mechanism.
  2. Give the skeleton first: a three-way reconciliation between the full set from the source and the full set in the index. In source but not indexed is an add; in both but with different content hashes is a modify; indexed but absent from the source is a delete. A modify must replace the document wholesale, deleting old chunks before writing new ones, otherwise a shortened document leaves a tail behind in the index.
  3. Then the fingerprint itself, which is where points are won: sha256 truncated, but normalize line endings and trim before hashing. The same file uploaded from Windows and from macOS differs byte-wise but not in content; skip normalization and every re-upload counts as a change, which is a full rebuild in disguise. It never raises an error, it only shows up on the bill.
  4. The key insight in the second half: a deletion is not an event, it is an absence. Change feeds tell you what changed; nobody ever sends 'I no longer exist'. So deletion detection has to run in the opposite direction — walk the index and find ids the source no longer has. A synchronizer that only listens to change events will wait forever.
  5. At the storage layer, cascade the foreign keys across documents, chunks and embeddings so deleting a document is a single statement and the database does the rest. Hand-written three-step deletes eventually miss one, and the one they miss is a ghost in the index. Close with a verifiable invariant: chunk count must equal embedding count, and a mismatch means orphans.
  6. Expected follow-up: what if the source system itself is unreliable and a pull comes back incomplete? Make pull completeness a precondition for deletion: on a partial pull, apply adds and modifies only, or one failed fetch wipes half your index. Also soft-delete with a retention window so a mistake is recoverable.

  分析过程 · 先想清楚再作答

  1. 这题有两半，区分度全在后半。前半几乎人人答得出「算个哈希比一比」，能不能拿到分取决于你有没有主动讲删除——那是同一套机制里唯一不对称的一种变更。
  2. 先给增量的骨架：拿来源的全集和索引的全集做三向对账。来源有、索引没有是新增；两边都有但内容指纹不同是修改；索引有、来源没有是删除。修改的处理是整篇替换，先删旧块再写新块，不能只追加——不然改短了的文档会在索引里留下一截尾巴。
  3. 接着讲指纹本身，这是给分点：sha256 取前若干位，但**算之前必须先做换行归一化再去首尾空白**。同一份文件从 Windows 传一次、从 Mac 传一次，字节不同内容相同，不归一化就每次都判成变了，等于天天在做全量重建。这个 bug 不报错，只体现在账单上。
  4. 然后是删除这一半的关键判断：**删除不是一个事件，是一个缺席**。文件变动类的通知只告诉你哪些东西变了，永远不会有人发一条「我不存在了」。所以删除检测必须反着来——遍历索引，找出来源里已经没有的 id。只监听变更事件的同步器永远等不到这条消息。
  5. 落到存储上：文档、块、向量三张表用外键级联删除，删文档只写一条语句，剩下的交给数据库。手写三条删除的版本迟早会漏掉一条，而漏掉的那条就是索引里的幽灵。收尾时报一个可验证的指标：块数与向量数必须相等，不等就说明有孤儿。
  6. 可预期的追问：来源系统本身就不可靠、拉不全怎么办？那就把「本次拉取是否完整」当成删除检测的前置条件——拉取不完整时只做新增和修改，不做删除，否则一次拉取失败会把半个索引清空。另外给删除加软删标记和保留期，误删还能回滚。

  Key points

  • Three-way reconciliation covering adds, modifies and deletes; a modify replaces the whole document, old chunks first.
  • Normalize line endings and trim before hashing, or cross-platform re-uploads look like edits and you are doing a full rebuild every night.
  • Deletion is an absence, not an event: walk the index for ids the source no longer has instead of waiting on a change feed.
  • Cascade deletes from documents to chunks to embeddings so one statement suffices; assert chunk count equals embedding count to catch orphans.
  • On an incomplete pull, apply adds and modifies only, and soft-delete with a retention window so mistakes are reversible.

  答题要点

  • 三向对账：新增、修改、删除，缺一不可；修改是整篇替换，先删旧块再写新块。
  • 内容指纹算之前必须先做换行归一化再 trim，否则跨系统重传会被误判为修改，等于天天全量重建。
  • 删除是缺席不是事件，必须反过来遍历索引找出来源里已消失的 id，不能只监听变更通知。
  • 文档、块、向量用外键级联删除，删文档只写一条语句；用「块数等于向量数」当可验证的收尾指标。
  • 来源拉取不完整时只做新增与修改、跳过删除，并给删除加软删与保留期以便回滚。