Interview Bank

RAG in 14 Days: From Retrieval to Trustworthy Answers

D2 Embeddings and Vector Search: Similarity, Dimensionality, and Model Choice; Storing Text in pgvector

• What do you lose when you cut embedding dimensions from 1536 to 512, and when is that loss acceptable?把 embedding 维度从 1536 降到 512，你会损失什么？什么场景下这个损失可以接受？

  Common in ChinaCommon overseasIntermediate#embeddings#dimensions#cost

  How to reason about it · think before answering

  1. This is a cost-modelling question. 'Lower dimensions are cheaper but less accurate' earns nothing; the interviewer wants a cost model and a decision order.
  2. Lay out three costs: storage and memory (vector count times dimensions times bytes per dimension, which an ANN index must hold in RAM), query latency (roughly linear in dimensions), and retrieval quality, whose returns diminish sharply at the high end.
  3. Explain why truncation works at all: models trained with Matryoshka representations pack the most important information into the leading dimensions, so truncating and re-normalising keeps the vector usable. It is still lossy, and how lossy is an empirical question on your own data.
  4. Give the decision order: derive a dimension ceiling from your memory budget, then step down two or three notches and measure the metric drop. Choosing the largest model first and optimising cost later usually means redoing the work.
  5. Name the acceptable cases: large corpora of low individual value, pipelines where a reranker recovers some of the loss, and latency-critical online paths. Be conservative where a single miss is expensive, such as legal or clinical retrieval.
  6. Expected follow-up: can different documents use different dimensions? No. Every vector in an index must share one dimension, so changing it means rebuilding the whole index, the same migration cost as changing models.

  分析过程 · 先想清楚再作答

  1. 这题考的是你会不会算账。只说「维度越低越省、精度越低」的答案没有区分度，面试官在等一个具体的成本模型和一个决策顺序。
  2. 先把三笔账列出来：存储与内存（向量数量乘维度乘每维字节数，近似最近邻索引要把它放进内存，所以基本等于机器预算）、检索延迟（每次比较就是一轮乘加，维度大致线性影响耗时）、检索质量（收益递减，低维段每加一档提升明显，高维段加倍只换来很小的改善）。
  3. 再说清降维为什么可行：主流模型用套娃式表示训练，重要信息压在靠前的维度上，所以直接截短再归一化仍然可用，这不是另训了一个小模型。截短必然有损失，损失多少只能在自己的数据上跑评估才知道。
  4. 给出决策顺序：先按存储与内存预算倒推一个维度上限，再从上限往下试两三档，看指标掉多少，掉得能接受就用低的。反过来「先选最高维再想办法省钱」基本都会返工。
  5. 点出可接受的典型场景：库很大而单条价值不高（比如日志、工单）、召回之后还有重排兜底（重排能把粗排的损失补回来一部分）、或者对延迟极敏感的在线场景。反过来法务、医疗这类一条都不能漏的场景就要谨慎。
  6. 可预期的追问：能不能不同文档用不同维度？不能——同一个索引里所有向量必须同维，改维度等于全库重建，这跟换模型是同一类迁移成本。

  Key points

  • Three costs: storage and index memory, query latency, and retrieval quality; the first two scale with dimensions, the third has diminishing returns.
  • Matryoshka representations make truncation viable, but it is lossy and the loss must be measured on your own data.
  • Decide by deriving a ceiling from the memory budget, then stepping down and measuring.
  • Truncation pays off for large corpora, low-value items, latency-sensitive paths, and pipelines with a reranker.
  • All vectors in one index share a dimension, so changing it forces a full rebuild.

  答题要点

  • 三笔账：存储与索引内存、检索延迟、检索质量，前两笔随维度近似线性，第三笔收益递减。
  • 套娃式表示让截短再归一化仍然可用，但一定有损失，损失多少要在自己的数据上评估。
  • 决策顺序是先按内存预算定上限，再往下试档位看指标掉多少。
  • 库大、单条价值低、后面还有重排兜底、对延迟敏感的场景，降维划算。
  • 同一索引里维度必须一致，改维度等于全库重建。