Interview Bank

How to reason about it · think before answering

1. Three questions, three layers: mechanism, boundaries, debugging. The third layer is what shows production experience.
2. Mechanism: the cache matches the exact byte prefix from the start of the request to the cache_control marker (tools, then system, then messages). A hit bills that prefix at 0.1x input price; the write costs 1.25x (2x for the one-hour TTL).
3. When it costs more: a prefix used only once (+25%); volatile content inside the prefix — timestamps, random ids, unsorted JSON, user names — so every call writes a cache nothing will read; a prefix below the minimum (1024 tokens on current flagship models, 4096 on Haiku 4.5) that silently never caches; requests spaced beyond the TTL.
4. Debug order by likelihood: dynamic content at the head of system or tool definitions; model id mismatch between calls; prefix under the minimum; gap over five minutes; unstable tool ordering. The single signal is usage.cache_read_input_tokens greater than zero.
5. Follow-up: where do breakpoints go? At the end of stable sections — tools, system, the long document, the second-to-last message in a multi-turn chat — at most four; a breakpoint on per-turn content is a wasted write.

分析过程 · 先想清楚再作答

1. 三问对应三层：原理、边界、排查。只答第一层是背文档，第三层才体现有没有真的上过线。
2. 原理一句话：缓存匹配的是请求开头到 cache_control 标记为止的精确前缀（顺序是工具、system、messages），命中时这段只收正常输入价的 0.1 倍；代价是写入那一次收 1.25 倍（1 小时档 2 倍）。
3. 不省的情况由此推出：同一前缀只用一次（多付 25%）；前缀里有每次都变的内容（时间戳、随机 id、未排序 JSON、用户名），导致每次都在写永远用不上的缓存；前缀短于最小门槛（主力模型 1024 token，Haiku 4.5 是 4096）根本不会缓存；两次请求间隔超过 TTL。
4. 排查清单按发生概率排：一看 system 或工具定义开头有没有动态内容；二看两次请求的模型 id 是否一致；三看前缀长度是否过门槛；四看间隔是否超 5 分钟；五看工具列表顺序是否稳定。判据只有一个字段：usage.cache_read_input_tokens 是否大于 0。
5. 可预期的追问：断点应该打在哪？不变的末尾——工具定义末尾、system 末尾、长文档末尾、多轮对话倒数第二条消息，最多四个；打在每轮都变的内容上等于白写。

How to reason about it · think before answering

1. The first trap is the phrase one task. Many people quote a single request's input size, which is a weight reading, not a bill. Stateless models resend everything each turn, so the bill is the sum of every turn's input.
2. Sum the four buckets by their growth patterns. The stable prefix (system prompt plus tool definitions) is resent verbatim, so multiply by turn count. History grows linearly, so it is an arithmetic series. Tool results grow in steps, so estimate calls times size. For scale: a twenty-turn support task whose final request is 11256 tokens totals 133502 across the session, nearly twelve times larger.
3. Then caching. The cacheable part is the stable prefix, ordered tools, system, messages, where editing anything earlier invalidates everything after. Writes cost about 1.25 times base (about 2 times for a one-hour lifetime) and hits about 0.1 times, so twenty full-price prefixes become one write plus nineteen hits, an eighty percent saving.
4. State the threshold: the prefix must reach the model's minimum cacheable length or caching silently does nothing. That produces the counterintuitive result where halving your system prompt lowers token count but raises the bill, because the prefix fell below the threshold.
5. Expect the follow-up on whether to trim anyway. Yes, but report two numbers: the raw token reduction and the cache-adjusted effective reduction, and check whether the prefix crossed the threshold. If it did, add stable reference content back into the prefix or move to a model with a lower threshold.

分析过程 · 先想清楚再作答

1. 这题的第一个坑在「一次任务」四个字。很多人报的是单次请求的输入量，那是称重不是账单——模型没有记忆，每一轮都要把前面全部重发，账单是整场会话每轮输入的累加值。
2. 怎么拆：按四块各自的增长方式分别求和。稳定前缀（系统提示加工具定义）每轮原样重发，乘轮数；对话历史线性增长，是等差数列求和；工具结果阶梯增长，按调用次数与每次体积估。举个量级：一个 20 轮的客服任务，最后一轮单次输入 11256，整场累加是 133502，差了将近 12 倍。
3. 再谈缓存。可缓存的是稳定前缀这一段，顺序是工具定义、系统提示、消息，改前面的会让后面全部失效。经济学是写入约 1.25 倍原价（一小时存活期约 2 倍）、命中约 0.1 倍，所以 20 轮的前缀从 20 次全价变成一次写入加十九次命中，能便宜八成以上。
4. 结论要带上那条门槛：前缀必须达到模型的最小可缓存长度才生效，达不到既不报错也不告警。这直接导致一个反直觉现象——把系统提示精简掉一半，token 数降了，账单反而可能涨，因为前缀掉到门槛以下、缓存静默失效。
5. 可预期的追问：那还该不该精简？该，但要同时报两个数——不含缓存的 token 降幅与含缓存的等效开销降幅，并检查前缀有没有跨过门槛。跨过了就把稳定的引用内容放回前缀抬回去，或者换一个门槛更低的模型。

How to reason about it · think before answering

1. This checks whether you have actually done the arithmetic. Saying 'prompt caching makes it cheap' without knowing which line item it touches is a tell.
2. Split the bill first: one-off = per-chunk input + output + full re-embedding; per-query = the header read twice, once by the reranker and once in the context. Keep them separate, because they scale with completely different things.
3. The dominant term on the one-off side is how many times the same document is re-read. A doc split into n chunks is read n times. Prompt caching attacks exactly that: put the whole document first and mark it cacheable, pay a cache write once, then cache reads for the remaining n-1, typically an order of magnitude cheaper than input.
4. Order matters. Caching is prefix-matched, so the document must come first and the chunk after. Put the varying part first and the prefix changes every call — zero cache hits. This is the most common way people get it wrong.
5. Our measurement: 30 docs, 134 chunks. Without caching, 103017 input tokens; with caching, 17340 written plus 60137 read, cutting the one-off cost by roughly 29%. The finer the chunks, the bigger the saving, because re-reads multiply.
6. The counter-intuitive part is the useful part: the one-off cost amortizes below 10% of per-query cost after about 217 queries. The lasting bill is the extra tokens every query carries (we measured +12.3%). So the first lever is not cheaper index building — it is keeping the header out of the context, keeping it short, and not generating it for the whole corpus indiscriminately.
7. A bonus point: before spending any of it, confirm your evaluation setup can actually detect the benefit. In our offline harness the vector route contributed exactly zero unique answer documents, so it cannot answer whether headers help embeddings at all — an A/B run there hands you a wrong conclusion that looks numerically supported.

分析过程 · 先想清楚再作答

1. 这题考的是你有没有真的算过账。只会说『用提示词缓存就便宜了』属于听过没做过——面试官会追问缓存到底省在哪一项上。
2. 先把成本拆开：一次性 = 每块的输入 + 输出 + 全量 embedding；每次查询 = 块头在重排和上下文里各被读一遍。**这两笔要分开记**，因为它们随业务量的增长方式完全不同。
3. 一次性那笔的主项是『同一篇文档被重复读了多少遍』。一篇切成 n 块就要读 n 遍，这是成本的大头。提示词缓存省的正是这一项：把整篇放在提示词最前面并标记为可缓存，第一块付一次缓存写入，后面 n-1 块只付缓存读取，而读取价通常比输入价低一个数量级。
4. 顺序不能反：缓存按前缀匹配，整篇必须在前、块内容在后。把变化的块放前面，前缀次次都变，缓存一次都不会命中——这是最常见的翻车点。
5. 我们的实测：30 篇、134 块，不开缓存输入 103017 token，开缓存后拆成写入 17340 加读取 60137，一次性成本降约 29%。**块切得越碎这个比例越高**，因为重复读的次数更多。
6. 结论反直觉但很实用：一次性那笔是小钱，摊到 217 次查询就降到每次查询成本的一成以下；真正的长期账是每次查询多出来的那几十个 token（我们量到 +12.3%）。所以压成本的第一优先级不是压建索引，而是让块头别进上下文、别过长、别对全库无差别地生成。
7. 最后一条是加分项：花这笔钱之前先确认你的评估环境**测得出**收益。我们的离线环境里向量路对召回的独立贡献实测为 0，所以它根本没法回答『块头对向量侧有没有用』——在这种环境里做的 A/B 会给你一个看起来有数字支撑的错误结论。