面试题库

5 天上下文工程

D5 度量与调优：token 账单、上下文利用率、失败模式排查与综合面试专题

• 怎么给一个 Agent 算一次任务的 token 账单？哪些部分是可以被缓存掉的？How do you compute the token bill for one agent task, and which parts can be cached away?

  国内高频海外高频深入#token-accounting#prompt-caching

  分析过程 · 先想清楚再作答

  1. 这题的第一个坑在「一次任务」四个字。很多人报的是单次请求的输入量，那是称重不是账单——模型没有记忆，每一轮都要把前面全部重发，账单是整场会话每轮输入的累加值。
  2. 怎么拆：按四块各自的增长方式分别求和。稳定前缀（系统提示加工具定义）每轮原样重发，乘轮数；对话历史线性增长，是等差数列求和；工具结果阶梯增长，按调用次数与每次体积估。举个量级：一个 20 轮的客服任务，最后一轮单次输入 11256，整场累加是 133502，差了将近 12 倍。
  3. 再谈缓存。可缓存的是稳定前缀这一段，顺序是工具定义、系统提示、消息，改前面的会让后面全部失效。经济学是写入约 1.25 倍原价（一小时存活期约 2 倍）、命中约 0.1 倍，所以 20 轮的前缀从 20 次全价变成一次写入加十九次命中，能便宜八成以上。
  4. 结论要带上那条门槛：前缀必须达到模型的最小可缓存长度才生效，达不到既不报错也不告警。这直接导致一个反直觉现象——把系统提示精简掉一半，token 数降了，账单反而可能涨，因为前缀掉到门槛以下、缓存静默失效。
  5. 可预期的追问：那还该不该精简？该，但要同时报两个数——不含缓存的 token 降幅与含缓存的等效开销降幅，并检查前缀有没有跨过门槛。跨过了就把稳定的引用内容放回前缀抬回去，或者换一个门槛更低的模型。

  How to reason about it · think before answering

  1. The first trap is the phrase one task. Many people quote a single request's input size, which is a weight reading, not a bill. Stateless models resend everything each turn, so the bill is the sum of every turn's input.
  2. Sum the four buckets by their growth patterns. The stable prefix (system prompt plus tool definitions) is resent verbatim, so multiply by turn count. History grows linearly, so it is an arithmetic series. Tool results grow in steps, so estimate calls times size. For scale: a twenty-turn support task whose final request is 11256 tokens totals 133502 across the session, nearly twelve times larger.
  3. Then caching. The cacheable part is the stable prefix, ordered tools, system, messages, where editing anything earlier invalidates everything after. Writes cost about 1.25 times base (about 2 times for a one-hour lifetime) and hits about 0.1 times, so twenty full-price prefixes become one write plus nineteen hits, an eighty percent saving.
  4. State the threshold: the prefix must reach the model's minimum cacheable length or caching silently does nothing. That produces the counterintuitive result where halving your system prompt lowers token count but raises the bill, because the prefix fell below the threshold.
  5. Expect the follow-up on whether to trim anyway. Yes, but report two numbers: the raw token reduction and the cache-adjusted effective reduction, and check whether the prefix crossed the threshold. If it did, add stable reference content back into the prefix or move to a model with a lower threshold.

  答题要点

  • 账单是整场会话每轮输入的累加值，不是最后一次请求的输入量。
  • 按四块的增长方式分别求和：前缀乘轮数、历史等差求和、工具结果按调用次数估。
  • 可缓存的是稳定前缀，顺序是工具定义、系统提示、消息，改前面会让后面全失效。
  • 写入约 1.25 倍、命中约 0.1 倍；但前缀必须达到最小可缓存长度，否则静默失效。
  • 所以精简可能让 token 降而账单涨，必须同时报含缓存与不含缓存两个口径。

  Key points

  • The bill is the sum of every turn's input across the session, not the last request's size.
  • Sum by growth pattern: prefix times turns, history as an arithmetic series, tool results by call count.
  • The cacheable part is the stable prefix ordered tools, system, messages; editing earlier segments invalidates later ones.
  • Writes cost about 1.25 times base and hits about 0.1 times, but only above the model's minimum cacheable length, which fails silently.
  • So trimming can lower tokens while raising cost; always report both cached and uncached figures.