Measuring and Tuning: the Token Bill, Context Utilization, Failure-Mode Triage, and a Comprehensive Interview Deep Dive

Interview questions

• How do you compute the token bill for one agent task, and which parts can be cached away?怎么给一个 Agent 算一次任务的 token 账单？哪些部分是可以被缓存掉的？

  Common in ChinaCommon overseasDeep dive#token-accounting#prompt-caching

  How to reason about it · think before answering

  1. The first trap is the phrase one task. Many people quote a single request's input size, which is a weight reading, not a bill. Stateless models resend everything each turn, so the bill is the sum of every turn's input.
  2. Sum the four buckets by their growth patterns. The stable prefix (system prompt plus tool definitions) is resent verbatim, so multiply by turn count. History grows linearly, so it is an arithmetic series. Tool results grow in steps, so estimate calls times size. For scale: a twenty-turn support task whose final request is 11256 tokens totals 133502 across the session, nearly twelve times larger.
  3. Then caching. The cacheable part is the stable prefix, ordered tools, system, messages, where editing anything earlier invalidates everything after. Writes cost about 1.25 times base (about 2 times for a one-hour lifetime) and hits about 0.1 times, so twenty full-price prefixes become one write plus nineteen hits, an eighty percent saving.
  4. State the threshold: the prefix must reach the model's minimum cacheable length or caching silently does nothing. That produces the counterintuitive result where halving your system prompt lowers token count but raises the bill, because the prefix fell below the threshold.
  5. Expect the follow-up on whether to trim anyway. Yes, but report two numbers: the raw token reduction and the cache-adjusted effective reduction, and check whether the prefix crossed the threshold. If it did, add stable reference content back into the prefix or move to a model with a lower threshold.

  分析过程 · 先想清楚再作答

  1. 这题的第一个坑在「一次任务」四个字。很多人报的是单次请求的输入量，那是称重不是账单——模型没有记忆，每一轮都要把前面全部重发，账单是整场会话每轮输入的累加值。
  2. 怎么拆：按四块各自的增长方式分别求和。稳定前缀（系统提示加工具定义）每轮原样重发，乘轮数；对话历史线性增长，是等差数列求和；工具结果阶梯增长，按调用次数与每次体积估。举个量级：一个 20 轮的客服任务，最后一轮单次输入 11256，整场累加是 133502，差了将近 12 倍。
  3. 再谈缓存。可缓存的是稳定前缀这一段，顺序是工具定义、系统提示、消息，改前面的会让后面全部失效。经济学是写入约 1.25 倍原价（一小时存活期约 2 倍）、命中约 0.1 倍，所以 20 轮的前缀从 20 次全价变成一次写入加十九次命中，能便宜八成以上。
  4. 结论要带上那条门槛：前缀必须达到模型的最小可缓存长度才生效，达不到既不报错也不告警。这直接导致一个反直觉现象——把系统提示精简掉一半，token 数降了，账单反而可能涨，因为前缀掉到门槛以下、缓存静默失效。
  5. 可预期的追问：那还该不该精简？该，但要同时报两个数——不含缓存的 token 降幅与含缓存的等效开销降幅，并检查前缀有没有跨过门槛。跨过了就把稳定的引用内容放回前缀抬回去，或者换一个门槛更低的模型。

  Key points

  • The bill is the sum of every turn's input across the session, not the last request's size.
  • Sum by growth pattern: prefix times turns, history as an arithmetic series, tool results by call count.
  • The cacheable part is the stable prefix ordered tools, system, messages; editing earlier segments invalidates later ones.
  • Writes cost about 1.25 times base and hits about 0.1 times, but only above the model's minimum cacheable length, which fails silently.
  • So trimming can lower tokens while raising cost; always report both cached and uncached figures.

  答题要点

  • 账单是整场会话每轮输入的累加值，不是最后一次请求的输入量。
  • 按四块的增长方式分别求和：前缀乘轮数、历史等差求和、工具结果按调用次数估。
  • 可缓存的是稳定前缀，顺序是工具定义、系统提示、消息，改前面会让后面全失效。
  • 写入约 1.25 倍、命中约 0.1 倍；但前缀必须达到最小可缓存长度，否则静默失效。
  • 所以精简可能让 token 降而账单涨，必须同时报含缓存与不含缓存两个口径。
• When context is the problem, how do you localize which of the four buckets is at fault?上下文出问题的时候，你怎么定位是四块里的哪一块？

  Common in ChinaCommon overseasIntermediate#diagnostics#metrics

  How to reason about it · think before answering

  1. This tests a diagnostic path. Answering with check the logs or try again reads as having no method; interviewers want a fixed chain from symptom to metric to change.
  2. Start with two metrics. Window occupancy is per-turn input over the window limit and governs whether you will overflow. Useful-token share is the tokens later steps actually use over total tokens and governs whether the spend is worth it. Approximate the latter with your trimmer: whatever survives trimming is the numerator.
  3. Then four failure modes with their fingerprints: overstuffed (early instructions ignored, high occupancy), buried (the fact is in the window yet the model denies it, low useful share), underspecified (answers waver across identical questions, low occupancy but high error rate), and drifting (original constraints violated late in the session, retention checks failing after compaction).
  4. Highlight the two most misdiagnosed. Buried is routinely blamed on model capability; the test is to print the context and search for the fact by hand, and if it is there the problem is context, not the model. Underspecified is blamed on instability, when it usually means contradictory rules in the system prompt or two overlapping tools making the model waver.
  5. Expect the follow-up on conflicting metrics. Low occupancy with a low useful share is the dangerous combination, because nothing looks urgent while you pay full price to move noise and dilute attention. Trust the useful-token share there.

  分析过程 · 先想清楚再作答

  1. 这题在考排查路径。答「先看日志」「多试几次」的会被判成没有方法论，面试官想听的是从现象到指标再到改动的一条固定链路。
  2. 怎么拆：先给两个指标。窗口占用率是单轮输入除以窗口上限，管的是会不会撑爆；有效信息占比是后续步骤真正用到的 token 除以总 token，管的是值不值。后者可以用裁剪器近似量：裁完还剩的那部分就是分子。
  3. 再给四种失败模式与各自的指纹：塞太满（漏读早期指令，占用率高）、找不到（信息在窗口里但模型说没有，有效信息占比低）、说不清（同类问题答法摇摆，占用率不高但错误率高）、越走越偏（跑久了违反最初约束，压缩前后的保留检查出现失败项）。
  4. 结论给最容易误判的两种。「找不到」常被误判成模型能力不足，判据是把上下文打印出来人肉搜一遍那条信息在不在——在就是上下文问题，不是模型问题。「说不清」常被误判成模型不稳定，实际多半是系统提示里有互相矛盾的规则，或者两个职责重叠的工具让模型在决策点上横跳。
  5. 可预期的追问：两个指标冲突时听谁的？答：占用率低但有效信息占比也低的情况最危险，因为看起来毫无压力却在按原价搬运垃圾，同时还在稀释注意力。这时应该以有效信息占比为准。

  Key points

  • Two metrics: occupancy for overflow risk, useful-token share for whether the spend earns its place, approximated with a trimmer.
  • Each mode has a fingerprint: occupancy for overstuffed, useful share for buried, error rate for underspecified, post-compaction retention checks for drifting.
  • Buried is most often misdiagnosed as model capability; print the context and search by hand.
  • Underspecified usually means contradictory rules or overlapping tools; hunt the contradiction rather than swapping models.

  答题要点

  • 两个指标：窗口占用率管会不会撑爆，有效信息占比管值不值，后者可用裁剪器近似量。
  • 四种模式各有指纹：塞太满看占用率、找不到看有效信息占比、说不清看错误率、越走越偏看压缩后的保留检查。
  • 找不到最容易被误判成模型能力问题，判据是把上下文打印出来人肉搜一遍。
  • 说不清多半是规则互相矛盾或工具职责重叠，去搜矛盾比换模型有用。
• How much context engineering is enough, and how do you know when to stop?上下文工程做到什么程度算够？你怎么知道该停手了？

  Common in ChinaCommon overseasIntermediate#tuning#stopping-criteria

  How to reason about it · think before answering

  1. This is open-ended but has a clear right shape. Saying more optimization is always better reads as lacking cost awareness, because context work is unbounded and will be overdone without a stopping rule.
  2. Name the concrete cost of overdoing it rather than stopping at wasted time. Trim too hard and you cut fields needed later; compact too hard and you lose hard requirements that do not read like conclusions; cut tools too far and the agent cannot finish the task. None of these raise errors; they show up only in accuracy, the most expensive bill.
  3. Give at least three checkable stopping conditions: useful-token share stable in a healthy band such as above fifty percent with no headroom across several measurements; per-turn occupancy under fifty percent on your longest case; and the last change delivering less than a five percent bill reduction.
  4. Land on the third: a sub-five-percent gain means what remains is necessary overhead, and squeezing further trades accuracy for money. It matters most because it is the only condition that transfers across projects unchanged.
  5. Expect the follow-up on preventing regression. Freeze the measurement into a regression suite: fixed cases, rerun on every change, bill and both metrics under monitoring. Model upgrades, tool churn, and downstream field changes each degrade it again.

  分析过程 · 先想清楚再作答

  1. 这题是开放题，但它有明确的好坏。答「越优化越好」的人会被判成没有成本意识，因为上下文工程是个能无限做下去的活，不定停手判据就一定会做过头。
  2. 怎么拆：先说清过度优化的具体代价，不要停在「浪费时间」。裁得太狠会把后面才用得上的字段裁掉，压得太狠会丢掉不像结论的硬性要求，工具裁得太少会让模型没法完成任务。这些都不报错，只在正确率上体现，而正确率是最贵的一笔账。
  3. 给可核对的停手条件，至少三条：有效信息占比稳定在一个合理区间（比如 50% 以上）且连续几次测量没有上升空间；最长那条用例上的单轮窗口占用率不超过 50%；最近一次改动带来的账单降幅低于 5%。
  4. 结论落在第三条：降幅低于 5% 说明剩下的都是必要开销，继续压就是在拿正确率换钱。这条比前两条更重要，因为它是唯一一条与具体项目无关、可以直接复用的判据。
  5. 可预期的追问：那怎么保证停手之后不退化？把这套度量固化成回归：一批固定用例、每次改动都重跑、账单与两个指标进监控。上下文工程不是一次性项目，模型换代、工具增减、下游接口改字段，任何一件都会让它重新变差。

  Key points

  • Overdoing it fails silently in accuracy: fields needed later get cut, hard requirements get summarized away, and too few tools leave the task unfinishable.
  • Three stopping conditions: a stable useful-token share with no headroom, per-turn occupancy under fifty percent on the longest case, and a last change worth under five percent of the bill.
  • The third transfers best: under five percent means what remains is necessary overhead and further squeezing trades accuracy for money.
  • After stopping, freeze it into regression: fixed cases, rerun on every change, and monitor the bill plus both metrics.

  答题要点

  • 过度优化的代价不报错，只在正确率上体现：裁掉后面才用的字段、压掉不像结论的硬性要求、工具少到做不完任务。
  • 三条停手判据：有效信息占比稳定且无上升空间、最长用例的单轮占用率不超过 50%、最近一次改动账单降幅低于 5%。
  • 第三条最通用：降幅低于 5% 说明剩下的是必要开销，再压就是拿正确率换钱。
  • 停手后要固化成回归：固定用例、每次改动重跑、账单与两个指标进监控。

Comments

Sign in to join the discussion

No comments yet — be the first.