Interview Bank

Context Engineering in 5 Days

D5 Measuring and Tuning: the Token Bill, Context Utilization, Failure-Mode Triage, and a Comprehensive Interview Deep Dive

• How much context engineering is enough, and how do you know when to stop?上下文工程做到什么程度算够？你怎么知道该停手了？

  Common in ChinaCommon overseasIntermediate#tuning#stopping-criteria

  How to reason about it · think before answering

  1. This is open-ended but has a clear right shape. Saying more optimization is always better reads as lacking cost awareness, because context work is unbounded and will be overdone without a stopping rule.
  2. Name the concrete cost of overdoing it rather than stopping at wasted time. Trim too hard and you cut fields needed later; compact too hard and you lose hard requirements that do not read like conclusions; cut tools too far and the agent cannot finish the task. None of these raise errors; they show up only in accuracy, the most expensive bill.
  3. Give at least three checkable stopping conditions: useful-token share stable in a healthy band such as above fifty percent with no headroom across several measurements; per-turn occupancy under fifty percent on your longest case; and the last change delivering less than a five percent bill reduction.
  4. Land on the third: a sub-five-percent gain means what remains is necessary overhead, and squeezing further trades accuracy for money. It matters most because it is the only condition that transfers across projects unchanged.
  5. Expect the follow-up on preventing regression. Freeze the measurement into a regression suite: fixed cases, rerun on every change, bill and both metrics under monitoring. Model upgrades, tool churn, and downstream field changes each degrade it again.

  分析过程 · 先想清楚再作答

  1. 这题是开放题，但它有明确的好坏。答「越优化越好」的人会被判成没有成本意识，因为上下文工程是个能无限做下去的活，不定停手判据就一定会做过头。
  2. 怎么拆：先说清过度优化的具体代价，不要停在「浪费时间」。裁得太狠会把后面才用得上的字段裁掉，压得太狠会丢掉不像结论的硬性要求，工具裁得太少会让模型没法完成任务。这些都不报错，只在正确率上体现，而正确率是最贵的一笔账。
  3. 给可核对的停手条件，至少三条：有效信息占比稳定在一个合理区间（比如 50% 以上）且连续几次测量没有上升空间；最长那条用例上的单轮窗口占用率不超过 50%；最近一次改动带来的账单降幅低于 5%。
  4. 结论落在第三条：降幅低于 5% 说明剩下的都是必要开销，继续压就是在拿正确率换钱。这条比前两条更重要，因为它是唯一一条与具体项目无关、可以直接复用的判据。
  5. 可预期的追问：那怎么保证停手之后不退化？把这套度量固化成回归：一批固定用例、每次改动都重跑、账单与两个指标进监控。上下文工程不是一次性项目，模型换代、工具增减、下游接口改字段，任何一件都会让它重新变差。

  Key points

  • Overdoing it fails silently in accuracy: fields needed later get cut, hard requirements get summarized away, and too few tools leave the task unfinishable.
  • Three stopping conditions: a stable useful-token share with no headroom, per-turn occupancy under fifty percent on the longest case, and a last change worth under five percent of the bill.
  • The third transfers best: under five percent means what remains is necessary overhead and further squeezing trades accuracy for money.
  • After stopping, freeze it into regression: fixed cases, rerun on every change, and monitor the bill plus both metrics.

  答题要点

  • 过度优化的代价不报错，只在正确率上体现：裁掉后面才用的字段、压掉不像结论的硬性要求、工具少到做不完任务。
  • 三条停手判据：有效信息占比稳定且无上升空间、最长用例的单轮占用率不超过 50%、最近一次改动账单降幅低于 5%。
  • 第三条最通用：降幅低于 5% 说明剩下的是必要开销，再压就是拿正确率换钱。
  • 停手后要固化成回归：固定用例、每次改动重跑、账单与两个指标进监控。