面试题库

5 天上下文工程

D5 度量与调优：token 账单、上下文利用率、失败模式排查与综合面试专题

• 上下文出问题的时候，你怎么定位是四块里的哪一块？When context is the problem, how do you localize which of the four buckets is at fault?

  国内高频海外高频进阶#diagnostics#metrics

  分析过程 · 先想清楚再作答

  1. 这题在考排查路径。答「先看日志」「多试几次」的会被判成没有方法论，面试官想听的是从现象到指标再到改动的一条固定链路。
  2. 怎么拆：先给两个指标。窗口占用率是单轮输入除以窗口上限，管的是会不会撑爆；有效信息占比是后续步骤真正用到的 token 除以总 token，管的是值不值。后者可以用裁剪器近似量：裁完还剩的那部分就是分子。
  3. 再给四种失败模式与各自的指纹：塞太满（漏读早期指令，占用率高）、找不到（信息在窗口里但模型说没有，有效信息占比低）、说不清（同类问题答法摇摆，占用率不高但错误率高）、越走越偏（跑久了违反最初约束，压缩前后的保留检查出现失败项）。
  4. 结论给最容易误判的两种。「找不到」常被误判成模型能力不足，判据是把上下文打印出来人肉搜一遍那条信息在不在——在就是上下文问题，不是模型问题。「说不清」常被误判成模型不稳定，实际多半是系统提示里有互相矛盾的规则，或者两个职责重叠的工具让模型在决策点上横跳。
  5. 可预期的追问：两个指标冲突时听谁的？答：占用率低但有效信息占比也低的情况最危险，因为看起来毫无压力却在按原价搬运垃圾，同时还在稀释注意力。这时应该以有效信息占比为准。

  How to reason about it · think before answering

  1. This tests a diagnostic path. Answering with check the logs or try again reads as having no method; interviewers want a fixed chain from symptom to metric to change.
  2. Start with two metrics. Window occupancy is per-turn input over the window limit and governs whether you will overflow. Useful-token share is the tokens later steps actually use over total tokens and governs whether the spend is worth it. Approximate the latter with your trimmer: whatever survives trimming is the numerator.
  3. Then four failure modes with their fingerprints: overstuffed (early instructions ignored, high occupancy), buried (the fact is in the window yet the model denies it, low useful share), underspecified (answers waver across identical questions, low occupancy but high error rate), and drifting (original constraints violated late in the session, retention checks failing after compaction).
  4. Highlight the two most misdiagnosed. Buried is routinely blamed on model capability; the test is to print the context and search for the fact by hand, and if it is there the problem is context, not the model. Underspecified is blamed on instability, when it usually means contradictory rules in the system prompt or two overlapping tools making the model waver.
  5. Expect the follow-up on conflicting metrics. Low occupancy with a low useful share is the dangerous combination, because nothing looks urgent while you pay full price to move noise and dilute attention. Trust the useful-token share there.

  答题要点

  • 两个指标：窗口占用率管会不会撑爆，有效信息占比管值不值，后者可用裁剪器近似量。
  • 四种模式各有指纹：塞太满看占用率、找不到看有效信息占比、说不清看错误率、越走越偏看压缩后的保留检查。
  • 找不到最容易被误判成模型能力问题，判据是把上下文打印出来人肉搜一遍。
  • 说不清多半是规则互相矛盾或工具职责重叠，去搜矛盾比换模型有用。

  Key points

  • Two metrics: occupancy for overflow risk, useful-token share for whether the spend earns its place, approximated with a trimmer.
  • Each mode has a fingerprint: occupancy for overstuffed, useful share for buried, error rate for underspecified, post-compaction retention checks for drifting.
  • Buried is most often misdiagnosed as model capability; print the context and search by hand.
  • Underspecified usually means contradictory rules or overlapping tools; hunt the contradiction rather than swapping models.