Interview Bank

Build an AI Short-Drama Production Pipeline With Agents in 14 Days

D9 Concurrency and Quotas: Starting Multiple Episodes at Once Without Blowing Through Any Provider's Limits

• Many jobs share one vendor's quota. How would you design the rate limiting?多个任务共用一家厂商的额度，你会怎么设计限流？

  Common in ChinaCommon overseasIntermediate#rate-limiting#concurrency#scheduling

  How to reason about it · think before answering

  1. The hinge is the word shared. Naming a token bucket only answers half of it; the interviewer wants your bucketing dimension and what else sits around the bucket.
  2. Dimension first: bucket per vendor and per API family, never one global bucket. Image and speech quotas at the same vendor are separate pools, and merging them lets the tight one throttle the loose one.
  3. Then the parts: one bucket is not enough. A token bucket caps rate (calls per minute, a number the vendor sets); a semaphore caps concurrency (how many are in flight, a number you set to protect memory and spend). Rate alone lets a fast endpoint fire hundreds per minute; concurrency alone lets twenty downloads pile up.
  4. The engineering detail that decides everything: admission must be non-blocking. If a job is dispatched first and then waits for a token inside the worker, low-priority work pins every worker slot and priority silently stops working. Ask the gate before dispatch, and skip to the next candidate when it says no.
  5. Finally the numbers: concurrency should not track CPU cores, since this pipeline is almost all network wait. Derive it from vendor quota and from how much work one failure forces you to redo.
  6. Expected follow-up: where does bucket state live. In-memory is fine for one process; across processes it belongs in Redis behind an atomic token-take script, or each process limits itself and the sum still blows the quota.

  分析过程 · 先想清楚再作答

  1. 这题的题眼是「共用」两个字。只回答一个令牌桶算答了一半，面试官想听的是你按什么维度分桶、以及桶之外还需要什么。
  2. 先给维度：限流要按「厂商 + 接口类别」分桶，不能全局一个桶。同一家的图像和语音是两个独立的配额池，混在一起会让紧的那个把松的那个也拖住。
  3. 再给部件：一个桶不够，要两个。令牌桶管速率（一分钟发几次，数字是厂商定的），信号量管并发（同一时刻挂着几个，数字是你自己定的用来保护内存和钱包）。只有速率控制的话，快速返回的接口一分钟能发几百次；只有并发控制的话，二十个请求同时在飞会把内存挂满。
  4. 然后是关键的工程细节：拿许可的动作必须是非阻塞的。如果任务先被派出去、再在执行流里等令牌，工作槽会被一批低优先任务占死，优先级就静默失效了。正确结构是调度器派活之前先问闸门要许可，拿不到就跳过它去看下一个候选。
  5. 最后落到取值：并发上限不该按 CPU 核数定，这条线几乎没有本地计算，全在等网络；它该按「一次失败要重跑多少东西」和厂商配额来定。
  6. 可预期的追问是「桶的状态放哪」。单进程放内存就够；多进程要放 Redis，用一个原子脚本取令牌，否则每个进程各限各的，加起来照样超。

  Key points

  • Bucket per vendor and per API family; image and speech at one vendor get separate gates.
  • Each gate has two parts: a token bucket for rate (the vendor's RPM) and a semaphore for in-flight concurrency (your own number).
  • Admission is non-blocking: if the gate says no, the job stays queued instead of holding a worker slot.
  • Take the concurrency slot before the token, or a rejected admission silently burns quota.
  • Across processes, move bucket state to Redis and take tokens atomically.

  答题要点

  • 按「厂商 + 接口类别」分桶，一家的图像和语音各一道闸门。
  • 每道闸门两个部件：令牌桶控速率（厂商给的 RPM），信号量控并发（自己定的在飞上限）。
  • 准入必须非阻塞，拿不到许可就把任务留在队列里，绝不占着工作槽干等。
  • 先抢并发票再取令牌，顺序反了会白白扣掉配额。
  • 多进程部署时桶的状态要外置到 Redis，用原子操作取令牌。