Interview Bank

From Frontend Engineer to Agent Engineer in 30 Days

D9 A Redis Streams Message Bus: XADD/XREADGROUP/XACK/XAUTOCLAIM, Consumer Groups, Poison Messages

• What problems do XACK and XAUTOCLAIM each solve, and what changes if you XACK before instead of after doing the work?XACK 和 XAUTOCLAIM 分别解决什么问题？XACK 放在业务处理之前和之后有什么区别？

  Common in ChinaCommon overseasIntermediate#message-bus#redis-streams#error-handling

  How to reason about it · think before answering

  1. The hinge is the second half. The first half is documentation; the second asks whether you know that ack timing decides the delivery semantics of the whole system.
  2. Split the two commands: XACK clears a message from the pending list, meaning the work is genuinely finished; XAUTOCLAIM reassigns a pending message that has been idle past a threshold, meaning its previous owner may be dead. One is the normal path, the other is the failure path.
  3. Then answer the timing question categorically: ack-then-work is at-most-once, work-then-ack is at-least-once. In the first, a crash makes the message vanish — it is not in the pending list, so XAUTOCLAIM cannot recover it. In the second, the worst case is duplicate execution, and duplicates can be blocked by idempotency while lost work cannot. Always work first, except for fire-and-forget telemetry.
  4. Add the point most people miss: on failure the correct action is to do nothing and leave the message pending for XAUTOCLAIM. Acking inside the catch block silently discards failures, which is worse than no retry because you no longer know what you lost.
  5. Add the parameter trade-off: the idle threshold must exceed the worst-case normal processing time. Too small and a healthy in-flight message gets stolen and executed twice; too large and recovery is slow. Be explicit that tuning it only lowers the probability of duplicates — the real backstop is a uniqueness constraint on the consumer side.
  6. Expect: why XAUTOCLAIM rather than XCLAIM? XCLAIM needs an XPENDING scan first and then a named claim, with a race in between; XAUTOCLAIM scans and returns a cursor in one command, and is the recommended approach since Redis 6.2.

  分析过程 · 先想清楚再作答

  1. 题眼在后半句。前半句背文档就能答，后半句在考你知不知道 ack 的时机直接决定了整个系统的投递语义——答不出这一点，面试官会判定你没在生产里管过队列。
  2. 先把两个命令的分工说清：XACK 是「销号」，把消息从 pending 清单里删掉，代表这件事真的做完了；XAUTOCLAIM 是「接手」，把闲置超过阈值的 pending 消息改判给另一个消费者，代表原来那个人可能已经死了。一个负责正常收尾，一个负责异常兜底。
  3. 然后回答时机问题，用一句话定性：先 ack 再干活是 at-most-once，先干活再 ack 是 at-least-once。前者进程一崩消息就人间蒸发，pending 清单里查不到、XAUTOCLAIM 也捡不回来；后者最坏是重复执行，而重复可以用幂等挡掉，丢单挡不掉。所以除了埋点日志这类丢一条无所谓的场景，一律先干活再 ack。
  4. 补一个大多数人漏掉的点：处理失败时正确的动作是**什么都不做**，让消息留在 pending 里等 XAUTOCLAIM。很多人会在 catch 里顺手 ack 掉，那等于把失败的消息静默丢弃，比不重试更糟——因为你连丢了什么都不知道。
  5. 再补一条 XAUTOCLAIM 的参数取舍：空闲阈值要大于「一次正常处理的耗时上限」。给太小会把还在正常处理的消息抢走，同一件事被跑两遍；给太大则故障恢复变慢。但要说清，调大阈值只降低重复概率，不消灭重复，兜底始终是消费端的唯一约束。
  6. 可以预期的追问：为什么用 XAUTOCLAIM 而不是 XCLAIM？答：XCLAIM 要你先 XPENDING 查出候选 id 再点名认领，两步之间还有竞态；XAUTOCLAIM 自己扫 pending 并返回游标，一条命令搞定，是 Redis 6.2 之后的推荐做法。

  Key points

  • XACK is the happy-path close-out: it clears the message from the pending list; repeat acks return 0, so it is naturally idempotent
  • XAUTOCLAIM is the failure backstop: it reassigns pending messages idle past a threshold, answering 'what happens to work held by a dead consumer'
  • Ack-before-work is at-most-once and loses work on a crash; work-before-ack is at-least-once and at worst duplicates, which idempotency can absorb
  • Never ack on failure — leave the message pending for takeover; acking in the catch block silently discards failures
  • The idle threshold should exceed worst-case processing time, but tuning it only reduces duplicates; uniqueness constraints are the real guarantee

  答题要点

  • XACK 负责正常收尾：把消息从 pending 清单里销号，代表这件事真的做完了；重复 ack 返回 0，天生幂等
  • XAUTOCLAIM 负责异常兜底：把闲置超过阈值的 pending 消息改判给另一个消费者，解决「消费者死了它手上的消息怎么办」
  • 先 ack 再干活是 at-most-once，崩溃就丢单；先干活再 ack 是 at-least-once，最坏是重复，可以用幂等挡
  • 处理失败时不要 ack，让消息留在 pending 里等接手；在 catch 里顺手 ack 等于静默丢弃失败
  • 空闲阈值要大于正常处理耗时的上限，但调大只降低重复概率，兜底仍是消费端唯一约束
• How do you choose between Redis Streams and Kafka, and when is Streams clearly not enough?Redis Streams 和 Kafka 该怎么选？什么情况下 Streams 明显不够用？

  Common in ChinaCommon overseasIntermediate#message-bus#redis-streams#architecture

  How to reason about it · think before answering

  1. The bad answer is 'it depends on volume'. Throughput is never the first criterion — a single Redis node handles tens of thousands of XADDs per second, and most workloads never approach that ceiling. 'Small volume Streams, large volume Kafka' reads as never having run a real evaluation.
  2. Use two real criteria instead: how long the messages must be retained, and whether a second class of consumer will appear. If a message is useless once executed and the execution layer is the only consumer, Streams is plenty and saves an entire operational surface. If you need replay from any point in the last three months, or the same data must feed real-time execution, an offline warehouse and a risk engine, choose Kafka.
  3. Add three structural differences: Streams is memory-first with retention you enforce yourself via MAXLEN or XTRIM, while Kafka does sequential disk writes and keeps weeks by default; a Streams group takes any number of consumers, while Kafka consumers are capped by partition count and extras idle; ordering granularity differs — Streams orders a single stream but dispatches randomly within a group, Kafka pins a key to a partition and orders within it.
  4. Then volunteer the line that shows real depth: Redis persistence is lossy. AOF fsyncs once per second by default, so the last second of writes can vanish, and replication is asynchronous, so a failover can drop unreplicated messages. Using Streams therefore requires a source of truth elsewhere — here the Postgres runs table, with the stream acting only as a trigger; a lost message leaves the run pending and a sweeper republishes it. Treating the bus as the only datastore is the dangerous misuse.
  5. Land on a reusable rule: Streams suits triggering work, Kafka suits data pipelines. One carries one-shot commands, the other carries facts that many parties re-read.
  6. Expect: what about RabbitMQ or SQS? RabbitMQ wins on complex routing and delayed delivery (Streams has no native delay, you republish with a next-eligible timestamp); SQS wins on zero operations at the cost of replay and strict ordering (FIFO queues aside). Framing the criteria as retention, number of consumers, routing complexity and operational budget beats reciting product specs.

  分析过程 · 先想清楚再作答

  1. 这题的坏答案是「看数据量」。吞吐从来不是第一判据——单机 Redis 每秒几万条 XADD 毫无压力，绝大多数业务的量级根本碰不到天花板。答成「量小用 Streams、量大用 Kafka」会被认为没做过选型。
  2. 换成两个真正的判据来推：一、这些消息需要保留多久；二、会不会有第二类消费方。生命周期是「执行一次就没用了」、且只有执行层这一个消费方，Streams 完全够用，还省掉一整套运维；需要「三个月内任意时间点重放」、或者同一份数据要同时喂给实时执行、离线数仓、风控三条链路，那就该上 Kafka。
  3. 再补三条结构性差异：Streams 是内存为主、保留全靠你自己 MAXLEN 或 XTRIM，Kafka 是磁盘顺序写、保留几周是常态；Streams 一个组里加多少消费者都行，Kafka 的消费者数受分区数限制，多了就有人空转；顺序保证的粒度不同，Streams 是单条流内有序而组内分配随机，Kafka 是同 key 落同分区、分区内有序。
  4. 然后主动说出那条最能体现深度的话：Redis 的持久化是有损的。AOF 默认每秒刷盘，最坏丢最后一秒的写入；主从异步复制，故障切换时未同步的消息会消失。所以用 Streams 时架构上必须有一个真相之源——本课是 Postgres 的 runs 表，流只是触发器，丢了消息那个 run 还停在 pending，补投任务会把它捡回来。把总线当唯一数据源是最危险的误用。
  5. 结论落成一句可复用的判断：Streams 适合「触发执行」，Kafka 适合「数据管道」。前者的消息是一次性的命令，后者的消息是需要被多方反复读取的事实。
  6. 可以预期的追问：那 RabbitMQ、SQS 呢？答：RabbitMQ 强在复杂路由和延迟队列（Streams 没有原生延迟投递，要自己带「下次可执行时间」重投）；SQS 强在零运维，代价是没有回放、也没有严格顺序（FIFO 队列另算）。把判据说成「保留时长、消费方数量、路由复杂度、运维预算」四条，比背产品参数强得多。

  Key points

  • The first criterion is not throughput but retention length and whether a second class of consumer will exist
  • One consumer class and messages that expire on execution: Streams is enough, and you probably already run Redis
  • Long retention with arbitrary replay, or one dataset feeding several downstream pipelines: pick Kafka
  • Structural differences: Streams is memory-first with self-managed trimming and random in-group dispatch; Kafka is sequential-disk, key-partitioned with in-partition ordering, and caps consumers at partition count
  • Redis persistence is lossy (per-second AOF fsync, async replication), so the database must be the source of truth with the stream as a trigger plus a republish sweeper
  • One-line rule: Streams triggers work, Kafka moves data

  答题要点

  • 第一判据不是吞吐，是「消息要保留多久」和「会不会有第二类消费方」
  • 只有执行层一个消费方、消息执行完即失效：Streams 够用，且大概率你已经有 Redis，零新增运维
  • 需要长期保留与任意时间点回放、或多条下游链路共用同一份数据：选 Kafka
  • 结构差异：Streams 内存为主、保留靠自己裁剪、组内分配随机；Kafka 磁盘顺序写、按 key 分区且分区内有序、消费者数受分区限制
  • Redis 持久化有损（AOF 每秒刷盘、异步复制），所以真相之源必须是数据库，流只当触发器，靠补投任务兜底
  • 一句话判断：Streams 适合触发执行，Kafka 适合数据管道
• What do you do with a message that keeps failing? Design a poison-message isolation mechanism.一条消息反复处理失败怎么办？请设计一个毒消息隔离机制。

  Common in ChinaCommon overseasIntermediate#message-bus#error-handling#reliability

  How to reason about it · think before answering

  1. This question probes whether you have ever watched one bad message stall an entire stream. The test is simple: does your answer contain a concrete threshold and a concrete place where isolation happens? If not, you are talking theory.
  2. Describe the failure mode first: under at-least-once you do not ack on failure, so the message stays pending and gets redelivered. A message that fails for everyone therefore loops forever — delivered, failed, idle timeout, claimed, failed — never recovering while continuously consuming worker capacity.
  3. Then give the mechanism, three actions and all of them required. One, use the delivery count the pending list already tracks rather than building a counter table. Two, past the threshold (three deliveries in this course) move the message to a dead-letter stream carrying the original id, delivery count and failure reason. Three, XACK the original stream and mark the run failed with the error recorded. Moving without acking leaves it pending for another takeover; acking without moving makes both the message and its reason disappear, leaving the user stuck on 'thinking'.
  4. Justify the threshold: one delivery kills messages that a single network blip would have let through; ten wastes ten executions of money and time on a message that can never succeed. Three deliveries, spaced by the idle threshold, survives almost all transient faults.
  5. Volunteer a limitation: Redis Streams has no native exponential backoff — redelivery timing is governed by the idle threshold. Backoff requires republishing the message with a next-eligible timestamp, which means building a delay queue yourself. Naming this shows you know where Streams ends.
  6. Expect: is creating the dead-letter stream the end of it? No. Its depth must be alerted on, since going from zero to non-zero usually means a class of input your code cannot handle — a real bug, not bad luck. Keep a replay path too: republish the stored fields back to the original stream, and because the idempotency key is preserved, replay cannot cause duplicate execution. Teams that build a dead-letter stream and never open it have simply muted their failures.

  分析过程 · 先想清楚再作答

  1. 这题在考你有没有踩过「一条坏消息拖垮整条流」。判断标准很简单：你的回答里有没有出现一个具体的阈值和一个具体的落地位置，没有就是在讲概念。
  2. 先把故障模式说清楚：按 at-least-once 的规矩，失败就不 ack、留在 pending 等重投，于是一条无论谁来都会失败的消息进入死循环——投递、失败、闲置超时、被接手、再失败。它自己永远好不了，还持续占用消费者的处理能力。
  3. 然后给机制，三个动作缺一不可：一、判定依据用 pending 清单自己记的投递次数，不要另建计数表；二、超过阈值（本课固定 3 次）就把消息搬到一条死信流，字段里带上原始消息 id、投递次数和失败原因；三、对原流 XACK，同时把这次执行标成失败并写入错误原因。只搬不 ack，它还躺在 pending 里等着被接手；只 ack 不搬，消息和失败原因一起消失，用户永远停在「正在思考」。
  4. 阈值的取值要给出权衡：定 1 会让一次网络抖动就把本来能成功的消息判死；定 10 会在一条必死的消息上浪费十次执行的钱和时间。3 次配合每次之间的空闲阈值，足够熬过绝大多数瞬时故障。
  5. 还要主动说出一个缺口：Redis Streams 没有原生的指数退避，重投时机由空闲阈值决定。想要退避就得自己把消息重新投递并带上「下次可执行时间」，那已经是在实现延迟队列了——这一条能体现你知道 Streams 的边界在哪。
  6. 可以预期的追问：死信流建完就完了吗？答：不。死信条数必须接进告警，它从 0 变成非 0 通常意味着有一类输入你的代码处理不了，是真 bug 而不是运气差；还要留一个重放入口——把死信里的字段原样投回原流即可，因为幂等键还在，重放不会产生重复执行。见过团队把死信建起来半年没打开过，那等于把故障静音了。

  Key points

  • Failure mode: under at-least-once you do not ack on failure, so an always-failing message is redelivered forever and keeps consuming worker capacity
  • Use the delivery count already tracked in the pending list rather than a separate counter table
  • Fix the threshold at three deliveries: one kills transient failures, ten wastes ten executions on a message that can never succeed
  • Isolation needs all three actions: move to a dead-letter stream with original id, delivery count and reason; XACK the original stream; mark the run failed with the error stored
  • Redis Streams has no native exponential backoff — redelivery timing follows the idle threshold, so backoff means implementing delayed republishing yourself
  • Alert on dead-letter depth and keep a replay path; the idempotency key survives, so replay cannot duplicate execution

  答题要点

  • 故障模式：at-least-once 下失败不 ack，一条永远失败的消息会无限重投并持续占用消费者
  • 判定依据用 pending 清单里记的投递次数，不需要另建计数表
  • 阈值固定 3 次：定 1 会误杀瞬时故障，定 10 会在必死消息上浪费十次执行成本
  • 隔离动作三件缺一不可：搬到死信流（带原始 id、投递次数、失败原因）、对原流 XACK、把这次执行标成失败并写入原因
  • Redis Streams 没有原生指数退避，重投时机由空闲阈值决定，要退避得自己实现延迟投递
  • 死信流要接告警并留重放入口；幂等键还在，重放不会导致重复执行