镇上只有一家小馆子，老板娘守着一部电话接订座。这部电话一次只能通一个人。

某个周五傍晚，十几桌客人几乎在同一分钟想订今晚的位子。第一个拨通了，其余的人听到的全是”嗨——嗨——”的忙音。

被挡住的人都很守规矩：他们看了看墙上的钟，决定”一分钟后再打”。于是整整一分钟后——所有人又在同一秒按下了拨号键。电话再次被第一个抢通，其余十几个人又撞了个满怀，还是忙音。一分钟又一分钟，这群人排着一条看不见、谁也进不去的队，把那条线活活堵死了。这种”大家一起重试、一起失败”的恶性循环，本身比线路繁忙更可怕。

后来一位常客想了个法子。他对大家说：第一次没打通，等一分钟；还不通，就等两分钟；再不通，等四分钟、八分钟……每失败一次，等待时间翻一倍。这样越往后，单位时间里冒出来的电话越少，那条线终于能喍口气。

可麻烦没完全解决：因为大家是”同时”开始数数的，翻倍归翻倍，他们的”两分钟”“四分钟”还是踩在同一个点上，只是撞车的次数变少、间隔变长而已——人一多，照样会成片地一起回来。

于是常客又补了一句：别死板地数整分钟。每个人在该等的时间上，再随手加一段零碎的、谁也猜不到多少的零头——你多等十七秒，我多等四十三秒。就这一点随机的错位，把原本挤成一团的电话，摊成了均匀的细水长流。老板娘那头，终于一个一个、从从容容地接了起来。

——到这儿你大概已经认出来了：这位常客教的，正是分布式系统里对付”重试风暴”的两件法宝——exponential backoff（指数退避）加上 jitter（抖动）。

这是什么、为什么重要 当很多 client 同时访问一个繁忙或刚恢复的服务，失败后立刻、且整齐划一地重试，就会形成 thundering herd（惊群）：重试本身制造的负载，反而让服务永远缓不过来。Exponential backoff 让每个 client 在连续失败后成倍拉长等待间隔，迅速压低整体请求速率；但纯粹的退避仍是”同步”的，大家步调一致，峰值依旧扎堆。Jitter 给每次等待加一段随机量，把这些本来 correlated（相关）的重试在时间轴上打散，削平尖峰。两者合起来，是 AWS、gRPC、几乎所有成熟 SDK 里 retry 逻辑的标准配方，也是保护服务、避免 retry storm 级联故障的基本功。

隐喻对应表

  • 那部一次只能通一个人的电话 → 容量有限的 service / shared resource
  • 想订座的客人 → 发起请求的 clients
  • “嗨嗨”忙音 → 请求失败 / 被限流（error / throttling）
  • 所有人卡着同一分钟重打 → synchronized retry，引发 thundering herd
  • 等待翻倍（1→2→4→8 分钟）→ exponential backoff
  • 每人额外加的随机零头 → jitter
  • 最后均匀、从容接起的电话 → 负载被摊平，服务得以恢复

There was only one small restaurant in town, and the owner kept a single telephone for reservations. The line could carry just one caller at a time.

One Friday evening, a dozen tables’ worth of guests all decided, within the same minute, to book a seat for that night. The first got through; everyone else heard nothing but the beep — beep of a busy signal.

The blocked callers were all very disciplined. They glanced at the clock on the wall and decided, “I’ll try again in one minute.” So exactly one minute later, every one of them pressed dial in the very same second. The first caller snatched the line again, and the other dozen collided head-on — busy signal once more. Minute after minute they stood in an invisible queue that no one could enter, choking the line to death. This vicious loop — everyone retrying together, everyone failing together — was worse than the line simply being busy.

Then a regular came up with a trick. He told them: if your first call fails, wait one minute; still busy, wait two; then four, then eight… every time you fail, double how long you wait. That way fewer and fewer calls pop up per unit of time, and the line can finally catch its breath.

But the trouble wasn’t fully solved. Because everyone had started counting at the same moment, doubling or not, their “two minutes” and “four minutes” still landed on the same instant — just colliding less often, at longer intervals. With a big crowd, they’d still come back in synchronized waves.

So the regular added one more rule: don’t count whole minutes so rigidly. On top of the time you’re supposed to wait, each of you tack on a little scrap of extra time that nobody can predict — you wait seventeen seconds more, I wait forty-three. That small random misalignment spread the once-clustered calls into a smooth, even trickle. At the owner’s end, the calls finally came in one at a time, calm and unhurried.

— By now you’ve probably recognized it: what the regular taught is the pair of tricks distributed systems use against “retry storms” — exponential backoff plus jitter.

What it is and why it matters When many clients hit a busy or just-recovered service and all retry immediately and in lockstep after failing, they create a thundering herd: the load generated by the retries themselves keeps the service from ever recovering. Exponential backoff makes each client stretch its wait multiplicatively after consecutive failures, quickly lowering the overall request rate. But pure backoff is still synchronized — everyone moves in step, so the peaks still pile up. Jitter adds a random amount to each wait, scattering these otherwise correlated retries along the time axis and flattening the spikes. Together they’re the standard recipe in AWS, gRPC, and nearly every mature SDK’s retry logic — the basic discipline for protecting a service and avoiding cascading retry storm failures.

Metaphor mapping

  • The phone that serves one caller at a time → a service / shared resource with limited capacity
  • The guests wanting a reservation → the clients making requests
  • The busy signal → a failed / throttled request (error / throttling)
  • Everyone redialing on the same minute → synchronized retry, triggering a thundering herd
  • Doubling the wait (1→2→4→8 min) → exponential backoff
  • The random scrap each person adds → jitter
  • The calls finally answered smoothly → load flattened, service recovers