The wedding banquet was set up in the courtyard in front of the ancestral hall. Forty tables. Old Zhou ran the floor.

Along the east wall stood forty small stoves — one stove per dish, one cook per stove. A dozen runners wove between the stoves and the tables with round trays. The rule was the old rule: forty dishes to a table, and nobody eats until all forty have landed. One dish missing and the whole table sits there watching the ones that already arrived go cold.

Old Zhou was pleased with himself that first year. He had done the arithmetic. A cook turns out a dish in one stick of incense on a good run, two on a bad one, and maybe once in a hundred times something drags it out to three. Once in a hundred — he told the family it was nothing to worry about.

At the first banquet, seventeen of the twenty tables started half an hour late.

The family asked him why. He went down the line, cook by cook, all forty of them, and every one of them said the same thing: my dish wasn’t the problem. They were all telling the truth. Nearly every dish that night came off the fire in one stick of incense. But a table needs forty dishes, and if any one of them lands on that once-in-a-hundred, the whole table waits for that one. Forty cooks, each missing once in a hundred — the odds of a table getting through clean weren’t even two in three.

That was when Old Zhou saw where his arithmetic had gone wrong. He had been reckoning how fast a cook was on average, but the hour the table starts eating is never set by the average cook. It’s set by the slowest one. And the more dishes on the table, the harder that slowest one is to dodge.

The second year he changed his method.

First, he stopped asking cooks whether they were having a good night. Instead he posted an apprentice at the courtyard gate with one job: write down how long every single dish took, from the order going out to the plate landing. A banquet produced more than a thousand lines. He didn’t look at the average. He laid the lines out fastest to slowest and read only the few dozen at the very bottom. After enough banquets he started seeing the shape of it. The stragglers were rarely a lazy cook. Sometimes forty stoves were all drawing from the one well in the middle of the courtyard, and there was a queue at the rope. Sometimes a cook had to stop after two dishes to scrape the wok and bank the coals — necessary work, but it ate half a stick of incense. Sometimes a table of nine was jammed up ahead in the line and everything small behind it simply waited. Slowness was never random bad luck. It was the shared things all crowding into the same moment.

Then came the trick he became known for.

These days, once the order goes out, Old Zhou doesn’t watch the cooks at all. He keeps a row of bamboo tallies on the tray rack, one per dish. When a tally goes in and doesn’t come back — not the instant it goes in, but once it has clearly fallen behind the rest of the tallies on the rack — he raises a hand and calls an idle runner: the spare stove at the west end, same dish, one more.

So two cooks are making the same dish. Whichever comes off the fire first goes to the table, and the other one gets waved off.

A new runner said it was firewood thrown away. Old Zhou told him to count it himself. Over a thousand dishes in a banquet, and fewer than fifty ever got a “one more” — because he never sends two from the start, only after that dish has already fallen behind almost all of its peers. Less than a tenth extra firewood. Seventeen late tables became one or two.

Later he added two more rules. One: the spare stove and the original stove each know about the other, so whichever picks up the second ladle first shouts across the courtyard, and the other drops it immediately — no sense tying up two woks. Two: he keeps a ledger. Any cook who lands in that bottom few dozen three banquets running comes off the roster for a few days and gets only the side dishes that never reach the banquet table, until a few quiet runs prove he’s steady again.

— And by now you’ve probably recognized it: what Old Zhou is fighting is tail latency amplified by fan-out, and his “one more” is the hedged request from Google’s The Tail at Scale.

That winter the family wanted to be extra safe and asked him: why not just cook every dish twice? Old Zhou said no. Forty stoves share the same coal and the same well. Double the work and you double the queue at the rope, and then everyone is slow — instead of one dish dragging the table, every dish drags.

The cure for the slowest one can never be making all of them slower.

What this is

A request fans out to many backends — forty shards, forty parallel RPCs, forty pods — and the response can’t be assembled until all the sub-requests come back. Overall latency therefore equals the latency of the slowest one. Even if each backend takes a slow path only 1% of the time, the chance that at least one of forty is slow is close to 33%; at a hundred, it’s 63%. That’s tail latency amplification.

Why it matters

• Under fan-out, mean latency has almost no diagnostic value. You need p99 / p99.9. Users feel the tail, not the average.
• The tail is usually structured, not random: shared-resource contention, background work (GC, compaction, log rotation, CPU throttling), head-of-line blocking, cold caches. Finding the structure beats adding machines.
• The main mitigations: hedged requests (wait until p95, then send a duplicate, take the first, cancel the other), tied requests (replicas notify each other; first to start wins), micro-partitioning with selective replication, and probation for persistently slow replicas.
• The whole trick of hedging is the threshold: duplicate only the small slice past p95 and the extra load stays in the single-digit percents. Unconditional duplication drives the system toward saturation and makes the tail worse.

Metaphor mapping

• forty dishes to a table, nobody eats until all arrive → fan-out request; the response waits on every sub-request
• each cook slow “once in a hundred” → a single backend’s p99 slow path
• seventeen of twenty tables late → tail latency amplification, 1 − 0.99⁴⁰ ≈ 33%
• Old Zhou’s first-year arithmetic about the average cook → mean latency, a misleading metric under fan-out
• the apprentice at the gate timing every dish → per-request latency instrumentation
• reading only the bottom few dozen lines → p99 / p99.9 percentiles
• the shared well, scraping the wok and banking coals, the table of nine jammed ahead → resource contention, background maintenance (GC / compaction), head-of-line blocking
• calling “one more” only after a tally clearly falls behind → hedged request with the threshold at p95, not immediate duplication
• first off the fire goes to the table, the other is waved off → take the first response, cancel the remaining replicas
• less than a tenth extra firewood → the threshold caps hedging overhead at single-digit percent
• the two stoves shouting across the courtyard → tied requests, cross-server cancellation
• three bad banquets and off the roster for a few days → latency-induced probation, pulling a slow replica out of service
• “we can’t cook every dish twice” → unconditional redundancy exhausts shared capacity and worsens overall latency