The Jade Ring Pond was round. An isolated hill sat at its centre, and a channel ran around the hill’s foot like a ring slipped over a stone. Along the channel stood nine mills. Every mill had a sluice, and every sluice opened into the pond.

Uncle Jing had kept those nine sluices for forty years. The rule was not complicated. On the hour, each of the nine mills carved its nine sluice readings onto a bamboo ledger — one mill, one ledger; one ledger, nine chapters — and the nine ledgers were combined into a single master account. With the master account, you knew which mill opened and which mill closed for that hour. Pond water does not recognize people. It recognizes only the master account.

For the first few years the nine mills lived in peace. Uncle Jing’s old method: every hour, each mill sent out eight riders, carrying its complete nine-chapter ledger, one copy to each of the other eight mills. Eight eights make sixty-four, plus nothing from home — seventy-two riders on the road.

The ledgers were thin and the horses were fast; an hour was enough. But Uncle Jing carried a complaint he could never name. The riders arrived early and late. The mills that received first combined first; the mills that received last combined last. The nine clocks differed by nearly half an hour, and the sluices trembled in sympathy. When the water trembled, the downstream waterwheels trembled. When the millstones trembled, the flour came out fine for a while and coarse for a while.

The real trouble came in the seventh year. Five new mills opened west of the pond. Nine mills became fourteen.

Uncle Jing arranged the riders by the old rule: fourteen mills, thirteen riders each, one hundred and eighty-two horses. The dirt road along the pond admitted one rider at a time. The queue ran from noon to sunset. The ledgers were still thin, the road still the same length — but the riders had doubled, because “every mill to every mill” multiplies. Double the mills, quadruple the horses.

By the fourteenth year there were twenty-seven mills on the pond. Uncle Jing stopped doing the arithmetic.

That spring his granddaughter Alu came home from the county school. She stood at the pond’s edge and watched the queue for a full hour. Then she said: “Grandfather, you are sending the wrong book.”

“The wrong book?”

“You make every rider carry the whole ledger. But to build a master account, a rider only needs to carry one chapter.”

Alu drew a circle in the sand and marked nine points on it.

“The first pass is reduce-scatter. Each mill sends exactly one rider, carrying exactly one chapter. Mill one carries chapter one, mill two carries chapter two, mill nine carries chapter nine — no duplicates anywhere. Nine riders mount at the same instant, leave nine mills at the same instant, ride the same direction. Nobody overtakes. Nobody blocks.

When a rider reaches the next mill, the miller asks no questions. He does one thing: he opens that chapter in his own ledger and adds his numbers to the rider’s. The chapter in the saddlebag stops being “my numbers” and becomes “my numbers and yours,” then “mine and yours and his — thirty-five hundred acres of numbers.” The rider does not stop. He goes on, station by station.

After eight stations the rider returns to the mill he started from. The chapter in his saddlebag is now the true sum of nine mills’ nine numbers. That mill is the owner of that chapter, from then on.”

“And the others?”

“Eight riders return to eight mills, each carrying one chapter’s true sum. Nine mills, nine chapters, every chapter owned. That is the whole of the first pass.”

Uncle Jing stared at the sand for a long while. “The second pass?”

“The second pass is all-gather. Now each mill holds one true chapter and lacks the other eight. Still nine riders, still that circle, still eight stations. Only this time there is no arithmetic in the saddlebag — the rider carries a chapter already summed. At the next mill, the miller copies it and hands it back; the rider rides on. Eight stations later, all nine chapters sit on all nine shelves.”

“Copying is heavier than adding,” said Uncle Jing.

“Heavier, yes. But it stays heavy in the same amount. Look.” Alu wrote in the sand. “Both passes together, one rider covers sixteen stations. Nine mills is eight plus eight. Fourteen mills is thirteen plus thirteen. Twenty-seven mills is twenty-six plus twenty-six. Each new mill adds two stations of road, and nothing more.

But what the saddlebag holds at each station gets thinner. At twenty-seven mills, one chapter is one twenty-seventh of the whole ledger. So the total weight a single mill sends out over both passes is two × (mills − 1) / mills ledgers — and the more mills there are, the closer that gets to two, and it never passes two.

Your old method? Each mill carried (mills − 1) full ledgers. Eight at nine mills. Thirteen at fourteen. Twenty-six at twenty-seven. It does not stop at two. It climbs until the horses’ backs give out.”

The flood season came early that year. On the night of the seventeenth of the third month the pond rose fast. The water level changed hourly, and the master account had to be rebuilt every quarter hour. Uncle Jing began calculating the old method’s riders, and put down his brush halfway through.

He did it Alu’s way.

Twenty-seven mills, fifty-four riders, a single thin sheet in each saddlebag. That night the lanterns along the pond lit one after another. The riders mounted together, arrived together, added together, turned back together. The first pass finished at first light. The second pass finished with the sun well up. For the first time, the nine clocks agreed, and the sluices did not tremble once.

Standing at the pond’s edge, Uncle Jing understood something he had missed for forty years. All that time he had been trying to make the horses faster. What he should have done was make the saddlebags lighter.

Once, a rider from the third mill broke his horse’s leg in the mud, and the whole ring waited for him at the next mill. Uncle Jing did not curse. He simply ordered two spare post horses. He understood: on this ring, everyone waits for the slowest one.

By now you have probably recognized it: this is Ring All-Reduce.

What it is

Ring All-Reduce is the collective-communication primitive behind multi-node training: take the vectors each of N workers holds (usually gradients), sum them into one total, and give every worker the complete result. It splits into two halves.

The first half, reduce-scatter: cut the vector into N chunks, worker i owning chunk i. Over N−1 steps, each chunk travels the ring once, picking up that worker’s slice of the sum at every hop, until each chunk’s true total rests with its owner.

The second half, all-gather: the finished chunks travel the ring again and are copied out by each worker, so that after N−1 steps everyone holds everything.

Total steps: 2(N−1), each moving only V/N of the data. So the volume a single worker sends is 2(N−1)/N · V — approaching 2V, independent of N. That is why it is called bandwidth-optimal.

Why it matters

The naive approach is all-to-all: every worker ships its full vector to the other N−1 workers, costing (N−1)V per worker and growing linearly with cluster size. Scale from 8 GPUs to 512 and communication grows 64×, exactly as much as compute — so training stalls on the network.

The ring algorithm pins per-worker traffic near 2V. Scaling out costs you a longer road, not a heavier load.

Two costs come with it. Every worker is strictly synchronized, so one slow worker stalls the whole ring (straggler sensitivity). And 2(N−1) steps of latency can overwhelm the bandwidth win on small messages, which is why small models sometimes run slower on a ring. That is why real clusters mix it with tree and NCCL hierarchical algorithms chosen by topology.

Metaphor mapping

• Nine mills → the N workers (GPUs/nodes) in a cluster
• One mill’s nine-chapter water ledger → each worker’s local gradient vector, sliced into N chunks by position
• The master account every mill needs → the semantics of all-reduce: everyone receives the sum
• The old method: eight riders per mill carrying whole ledgers → all-to-all, (N−1)V per worker
• Mills double, riders quadruple → traffic grows linearly with N; adding machines makes things slower
• The rider carrying only one chapter → chunk ownership in reduce-scatter
• The rider adding each mill’s figure as he rides → accumulate-in-flight; raw data never leaves home, only the running sum travels
• Eight stations later the chapter’s true total belongs to that mill → reduce-scatter done, each worker holds exactly one fully reduced chunk
• The second pass only copies, never computes → all-gather propagates reduced chunks with no arithmetic
• Sixteen stations = eight plus eight → total step count 2(N−1)
• Each station carries 1/N of a ledger; total weight approaches two ledgers → per-worker traffic 2(N−1)/N·V, approaching 2V regardless of N
• Mounting together, arriving together, sluices steady → uniform step counts, naturally aligned completion
• The third mill’s broken horse stalls the whole ring → straggler sensitivity, the inherent cost of a synchronous ring
• Uncle Jing ordering spare post horses → engineering mitigations: redundancy, timeouts, hierarchical topologies