Old Zhou had been carting goods out of River Valley Town for twenty years.

The road to White Stone Ferry, downstream, runs along the canal and passes under seven stone arch bridges. They were built in different centuries, they are not the same height, and Zhou had never measured a single one of them. He only knew the goods had to arrive.

That morning he stacked the crates as high as the cart would take, and on the topmost crate he wrote four words in chalk: do not unpack. That was his rule. A load travels whole or it comes back whole; nobody along the road gets to break his crates open and ferry them across in pieces. Split loads never reassemble properly at the other end, and pieces go missing.

First bridge, through. Second bridge, through. At the third, the bridge keeper put out a hand and stopped him.

“You won’t fit.”

“Then how high will fit?”

The keeper pulled a small slip of paper from his coat, wrote a number on it, and handed it over. “My arch takes seven foot two. No more.”

Zhou drove back to town, took two layers off the top, restacked to seven-two, and set out again. The third bridge let him through. The fifth stopped him — another slip of paper: six foot eight. Back to town, unload, restack, go.

After three round trips he had found what he actually needed: not the height of every bridge, but the lowest arch on the whole road. One number and the entire route opens up. He went home and chalked it on his own door frame — White Stone Ferry: six-eight — and loaded to that figure ever after, one trip, no drama.

He never wiped the chalk off. But he never quite trusted it forever either. Floods reroute the road; bridges get rebuilt and reinforced. So every so often he’d deliberately load a little high and try — in case the road had grown generous.

— The real trouble started with the new keeper.

That year the fourth bridge changed hands. The young man who took over had been told from above: don’t chat with strangers, don’t go handing out notes, keep it simple. So Zhou’s cart rolled into that dim arch as usual, the top crates scraped stone, the load came apart, and the carter and his mule backed out filthy and shaken — but with no slip of paper.

Zhou waited at the town gate for word. No word came. He didn’t know which bridge had stopped him or how much to take off. All he knew was this: small loads arrived, large loads left and were never heard from again. He tried, and tried, stacking to the same height every time, and every time the cart vanished into the same dark hole. He started to suspect the customer was lying to him.

It was the old clerk at the ferry office who finally solved it. He never bothered finding out which bridge was to blame. He simply put up a board at the town gate: no cart leaving here may load above six foot. Better to carry a little less on every trip than to have goods evaporate somewhere in the middle of the road.

— By now you’ve probably recognised it: Zhou’s cart is an IP packet, the chalked “do not unpack” is the DF (Don’t Fragment) bit, each arch height is a link’s MTU, the keeper’s slip of paper is an ICMP “Fragmentation Needed” message — and the young keeper who was told not to hand out notes is the thing every operator dreads: a PMTU black hole.

What this actually is

An IP packet crosses a series of links on its way to a destination, and each link has its own maximum frame size (MTU). The smallest value along the whole route is the Path MTU, and the sender doesn’t know it in advance. Path MTU Discovery (PMTUD) works like this: send at the local interface MTU with the DF bit set, meaning “do not fragment this.” Any router that finds the packet too large to forward and is forbidden to fragment it drops the packet and returns an ICMP Type 3 Code 4 message carrying “this hop only takes N bytes.” The sender shrinks and retries, iterating until it converges on the path minimum — then caches the result, with an expiry, because paths change.

Why it matters: the entire mechanism rests on one assumption — that the ICMP message makes it back. In practice, firewalls, security groups, and load balancers routinely drop all ICMP as a blanket policy. That produces the nastiest failure shape in networking: small packets (handshakes, pings, curl -I) all succeed, large packets (actual payload, TLS certificate chains, big responses) are silently discarded, and the connection just hangs with no error anywhere.

This bites hardest in container networking, where encapsulation like VXLAN, IPsec, or WireGuard eats tens of bytes of header per packet and drops the effective MTU from 1500 to 1450 or lower. Which is why the old clerk’s move — rewriting the MSS down during the TCP handshake (MSS clamping, --clamp-mss-to-pmtu) — is the standard defence: stop depending on a note coming back, and just load lower from the start.

Metaphor mapping

• Zhou’s cart — an IP packet
• How high the cart is loaded — packet size
• The chalked “do not unpack” — the DF (Don’t Fragment) bit
• The height of each stone arch — the MTU of each link
• The lowest arch on the whole road — the Path MTU
• The slip of paper handed back — ICMP Fragmentation Needed (Type 3, Code 4), carrying the next-hop MTU
• Three round trips of unloading and restacking — the iterative convergence of PMTUD
• Chalk on the door frame — the PMTU cache (kernel route cache entry)
• Occasionally loading high on purpose — re-probing after the cached PMTU ages out
• Floods and rebuilt bridges — routing changes that alter the path MTU
• The keeper who hands out no notes — a firewall or security group dropping ICMP
• Small loads arrive, big loads vanish — the classic PMTU black hole symptom: handshake fine, bulk transfer hangs
• The board at the town gate — MSS clamping (or simply lowering the interface MTU)