The Harbour Hotel had three hundred rooms, and a telephone at every bedside. Not one of them could reach the world outside. Room numbers meant something only inside the building — tell a man in town “I’m in 412” and he’d only ask you: “412 of what?”

The world outside knew a single number: the Harbour Hotel. Every voice going out or coming in had to pass through one hot little room. In it sat Aunt Lin, facing an upright wooden board studded with a row of brass pegs, each peg wired to a line running into the city. A ledger lay open at her elbow, always.

The rule was simple. A guest in 412 wanted the fish market: he rang the little room first. Aunt Lin chose a free brass peg — say the thirty-seventh — pushed it home, and wrote a line in the ledger: peg 37, room 412, calling the fish market. Only then did she put the call through. The lad at the fish market picked up and heard “Harbour Hotel, peg thirty-seven.” He would go his whole life not knowing a room 412 existed.

When the reply came back, it could name only “the Harbour Hotel, peg thirty-seven.” Aunt Lin bent over the ledger, found the line, and rang 412. Anything not in the ledger, she hung up on — however urgent the caller sounded, however legitimate the business. She had one answer: “I’ve no record of that here.”

There were not many pegs. But Aunt Lin was shrewd: peg 37 might be carrying the fish market and, at the same moment, a second call out to the cloth merchant — one peg per destination, and no two collide, so no conflict at all. Which is why three hundred rooms could all be talking at once on an ordinary day and nothing ever went wrong.

What went wrong, went wrong on temple-fair day.

That morning the whole hotel seemed to have agreed on it: everyone wanted the same place — the opera ticket office in the south of the city. The first guest took a peg, the second took another, then a third, a fourth. By noon the ledger’s “calling the ticket office” column was full, every brass peg on the board spoken for against that one destination. Then 251 rang down wanting the ticket office, and Aunt Lin’s hand came up empty. All she could say was: “Engaged. Try again later.” Yet looking up at her board, she could see plenty of pegs sitting idle — every one of them still good for the fish market, the cloth merchant, the docks. Only not for the ticket office.

Worse, plenty of guests had hung up by then, and still the pegs didn’t come free. Aunt Lin’s long habit was this: when a guest set down the receiver, she did not strike the line out at once. She let it hang on the page a while longer, until the sand-glass had run through, and only then cleared it. What she feared was a late reply — if the “right you are, delivered tomorrow” from the other end came trailing back after the line was gone, that message would never find room 412. And so, all that afternoon, half her pegs were held by lines already finished but not yet dared struck out. Nobody speaking, and still no room on the board.

By evening the ledger itself had run out of pages. A ledger has only so many. A new call could get no line at all, and Aunt Lin hadn’t even the chance to say “engaged” — the bell rang, nobody answered, and it simply fell into nothing.

— By now you’ve probably recognized it: Aunt Lin is a gateway performing SNAT (a NAT Gateway, an egress IP, the firewall in front of your cluster), the ledger is the conntrack connection-tracking table, and the brass pegs are source ports.

What it is: private addresses cannot travel the public internet, so the gateway rewrites the source address and source port of every outbound connection to its own public IP plus a temporary port (an ephemeral port), and records a five-tuple in its connection-tracking table: protocol + srcIP + srcPort + dstIP + dstPort. Return packets are matched against that entry, rewritten back, and delivered to the true originator. Packets with no matching entry are dropped.

The crux is that uniqueness is computed over the whole five-tuple: the same port on the same public IP can serve many different destinations without conflict — but toward one specific destination IP and port, there are only about sixty-odd thousand ports to go around. So when a great many Pods reach for the same downstream at once (the same registry, the same S3 endpoint, the same third-party API), ports are exhausted against that one destination. AWS NAT Gateway caps out around 55,000 simultaneous connections per unique destination; past that you get ErrorPortAllocation — while the gateway’s overall bandwidth and connection count still look comfortably idle.

Why it matters: this is the classic shape of “inexplicable intermittent timeouts” on OpenShift/ROSA egress. The symptom isn’t “the network is down”; it is that only connections to one particular place fail, and everything else is fine — and not failing steadily, but failing whenever traffic spikes. The thing to look at is the ErrorPortAllocation metric and nf_conntrack_count, not a packet-loss graph. The remedies follow directly: attach more egress IPs (multiplying the port pool), route hot downstreams through a VPC Endpoint to bypass NAT entirely, use keep-alives and connection pooling to need fewer connections at all, and shorten timeouts where appropriate so ports come back sooner. And don’t forget: NAT is stateful — return traffic must come back through the same gateway. If the path is asymmetric, there’s no line in the ledger, and the packet is dropped in silence.

Metaphor map:

• Three hundred room numbers → private addresses, meaningless outside their own network
• The world knowing only “the Harbour Hotel” → the gateway’s public IP (the post-SNAT source address)
• Aunt Lin and her little room → the gateway doing source NAT (NAT Gateway / egress node / firewall)
• The row of brass pegs → the pool of available source ports (~64K ephemeral ports)
• Each line in the ledger → one entry in the conntrack table
• “Peg N, room R, calling P” → the five-tuple proto + srcIP + srcPort + dstIP + dstPort
• The fish market hearing only “Harbour Hotel, peg 37” → downstream sees the gateway as the source; the real client is invisible
• Hanging up on anything not in the ledger → inbound packets with no matching entry are dropped — NAT’s statefulness, and why asymmetric routing breaks
• One peg serving both the fish market and the cloth merchant → ports are reusable so long as the destination differs
• The whole hotel calling one ticket office → many clients hitting the same destination IP and port concurrently
• Pegs gone while other destinations still work → port exhaustion happens per destination (ErrorPortAllocation)
• Lines left hanging after the receiver is down → TIME_WAIT / idle timeout: entries linger, so ports turn over slowly
• Fearing the late reply would find no one → entries are held to handle delayed and retransmitted packets correctly
• The ledger itself running out of pages → the conntrack table hitting nf_conntrack_max; new connections dropped silently