At one of our aggregates yards in north Lincolnshire, every lorry crosses a weighbridge, and the weighbridge sends each ticket to our sales system at head office. For years it did that over an old copper broadband line that dropped several times a week. In the spring we moved the site to satellite broadband. Speeds went up fivefold and the drops stopped.
Two things depended on the old line's fixed public IP address. Our site to site VPN from the yard to head office was set up to recognise the yard by that address. And the cloud service the weighbridge maker runs, for software updates and ticket backups, only accepted connections from addresses on its allowlist, which included ours.
On the plan we bought, the satellite service, like most mobile networks, puts customers behind carrier grade NAT. Many subscribers share one public IPv4 address, and the address a site appears from can change without notice. The VPN would not come up at all. The weighbridge supplier helpfully added our new address to their allowlist, and that worked for six days, until the address changed and the tickets stopped again.
It was one of our network engineers who saw the bigger problem. The address we had asked the supplier to trust had never been ours. It was shared with an unknown number of other customers of the same service, so for those six days anybody else behind it could reach a system that believed they were us. We asked the supplier to remove it that afternoon.
The fix was to stop identifying the site by its address. The yard now builds its VPN outward to head office, authenticating with a certificate held on its router, which works whatever address the connection presents. The weighbridge reaches its supplier through that tunnel and out of head office's fixed address, which stays on their allowlist. We also listed every allowlist, at suppliers and in our own firewalls, that names a depot's address, so that the next change of connection starts with that list rather than discovering it.
An IP address feels like a property of a site. It is a property of the connection, and when you change the connection, you change who you appear to be.
– Serguey Shinder
Top comments (1)
This is a great write-up, and the last line nails it. The thing that makes CGNAT allowlisting so dangerous is that the shared address usually doesn't just have other people behind it, it also rotates, so an entry you add today can point at a completely different set of subscribers next week. For those six days you weren't trusting one stranger, you were trusting whoever the carrier handed that address to next.
The outbound cert-based tunnel is the right fix. The rule I'd pull out of it: an IP allowlist is a network control, never an identity check. If something has to be reachable from a changing connection, pair the address rule with something that actually proves who is calling, like mTLS or a signed token, so a shared or rotated address can't stand in for the caller.
One cheap habit that would have caught this sooner: look at the ASN behind an address before you trust it. A mobile or satellite carrier's egress range is a very different thing to allowlist than a fixed business line, and that difference is visible the moment you check.