Most of our depots have one network switch in a cabinet, and everything at the site plugs into it: the tills, the card terminals, the wireless access points, the cameras and the desk phones. The switches are managed from head office, and when a depot needs something changed, an engineer connects to it remotely and makes the change in a few minutes.
On the switches we use, as on many, that change takes effect at once in what is called the running configuration, which lives in memory. The switch only keeps it through a restart if somebody also saves it to the startup configuration. Forget that step and everything works perfectly, for as long as the power stays on.
In October a storm cut the power to our Boston depot for just under two hours. The UPS in the cabinet lasted forty minutes. When the power came back, the switch started from a configuration saved in 2024. Since then an engineer had moved the card terminals onto their own network segment, as part of our card security work, and never saved it. The terminals came up on a segment that could no longer reach the payment provider. The depot took cash and cheques for three hours while we worked out what the switch had forgotten, because the only complete record of it had been in its memory.
The following week we checked every switch we own. Fourteen of sixty one had a running configuration that differed from the saved one, some by a single line, two by several months of work. Every one of those changes had a closed ticket marked successful.
The fixes were simple. Our configuration backup now collects both the running and the saved configuration from every switch each night, and raises a ticket whenever they differ. A network change cannot be closed until that night's comparison shows the two match. And when we restart a switch on purpose, we treat it as a test rather than an inconvenience, because if anything comes back different, we would rather find out on a Tuesday afternoon than in a storm.
A change that only lasts until the next restart is not finished. It is waiting, and restarts tend to choose the worst possible moment.
– Serguey Shinder
Top comments (1)
I've seen this exact failure mode after a power cut: everything looks fine until the switch reboots and comes back with an old startup config. The fix you landed on is the right one — a change isn't done until it's saved and the running and startup configs match, and the ticket shouldn't close until that check passes. I also like treating a planned restart as a test; finding the mismatch on a quiet afternoon beats finding it during an outage.