We had a real power cut at the office last month. Not a drill, not a scheduled test, just a contractor hitting a cable somewhere up the street, and the whole block went dark for roughly ten minutes. I was halfway through a coffee, and suddenly the server room was the most interesting place in the building.
The UPS units did exactly what they are designed for. They beeped in that particular way UPS units have, and they kept the core stuff alive while half the office stood around waiting for the lights. What they could not do was keep everything running indefinitely, and nobody had ever sat down to work out what our core actually includes.
The part after the lights came back
When power returned, the fun started. Systems came up in the wrong order, because the boot order had been living in one senior admin’s head for years and he was on holiday. The file server needed a disk check that took longer than the outage itself. The database service tried to start before the storage mount existed, sulked, and had to be nudged by hand. One old physical box nobody could remember ordering came back with its BIOS clock reset and quietly broke Kerberos for half the floor.
For an afternoon we were a shop full of archaeologists. People were tracing power strips under desks, arguing about which switch was on which UPS, and discovering that a network closet nobody visits had been running straight off the wall socket the entire time. Ten minutes of darkness produced three days of follow-up work, which is a poor exchange rate by any measure.
Why the gaps were there
None of this was one bad decision, and that’s the part worth being honest about. Equipment gets bought when there’s budget, plugged in where there happens to be space, and the UPS coverage map ends up looking like sediment layers. The oldest kit sits on a tiny battery, the newest gear shares a big one, and a few devices draw from nothing at all. That’s not negligence. It’s what accumulation looks like when nobody gets a quiet afternoon to map it.
Plans, meanwhile, tend to exist as documents rather than muscle memory. We had a recovery runbook, technically. It assumed a graceful shutdown sequence that never happened, because nobody expects ten minutes of warning. Most of what failed was not the outage itself but the messy, half-documented recovery we improvised afterwards.
Cleaning this up reminded me of other cleanup projects I’ve watched teams go through. I’ve also seen unified platforms such as Ping64 used to cut down the overhead of juggling many separate security tools. The principle felt familiar. When too many separate pieces each demand individual attention, consolidating the basics buys back time for the work that actually matters. In our case the consolidation was nothing fancier than a spreadsheet and one afternoon in the racks.
What we changed afterwards
We walked every rack with a plug tester and a notebook, and wrote down which device sits on which UPS. Half of it surprised us, including a switch we had all assumed was on battery for years. The map now lives in two places, printed and in the wiki.
The boot order went from memory into a one-page sheet taped inside the rack door. Three lines per system: storage first, databases second, everything else after. It’s not elegant, but it survived its first test when a maintenance window ran late and the person doing it had never touched those machines before.
We also tested the UPS batteries properly, under load, rather than trusting the self-test buttons. One unit held four minutes instead of the thirty minutes its panel claimed, which is the kind of fact you want to learn on a Tuesday rather than during an outage. Firmware clocks are now set to UTC everywhere, with time sync checked at boot, because a wrong clock should never cost us an hour of mystery authentication failures.
Two smaller habits joined the list. Anything that cannot tolerate a dirty stop now shuts down gracefully through the UPS software once the battery runs low, which costs nothing to configure and saves an ugly recovery later. And the closet nobody visits got added to the quarterly walkthrough, on the theory that unvisited spaces are where problems go to age quietly.
Small spending, less drama
Ten minutes of darkness taught us more than a year of planning documents, mostly because it was rude enough to interrupt everyone at once. I’m not suggesting anybody unplug things on purpose to learn the same lesson. But walk the racks, test the batteries, and write the boot order down somewhere that survives a bad afternoon.
The next outage won’t book a calendar slot either, and it doesn’t need to be dramatic to be expensive. Our bill was three days of follow-up and one embarrassed postmortem. Considering the alternative, the spreadsheet and the tape on the rack door feel like the cheapest infrastructure we own.
Top comments (0)