Three years ago we took our depot Wi-Fi off shared passwords. Every handheld scanner and laptop now proves who it is with a certificate, checked against our own certificate authority before the device is let on. The certificates renew themselves every year without anybody noticing, which is how it should work.
The certificate authority was built in 2016 by a contractor, as a single server running Microsoft's certificate services, and its root certificate was given a life of ten years. Everything it issues is signed by that root, so nothing it issues can outlive it. When a device renewed during the root's final year, the authority quietly cut the new certificate short to end at the same moment as the root. By this summer every certificate in use, about eleven hundred of them, was due to expire in the same minute.
That minute arrived at twenty past two on a Thursday afternoon in September. Within the hour every scanner at our sixty one depots had dropped off the network, each one as soon as it next tried to reconnect. Goods in went back to paper. Laptops on the wired network carried on, so the first calls were about scanners, not certificates.
The recovery taught us the second lesson. Building a new root took an afternoon. Getting new certificates onto devices took far longer, because the scanners collect them from our device management service over the network, and the only network they knew was the one now refusing them. We put up a temporary password protected network at each depot, able to reach nothing but device management, and walked the scanners across a depot at a time. It took two and a half days.
Nobody had exactly forgotten. The contractor had written the expiry date down in 2016, in a document nobody had a reason to open for ten years, and nobody who had read it still worked here.
We now keep a register of every expiry measured in years rather than months: certificate authorities, the signing certificates behind our single sign on, our domain names and the support contracts on core equipment. Each has an owner and a rehearsal date a year before it falls due, because a replacement root has to be trusted by every device before the old one ends.
The longer something lasts, the fewer people remember that it is going to end.
– Serguey Shinder
Top comments (0)