When security becomes the outage
Jul 22nd, 2024 by Jesper Kråkhede
Last Friday a security vendor pushed a faulty content update to its endpoint protection. About 8.5 million Windows machines crashed and couldn´t restart. Airlines, hospitals, banks and stores stopped. Many machines had to be fixed by hand, one by one, and on machines with BitLocker someone had to type in a 48 digit recovery key.
It wasn´t an attack. It was the protection itself that broke.
It reminded me of an outbreak many years ago, which we called the yellow snake. The worm spread through the infrastructure of the antivirus itself, the one system that every machine trusted and that could reach everything. The tool that was supposed to protect the network was the fastest way through it.
The technical lesson is the same now as then. Security software runs with the highest privileges, often in the kernel, and on every machine at the same time. That makes it extremely powerful, and extremely dangerous when something goes wrong.
What can you do? Ask your vendors how they roll out updates, including content and signatures, not just new versions. Do they use rings, canaries and automatic rollback? Can you control the pace yourself? Make sure you can recover at scale: BitLocker keys stored and reachable, a way to boot machines into recovery remotely, and a plan for the machines in remote locations. And don´t forget the servers, especially the ones that run the recovery.
Trust your tools. Just make sure you can survive when they fail.