500 milliseconds of strangeness
Apr 10th, 2024 by Jesper Kråkhede
At the end of March, a developer noticed that his SSH logins took about half a second longer than they should. Most of us would have shrugged. He didn´t. He investigated, and found a backdoor deliberately planted in xz utils, a compression library used in almost every Linux distribution. It had been inserted by a contributor who had spent more than two years building trust in the project.
If it hadn´t been found, it would have given whoever held the right key access to a huge number of servers. It was found because one person thought 500 milliseconds was strange.
That is the essence of detection. Every serious attack leaves traces that are just a little bit wrong. A logon at an odd hour. A process that uses more CPU than it should. A service account that suddenly talks to a new server. A DNS query to a domain nobody has seen before. Each of them is easy to ignore.
Technically, there are a few things that help. Know what normal looks like, so the strange stands out. User and entity behaviour analytics can do part of this for you. Keep the data long enough to compare. Make it easy for anyone, not only the SOC, to report something odd, and take the reports seriously. And give your analysts time to follow up on things that are not yet alerts.
The tools will find the obvious. People find the strange.
Never ignore the half second.