Comment Mountain of eggs, meet single basket... (Score 1) 13
Great set-up.
Great set-up.
Agreed. This is not on the one person that made a mistake with the config. This is on those that did not create a reliable process and those that did let that happen. Up to C-level. Reminds me of Boeing and some others.
Yes. The problem is the email failure mode: It is typically all-or-nothing, sometimes per sender. For monitored, expected communications, that is pretty good. For anything that must be delivered but nobody notices if it is not, that is catastrophic.
We need to start calling this "fuckup" or something. "Glitch" sounds way too much like nobody fucked up. But somebody, probably several somebodies, did.
Interesting. And about what I expected. A few years back, I came to the conclusion that MS might be doing tech now (including Azure) that they cannot actually manage or control. Too much tech debt stacked up, still no idea how to do clean architecture and design, management that does not care because the dollars keep rolling in, insufficient experience with dependable and secure software engineering, etc.
Obviously, long-term that is deadly because these problems are basically impossible to fix now. But since they have managed to lock in so many people, they will continue to do massive damage until their end.
Yes. A dangerous failure mode is a massive screw-up in itself. There no blame on MS for this incident. While MS is pretty incompetent and their products suck badly, that does not mean they are responsible for everything that goes wrong involving their products.
Email is typically pretty reliable, but it can fail completely and depends on the users noticing. That makes it unsuitable as alerting channel unless it gets closely monitored. The main thing email is is interoperable and convenient and quite cheap for what it offers.
Yes, clearly MS is poorly managed. But no German court will put any blame on them here.
It should also not just send an alarm but shut down the chlorination. That may become problem after a day or longer, while over-chlorination is a potentially deadly problem now. What they did is a really bad safety design made by complete idiots.
Indeed. A lot of incompetent assholes doing that these days. Blame technology when you were the one that screwed up the engineering and the configuration.
Even worse, the monitoring system should have defaulted to stop chlorination on error. Under-chlorinated water does not immediately become dangerous. Over-chlorinated water does.
This is just _really_ bad design and configuration of a dangerous system, nothing else. The MS outage is not at fault. It was just the last straw.
Yes. And probably a good red team would do the same for less money and in better quality.
The sad state of affairs is that most IT security sucks and that really cannot go on. If LLM forces advances in this space, that is a good thing.
You do not measure hacking effectiveness in time it takes to get in. You measure whether it gets in within a reasonable time (hours, says, weeks, does not really matter), how obvious the attack us, how much traces left, any way to identify the attacker, etc.
For an LLM, you also measure how expensive the attack was.
Makes sense, good old Microslop.
First, a 14% capacity reduction should never be enough to kill things, even with that Teams failover. They obviously went cheap on infrastructure.
Second, a cascade-failure is what you want to avoid at all cost and that means capacity, monitoring, robustness and tests. Apparently too expensive.
And third, don't they test their stuff under overload? Don't answer that, doing so would cost money.
All in all the usual mode of giving tech details to obscure that they screwed up massively. Again.
Yes. That is how you do it for critical networks. Apparently the network operator in question though that too expensive. Or maybe MS was down and they could not get communications and improvised. Which would of course be really gross misconduct.
Documentation is the castor oil of programming. Managers know it must be good because the programmers hate it so much.