Forgot your password?
typodupeerror

Comment reminds me of a power grid "black start" (Score 4, Interesting) 38

Those are a pain because the power grid isn't star topology, it's web. This makes it difficult to isolate a few individual nodes to start up first. And once you DO get an "island" working again, you have to slowly add on other isolated segments. If you add too much at once, the whole thing comes crashing down and you have to start all over again.

It sounds like MS's "automated recovery" was set up to fail over to healthy nodes, which overloaded them and took them down too, just like you see when a power grid fails.

Everything is near max capacity when something trips, causing load to fail over to another path, which then overloads it, causing the entire combined larger load to fail over to another segment and quickly trip it, and they go down like dominos. Basically what happens if you fix one gateway and bring it back up, it gets mobbed and immediately "falls over".

Microsoft

Microsoft Blames Monster Email Outage On Software Defect Introduced In July Fix (theregister.com) 38

Microsoft says last week's massive Microsoft 365 outage was caused by a software defect introduced in a July update that remained dormant until a Teams failover and a simultaneous 14% reduction in gateway capacity triggered a cascading failure. More than 80% of the affected gateway servers became unresponsive at the outage's worst point, and because the months-old change could not simply be rolled back, engineers spent roughly 15 hours developing and deploying a patch that restored the previous code path. The Register reports: According to a preliminary post incident report published by Microsoft on November 15, the defect was introduced in July 2026. The update looked fine at the time, Microsoft says, having passed load testing and been deployed without any impact. However, the testing wasn't good enough. Microsoft's report states: "Our load testing tended to focus on weaker hardware with fewer cores." The defect caused "lock contention inside an internal Windows process which caused the machine's resources (CPU & memory) to be exhausted."

"The contention issue is worsened on machines with more cores and newer hardware." The fix containing the defect was prepared specifically for Microsoft 365 network gateway servers. Then two things happened at once. Teams failed over to the "primary" gateway that supports Exchange Online, creating a surge in load that was not, in itself, particularly unusual. At the same time, almost 14 percent of server capacity for the networking gateway was out of service, which wasn't a problem by itself either: it was on its way to being restored and, according to Microsoft, "the service had sufficient capacity headroom." Together, they pushed the gateway into the defect and chaos ensued. It was a classic cascading failure scenario. When a server loses connectivity, traffic gets routed to the next healthy endpoint. Which encountered the same problem. So traffic was rerouted again and... you can see where this was going. Within two hours, the defect was encountered across Microsoft's fleet. "At its lowest," the report states, "more than 80 percent of servers were unresponsive."

So, what to do? Bringing servers back to service didn't help -- they were immediately overwhelmed and fell over again. "Our normally automated recovery processes were not designed to restore service from this broad unresponsive state." A straightforward rollback was also not possible because the defect was months old. Instead, engineers developed patch for the patch or, in Microsoft's words, "an update that returned the code path to the previously known good state in Windows." The patch was applied to groups of servers and service was gradually restored.

Comment Re:It's simpler than that, (Score 1) 82

it's a very popular feature on Macintosh computers. I worked for years as a service tech, and did hundreds of migrations. Apple has always made it very easy... I can remember plugging two computers together with a SCSI cable and running the migration assistant! I can still remember the days when windows users had to buy one of our "pc migration kits", that was a box with software and a special cable - USB-A on BOTH ends, and a plastic box in the middle. We sold quite a few of those.

Firewire on the mac was a breeze. Just launch the (included) migration assistant on the new computer, boot up the old computer in "firewire target disk mode" (would do that even if unbootable), connect computers with a firewire cable, and ALL your user data/account migrate, along with most or all of your fully installed apps. Apps with twitchy anti-piracy measures often needed to be reinstalled but not always. (and sometimes just required re-activating)

Apple did upgrade the hardware to do USB-target-disk-mode for awhile, but unfortunately Apple has dropped target disk mode in the newest systems. That's a shame, it was an INCREDIBLY useful tool for the technicians. I could plug in a hard drive that wouldn't even boot to another computer for remote repair or data recovery, never needed to pull a hard drive unless it was physically damaged. Nowadays you have the same migration assistant but it must be launched on both ends and you use an ethernet cable, so you can't just browse the other computer's hard drive if it won't boot.

Comment Re: I've said it before (Score 1) 185

"Generally this isn't true, instead it is stored in large quantities of stock or other assets"

It is still used as a leveraged mechanism with which to purchase things. How the hell do you think they live in big fucking houses and have chauffeurs? Thus it is effectively cash.

And with that cash they buy power, hoard resources, and generally fuck over the human population.

You're absolutely blind to how stock gets used. You're the one trolling, here.

Slashdot Top Deals

Lawrence Radiation Laboratory keeps all its data in an old gray trunk.

Working...