Power Grids On 5 Continents Pulse Every 60s. Scientists Think They Know Why (sciencealert.com) 53
"ScienceAlert is reporting that researchers may have finally determined why, for many years, power grids around the world have been pulsing every 60 seconds," writes longtime Slashdot reader fahrbot-bot. The likely culprit? Uninterruptible power supplies running routine battery checks. From the report: Every minute, some uninterruptible power supplies perform routine battery checks, creating small but measurable surges in electricity demand. And it's like clockwork. "Even benign control routines can become systemically relevant when many devices act coherently," writes a team led by physicist Dirk Witthaut of the University of Cologne, Germany, in a paper published in Nature Energy. "Unintended synchronization can generate recurrent and potentially amplified oscillatory behavior in large-scale cyber-physical energy systems."
To figure it out, Witthaut and his colleagues analyzed high-resolution electricity measurements from buildings at the Forschungszentrum Julich research campus in Germany. [... they found that] the 60-second pattern appeared in four buildings, all of which contained uninterruptible power supplies. None of the buildings they measured without these systems showed the same pattern. Following the electrical signals more closely, the researchers detected tiny bursts of current from the batteries at precisely one-minute intervals. They were able to trace the resulting fluctuations through one building's electrical system, all the way to the connection with the public high-voltage grid.
The manufacturer subsequently confirmed the culprit's identity -- routine battery self-tests, conducted automatically by the systems' management software. But why would equipment in different buildings, let alone different countries, be doing this at the same time? The answer appears to lie in the clocks. Digital devices can synchronize their internal clocks using external time signals, meaning that independently operating systems can end up performing their scheduled tasks together. Individually, these small checks are insignificant. But when enough devices act in synchrony, their combined electricity demand can produce a measurable effect across an entire power grid.
To figure it out, Witthaut and his colleagues analyzed high-resolution electricity measurements from buildings at the Forschungszentrum Julich research campus in Germany. [... they found that] the 60-second pattern appeared in four buildings, all of which contained uninterruptible power supplies. None of the buildings they measured without these systems showed the same pattern. Following the electrical signals more closely, the researchers detected tiny bursts of current from the batteries at precisely one-minute intervals. They were able to trace the resulting fluctuations through one building's electrical system, all the way to the connection with the public high-voltage grid.
The manufacturer subsequently confirmed the culprit's identity -- routine battery self-tests, conducted automatically by the systems' management software. But why would equipment in different buildings, let alone different countries, be doing this at the same time? The answer appears to lie in the clocks. Digital devices can synchronize their internal clocks using external time signals, meaning that independently operating systems can end up performing their scheduled tasks together. Individually, these small checks are insignificant. But when enough devices act in synchrony, their combined electricity demand can produce a measurable effect across an entire power grid.
Smdh (Score:2, Interesting)
Re:Smdh (Score:5, Informative)
The straightforward solution is for the UPS manufacturers to jitter the test time and it becomes a total non-issue.
From TFA:
Fortunately, the solution may be relatively straightforward, at least for big batteries and backup systems. Rather than having thousands of devices perform the same task at the same moment, manufacturers could introduce small variations into their software schedules, preventing the minutely pile-up.
Indeed, they have already contacted the manufacturer of one of the systems they identified, and the company issued a software update that relaxed the strict periodicity of its system checks.
Re: (Score:2)
Re:Smdh (Score:4, Insightful)
Dampen your Flux, Doc. (Score:2)
..And at some point in time, a single battery was added which was the last straw to break the camel's back.
Son of a, so that’s why my Flux Capacitor won’t cycle?
Where the hell am I gonna find a harmonic time dampener for an ‘81 DeLorean at this hour..
Not just UPS? (Score:1)
Lots of stuff on cron starts on *:00
Default for Windows scheduled tasks is not random delay (I do random delay for related reasons, but I'm sure most don't bother).
Nowadays lots of computers have a big difference idle power vs load power. So imagine if very many CPU/LLM intensive tasks are starting at the 00 sec mark.
Of course lots of UPSes "coincidentally" happen to have computers on them.
Re: (Score:2)
Lots of stuff on cron starts on *:00
At least cron weekly jobs run at a somewhat arbitrary time. Systemd weekly timers all run at midnight on Monday. That includes RHEL's fstrim timer. That's just great when you have 100s or 1000s of VMs starting a filesystem TRIM at the exact same seconds. Really kills the backend storage. /rant
Re: Not just UPS? (Score:2)
"[Timer]
OnCalendar=Sun *-*-* 00:00:00
RandomizedDelaySec=3h"
Re: (Score:2)
Microsoft used to use predefined NTP servers to set users' clocks, no provision for spreading the load and no consultation before they did it. Maybe millions of XP clients every day, and large numbers of them at pretty much the same time.
What could go wrong?
Oh.
Re: (Score:2)
What could go wrong?
You could forget to do load balancing, e.g. with WLBS nee Wolfpack.
Re: (Score:2)
They don't need an accurate clock. They just all need power restored at the same time. Assuming the timer is from first power on.
On the other hand, I don't know if it boots up and starts the clock when power is restored.
Re: (Score:2)
Yes, I know. That it ever happened is the bigger problem. The normalization of deviancy of customers being treated as nonconsensual beta testers is also a problem.
Uh, that’s kind of a two way problem. There’s a reason the new kitchen toaster needs voice control. Gen Touch N’ Tell tended to turn the task of building a better idiot into some kind of industrial goal in need of a stock ticker. Which was quite non consensual for most of society learing just how bad it is.
DR planning the battery cycling on the battery-backed DR device, isn’t a born skill. It’s a boring-ass one. Screw that. Just give it to Dewitt Fore-Me again, says the co
Re: (Score:2)
Exactly how would you test for this when it's an effect you're not going to see until you have installations measured in the tens of thousands?
Re: (Score:1)
Re: (Score:2)
The problem is that the "system" is much larger than the designers expected. The system is the complete set of buildings with UPS installed (thousands), interconnected through the power grid. For the designer, the system was just a single installation with no consideration of other installations.
The effects just aren't noticeable until you get to a large number of installations. No amount of local testing will find the issue. Just like in programming, no amount of unit tests will find all problems - there a
Re: Smdh (Score:2)
Re: (Score:2)
Well, THEY USED TO think of it, because having tons of tasks starting at once tended to bottleneck the hard drives.
Re: (Score:2)
I'd use a random starting offset instead. So then the average is spread all round the 60 second cycle.
Re: (Score:1)
It would make be question the reliability of those UPS systems if they can't get basic stuff like that right. Randomizing anything where multiple units doing it at the same time could be an issue is pretty basic engineering.
Re: (Score:2)
Adding jitter may be obvious but it's wrong because it's overly complicated. You don't need a single instantiation to change from cycle to cycle, you just need different instantiations to be evenly distributed.
Least complicated might be a timer not synchronized to external time, but that may still be dependent on some other external conditions. Next best would be a offset(skew) set once, e.g. serial number mod 60.
Re: Smdh (Score:2)
Yeah, honestly don't know why you'd choose wall time for this sort of scheduling, instead of uptime or some other metric.
Pedantry and meta-commentary on "Scientists" (Score:4, Insightful)
Perhaps the time is neigh to differentiate between samples of scientists and their populations.
Additionally, I would wager research findings are likely a more enticing draw to the slashdot rabble than what "Scientists Think"
Re: (Score:2)
It's a physicist, commenting on a software problem.
His findings are about as relevant as the janitor's.
It's a fucking bug. He should be fired for turning this into an opportunity for publishing churn.
one man's pedantry is another man's lesson-learned (Score:2)
It's useful as a "lesson learned". Scientists and facility managers in my organization traced strange instrument readings (maybe their FIB-SEM) back to cross-talk in a circuit-panel ground. I think there was a UPS involved in that one, too.
Re: (Score:2)
Apologies in advance, my petty ocd sensibilities were triggered by the headline.
Perhaps the time is neigh
Bad Horse [fandom.com], is that you?
A better explanation (Score:2, Funny)
Re: (Score:3)
studies of women who live together, whose menstrual cycles syncrhonize over time
I just want to point out that this widely cited study from the 70s has been thoroughly debunked and is not true.
Coherence (Score:5, Interesting)
When I was working for a startup in the early 2000s, we had our own servers. We were growing, so were adding servers at a pretty good clip. After one particular expansion, all of a sudden, Saturday night at 3 am, we blew breaker. No one had a good explanation, but the sysadmins were not happy about needing to go to the office in the middle of the night to fix things up.
A week later, the same thing happened. I suggested that it was the RAID check kickoff that might be doing it. And, indeed, it was. A couple of dozen servers all seeking their drives at once was too much for the circuits they were on. Staggering the startup time fixed it.
Coherence can be good, like when you're rowing crew, or bad, like when you have dozens of drives all drawing max power. Or, from the summary, gazoober UPS systems doing a battery check.
Re:Coherence (Score:4, Interesting)
Having corporate IT experience from administration to manufacture of banking systems, the system is supposed to be on a breaker that covers its maximum possible load, and such load be used in the calculations for sharing breakers. Wherever you worked was doing it wrong. The added amps from drive actuation should have been in the total breaker calculation along with maximum memory and CPU usage.
Simply staggering the array check time to slip by meant your circuit breakers were already on-edge without a safe margin, but my money is on your employer never fixing the problem after it was found.
Re: Coherence (Score:5, Informative)
Maximum amperage draw for 1/10th of a second is not usually taken into account, breakers don't usually blow that fast because of a current spike. (Those numbers are usually not on the specification sheet either.)
The maximum load is calculated over a period of time, not a very short period of time. It is in this fuzzy gap that issues like this arrive.
Re: (Score:2)
Breakers are analog devices. They have a trip curve which specifies the amount of current it will take to trip them. This is to handle inrush current as well as perform their duty to protect the wiring behind the walls.
The trip curve is time to trip versus current. And it can be surprisingly long -
Re: Coherence (Score:2)
Motor starting inrush current is massive compared to the running current, but it happens so briefly that it *usually* doesn't matter. An AC unit that runs at 8-9 amperes may take 90-100 amps or more to start, causing blown breakers, or even main breakers if the system is being taxed by another high current job like running an oven or charging a car. Nobody runs a 100 amp circuit to an AC system that calls for a 15 amp circuit, and neither does the unit have lugs big enough to accommodate that kind of wiring
Re: (Score:2)
Saturday night at 3 am, we blew breaker.
I spent decades managing data centers. PDU distribution should be designed to never pass 80% usage at peak draw.
Re: (Score:2)
Should being the key word, but we should have competent people in charge at all levels, but in many companies, the least competent are put in charge, and those who know what they are doing are limited in what we can control.
Blast from the past (Score:5, Interesting)
Lessons from the physical world - 1831 bridge fail (Score:5, Informative)
Lessons weren't learned 180 years later either, for the Milllennium Suspension Bridge [wikipedia.org], which I happened to work next to at the time and watched the construction and failure unfold. People tend to walk in sync. When things happen in sync, a resonant pattern emerges. It's true for people crossing bridges. It's true for people synchronising machines to exact clocks and nice rounded-off times as well.
Lessons, not learned. (Score:1)
This is eerily similar to the marching soldiers' bridge collapse [wikipedia.org] - Broughton Bridge in 1831, often misattributed to Napoleon but happened after his reign. Lessons weren't learned 180 years later either, for the Milllennium Suspension Bridge [wikipedia.org], which I happened to work next to at the time and watched the construction and failure unfold. People tend to walk in sync. When things happen in sync, a resonant pattern emerges.
Yes. The same thing has been happening in our cars for over a century now. Which begs the question; why does the purpose of the 100-year old harmonic dampener, sound eerily familiar to a 100-year old solution for bridge failures?
I joked about this earlier, but seriously. If we've known about the problem of harmonics in systems and why we need to dampen them, why have we not realized this need in larger structures long ago? Even for retro-fitting?
This is what Jitter is for (Score:2)
Re: Always wondered about our crontabs (Score:2)
It definitely causes issues sometimes. I've seen system crashes due to low memory, for example, get introduced by a relatively innocuous cron reconfiguration.
However, it's at least setup for consistency, so these sorts of issues get caught pretty quickly. Would be much harder to figure out if it happened through some sort of synchronizing attractor over time.
I don't get it (Score:3)
Is it the UPS or the shitty bundled software doing the test? How could every UPS be time synced that closely? None of mine switch to battery power every 60 seconds. Only the larger model does it every month or so.
Re: (Score:2)
Re: (Score:2)
Not to mention that propagation is limited to a fraction of the speed of light. The difference between distance and a delay are irrelevant.
Damn! (Score:2)
Digital devices can synchronize their internal clocks using external time signals, meaning that independently operating systems can end up performing their scheduled tasks together.
My first thought was of Huygens, and pendulum clocks syncing up. To this day physicists are investigating and trying to fully understand that. So it's a bit disappointing to read such a mundane explanation for UPS tests syncing up.
It's good to know what the mechanism is. But mystery and spookiness have their appeal too, and today I was hoping for spookiness.
Re: (Score:2)
I was more thinking of a power outage that then causes the devices to have the exact same boot time. If the devices then use the grid frequency to keep time, they will all have the same time.
Oscillation attack (Score:2)
I wonder how feasible it would be to run an oscillation attack, by syncing enough devices and machines. It could be hidden in layers of different cycles that converge into one. Probably stuff for a TV show?
59 or 61s (Score:2)
This is why I set all my timers to a prime number like 59 or 61s, 29 or 31s. You get the same relative frequency, but the timers sweep across the clock regularly.