Forgot your password?
typodupeerror
Data Storage Power Privacy

NSA's New Utah Data Center Suffering Meltdowns 241

Posted by timothy
from the high-grade-shadenfreude dept.
linuxwrangler writes "NSA's new Utah data-center has been suffering numerous power-surges that have caused as much as $100,000 damage per event. The root cause is 'not yet sufficiently understood' but is suspected to relate to the site's 'inability to simultaneously run computers and keep them cool.' Frustrating the analysis and repair are 'incomplete information about the design of the electrical system' and the fact that "regular quality controls in design and construction were bypassed in an effort to fast track the Utah project."" Ars Technica has a short article, too, as does ITworld.
This discussion has been archived. No new comments can be posted.

NSA's New Utah Data Center Suffering Meltdowns

Comments Filter:
  • by tomkost (944194)
    I hope the whole place melts down or any type of natural disaster. These tools will be used against the evil and the good alike.
  • Iranian Stuxnet? (Score:5, Interesting)

    by Anonymous Coward on Tuesday October 08, 2013 @11:35AM (#45070755)

    Hmm...so your equipment is randomly failing...you don't say?

    • by cold fjord (826450) on Tuesday October 08, 2013 @12:23PM (#45071409)

      Iranian Stuxnet? No, just government contractors on a rush job that badly documented and cut corners on a sensitive aspect of the design that controls massive resources (power (65 megawatts - enough to power a small city), cooling, etc.) critical to the function of the datacenter. This is generally referred to as, “your tax dollars at work.”

    • Given they apparently haven't even switched on any computers there yet, presumably the cyberattack fun still hasn't begun.

      This raises the question of where they're processing all their existing data. Fort Meade ran out of electricity some time ago, from what I understand, so presumably they have some other big datacenters in other places.

    • by 93 Escort Wagon (326346) on Tuesday October 08, 2013 @12:47PM (#45071745)

      Never attribute to malice what can adequately be explained by incompetence.

      • Never attribute to malice what can adequately be explained by incompetence.

        All right, let's explain it that way. Let's also make sure that incompetence is punished, while we're at it.

      • Never attribute to malice what can adequately be explained by incompetence.

        Try not to use absolutes like "Never" in general statements, you will most likely be wrong, and also a fool.

        • Never attribute to malice what can adequately be explained by incompetence.

          Try not to use absolutes like "Never" in general statements, you will most likely be wrong, and also a fool.

          Or at least a Sith.

      • by HiThere (15173)

        Sometimes you get both.

  • by stewsters (1406737) on Tuesday October 08, 2013 @11:36AM (#45070769)
    Save yourself some trouble and stop spying on your own people then.
  • by rcb1974 (654474)
    Respect our right to privacy. Stop the snooping. I am upset and ashamed that the people of Utah have allowed this spy center to be built in their state. I had higher expectations of them. They should have cherished our individual liberties, upheld the Constitution, and told the Federal government take their spy center elsewhere.
  • It probably lacks certification from the Department of Redundancy Department anyway...
  • You set out to snoop on the minutiae of the lives of tens of millions of innocent people. Then your data center melts down... Perhaps God/Angels had something to do with this? =)
  • by generic_screenname (2927777) on Tuesday October 08, 2013 @11:42AM (#45070863)
    The only thing that will save us from the massive dragnet of the NSA is apparently the incompetence of the NSA.
    • by Professr3 (670356) on Tuesday October 08, 2013 @11:44AM (#45070901)
      Bureaucratic incompetence has been the strongest protector of civil liberties to date :\
    • by atgaaa (1869296)

      We will be told more money will solve the problem. Closing it down will solve the problem also.

  • by TubeSteak (669689) on Tuesday October 08, 2013 @11:43AM (#45070871) Journal

    The submission had one article, the editors linked to two more.
    ALL THREE ARTICLES REFERENCE & LINK TO THE WALL STREET JOURNAL

    Is it so hard to include a link to the source of this story?
    http://online.wsj.com/article/SB10001424052702304441404579119490744478398.html [wsj.com]
    (Google Cache just in case [googleusercontent.com] /. does this far too often and I hope to see better in the future

    • Re: (Score:2, Funny)

      by sootman (158191)

      > /. does this far too often and I hope to see better in the future

      Don't hold your breath. :-)

  • Power management (Score:5, Interesting)

    by Antony T Curtis (89990) on Tuesday October 08, 2013 @11:43AM (#45070873) Homepage Journal

    They probably used a power budget similar to the public Facebook datacenter data but then decided to run their machines on Windows Azure.
    I have noticed that power consumption of my computers is significantly higher when running Windows - and the laptops have seriously reduced battery life, even while doing nothing.

  • by pla (258480) on Tuesday October 08, 2013 @11:45AM (#45070917) Journal
    Someone explain to me why the holy bloody fuck these enemies of the American people haven't taken an involuntary 8-day vacation along with the rest of the noncritical federal government?
    • by pellik (193063)
      What enemies of the American people? You mean the poor backwords farmers on the other side of the world? What are they gonna do, grow drugs?
  • afterall google revealed a good amount on how they go about building their data centers and keeping it cool. But then again, contractors...

  • by Salgak1 (20136) <salgak@speake a s y . n et> on Tuesday October 08, 2013 @11:51AM (#45070995) Homepage

    . . .eats huge amounts of power, not large amounts of water for cooling.

    And thus, power requirements go up, pushing the limits of your provisioned electrical infrastructure.

    And extremely-high-capacity circuit breakers tend to be explody when they fail. My guess: someone used some REALLY bad assumptions for electrical infrastructure planning. . .

    • by bill_mcgonigle (4333) * on Tuesday October 08, 2013 @12:08PM (#45071219) Homepage Journal

      My guess: someone used some REALLY bad assumptions for electrical infrastructure planning. . .

      Hey, don't be too hard on the electrical engineers - James Clapper told them that the power requirements would be really low.

    • by linuxwrangler (582055) on Tuesday October 08, 2013 @12:13PM (#45071287)

      Works for Switch in Las Vegas. Cold in winter and cools off at night so 70% of annual hours they can pull in ambient air through filters. Evaporative cooling, whether direct or to cool the hot-side of a refrigerated system, works best in dry climates but it's only used to improve efficiency as they can run fine with air-cooling albeit at much higher power costs.

      I'm still surprised at the number of places that think cooling is optional. We had equipment in a Sacramento data-center that had plenty of backup electricity for servers but couldn't run the AC in a power outage. The SLA only had provisions for exceeding 80-degrees for more than something like 90 or 120 minutes. *Ahem*, cold-comfort when a dense data-center can blow through 100 in minutes without AC.

      UC Berkeley had a widespread power outage about a week ago. The main campus data center had power but, you guessed it, couldn't run cooling and had to "gracefully" shut down most of the core systems while watching the center breach 100F.

      But I agree with your base assumption - really bad planning and/or execution on the power systems.

  • Well, well. (Score:4, Funny)

    by fuzzyfuzzyfungus (1223518) on Tuesday October 08, 2013 @11:51AM (#45071007) Journal
    I, for one, definitely trust an outfit that can't size a bloody datacenter power distribution system to build those magic technical safeguards that are allegedly allowing a spying operation of unprecedented size to occur with no abuses (And that's no bullshit!)

    I wonder if we could convince them to switch to a utility that conducts background checks on electrons before sending them to the customer? That would clearly help...
  • 'incomplete information about the design of the electrical system'

    Well, duh, it's secret.

    • by Lumpy (12016)

      No, knowing govt projects. Nothing labelled and changes made on the fly that are not documented. One building they had 48 breakers and NONE labeled.

  • Cry me a river, and all that.
  • by Catbeller (118204) on Tuesday October 08, 2013 @12:04PM (#45071185) Homepage

    This isn't the result of incompetence - rather the result of trying to racing to finish the thing before any more opposition builds up that may stop the project. Wile E. Coyote trying to run on air, knowing it's impossible, but trying to get to the cliff before gravity notices the flagrant violation.

    When that monster is done - and it seems that they are turning it on *right now*, this week - human history is done as we understand it. We will all behave as though someone is watching and recording us, because they will be.

    Scientology is going to *love* this - one stop shopping for all its spying needs. The NSA just last week asked permission for private corporations to access their new trove of data, Because Terrorism. The Unification Church and Scientology will be first in line with front corporations to drink deep of this wonderful new integrated terrorism enabling center - terrorism because bad guys like Scientology will be able to terrorize people with fresh, holistic super-knowledge not only of who they are, what they say, what they read and where they've been, but also of everyone their enemies ever talk to, email, walk next to, text or write to. That center isn't about just metadata, it's the *actual phone conversations* that will be recorded. Don't ever piss off the powerful, 'cause they can nail you and anyone who ever contacts you until you give up. Blackmail, extortion, we-know-where-you-kids-are... anything. And the coolest part is that it will all be secret! Persecutors with behind stage access to the NSA superboxes and analytic tools won't even be logged in any real sense. Political opposition, nullified, instantly. The possibilities for our brave new world owners are limitless.

  • ...the NSA tries to tail -f the [censored] planet!

    ; )

  • Frustrating the analysis and repair are 'incomplete information about the design of the electrical system' and the fact that "regular quality controls in design and construction were bypassed in an effort to fast track the Utah project."

    This sounds like someone was in such a great hurry to get their shiny new toy that they bypassed a lot of the steps they should have followed.

    And, somehow I doubt there's a lot of sympathy for the NSA here on Slashdot.

    • Gee, if they rushed that job, I bet they rushed QA too, esp. pen testing... I mean, Snowden showed how dumb it is to legislate computer security.

      Protip: Bad guys don't care about laws.

      The Chinese, Russians, etc. will probably be using it as soon as it comes online, certainly before the next Olympics.
      Hey, might be a good idea to secure those SCADA systems eh? You know, like when all our grid are belong to China?

  • by stenvar (2789879) on Tuesday October 08, 2013 @12:17PM (#45071331)

    to a nicer data center...

  • by Daetrin (576516) on Tuesday October 08, 2013 @12:20PM (#45071371)
    I'm sure it won't last, nothing this good ever does, but let's enjoy it while we can.
  • Couldn't happen to nicer people.

  • so we now have an official price tag on your stolen personal information.

  • by Okian Warrior (537106) on Tuesday October 08, 2013 @12:52PM (#45071821) Homepage Journal

    A friend of mine is researching power surges in the local town.

    Most building codes under-specify the gauge of the neutral/return wire. For illustration, if you have three phases each rated for N amps, there is one shared neutral/return wire rated at N amps going out. At the end of the runs all phases are connected to the shared neutral line.

    This is due to the nature of 3-phase electricity: the phases will tend to cancel out, so in a perfect setup you would need no neutral/return at all. Of course, the load on each phase won't exactly balance, and the load can vary as people connect/disconnect appliances, so you still need the neutral line in practice.

    (Not true for house wiring, which has one or two phases coming in. Each phase has a return with the same gauge as the supply.)

    This was fine when appliances were (generally) resistive loads, but nowadays switch-mode power supplies are common. When you do some math, it turns out that this type of load appears equivalent to 120 Hz power coming together at the neutral/return junction. Since 120 Hz [equivalent] power does not cancel out, the power in the return wire can be 3x as large as the building codes allow.

    I've got a book explaining all this. Typically the neutral line will heat up and catch fire, breaking the circuit. Once that happens the various phases are connected without a neutral, playing hob with whatever is on those circuits and making occasional high-power ground loops and other unexpected behaviours.

    • At the end of the runs all phases are connected to the shared neutral line.

      Poor choice of words: I don't mean to say that power and return are wired together. I meant to say that all three phases share one neutral line.

      Post before coffee, regret at leisure.

    • by iggymanz (596061)

      I'm suspecting a power factor problem, condensers not dealing with enough inductance and so surges occur when large amounts of equipment turn on simultaneously.

    • by jbengt (874751)
      I know from personal experience that this area has (at least had in the 90s) poor reliability from the electrical utility companies. When they found out the building we were designing was going to have 3.2 megawatts of back-up power, they offered to give the Owner a reduced electrical rate if they agreed to an interruptible feed from the utility. The Owner thought about it, but found that it the deal would not only require increasing the capacity of the back-up generators, but also require red tape to dea
  • Did they check for jellyfish in the intakes?

  • After being stuffed with all the information in the world, the network is starting to gain self-consciousness.

  • Isn't it terrible when bad things happen to absolute fascists?
  • There must be some clear objective to fast-track such a massive installation to skip certain of the build quality checks. This must exist under the guise of some approach not previously discussed since this is a new datacentre.

  • by PPH (736903) on Tuesday October 08, 2013 @03:12PM (#45073815)

    ... you folks wanted cheap power [slashdot.org]. So you get the cut-rate stuff. The good power we save for our paying customers.

The world is not octal despite DEC.

Working...