Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
Data Storage News Technology

Database Glitch Grounds American/US Airways 274

Posted by timothy from the abort-retry-fail-land dept.
An anonymous reader writes "According to numerous news sources, all American Airlines and US Airways flights were grounded for two or three hours this morning. Both problems were caused by a computer glitch in the systems hosted by EDS. Quote: The operating system that drives the airline's flight plans went down."
This discussion has been archived. No new comments can be posted.

Database Glitch Grounds American/US Airways More

Database Glitch Grounds American/US Airways

Comments Filter:

  • Well isn't that some great news, that makes me feel 20x better about taking my gf to the airport this morning. Fortunately she wasn't flying U.S. Airways or American Airlines.

    She is absolutely frightened of flying, and somewhat of a computer nerd, I can't wait to talk to her, and tell her the scary news.

    • Re:Great News! (Score:5, Informative)

      by Vlad_the_Inhaler ( 32958 ) on Sunday August 01, 2004 @01:43PM (#9858397)
      Typical Airline applications are Reservations, Check-In, Weight-and-Balance, Flight Planning (which route to take and how much fuel to carry) and Ticketing. Once you have left the terminal and are heading for the runway, software crashes cease to be relevant.

      Once you head for the runway, you care about Air Traffic Control's software. The only exception I can think of is for flights to the US where the authorities want passenger lists.

      I work for an airline and we host for other airlines. I feel sorry for whoever carries the can for this mess. As to the OS, those who said it will be MVS are almost certainly correct. AA and US Airways are/were IBM customers.
      • As to the OS, those who said it will be MVS are almost certainly correct.

        Actually, it's probably TPF.

        • Re:Great News! (Score:3, Interesting)

          by Vlad_the_Inhaler ( 32958 )
          Ho hum.

          After I submitted the grandfather post, I saw something I'd missed first time around:

          The operating system that drives the airline's flight plans went down. It might even be a Windows problem. A 'Flight Planning' application is a low volume application where you work out the optimum route for a plane based on the weather. That bit about the weather involves serious number crunching and the PC world has more of that kind of power to spare than the mainframe world. I helped write one of these apps 2

    • Re:Great News! (Score:5, Insightful)

      by Chess_the_cat ( 653159 ) on Sunday August 01, 2004 @01:44PM (#9858412) Homepage
      What scary news? The airplanes are piloted by people, not computers. And certainly not the computers that control flight plans. Do you think that airplanes will start falling from the skies because a computer went down somewhere? I guess you packed your basement with cans of beans for Y2K too.
      • Nice sig, but shouldn't it be "support diversity", or "support weirdness", or something similar? The first ammendment just guarantees people a right to speak, it doesn't mean I have to listen.

        For my money, I see so many "Fuck Whoever", and "GNAA" posts when I read at -1 that I only bother when I'm moderating. Their first ammendment rights a) don't apply to a privately owned board, and b) don't mean I have to wade through the crap they spew to see the good stuff.

        I'm completely offtopic here, but it bugs

    • She is absolutely frightened of flying, and somewhat of a computer nerd, I can't wait to talk to her, and tell her the scary news.

      One of my worst flights ever was on a business trip to Edinburgh, Scotland. I was accompanied by a genuine RAF pilot, who flies the tanker Boeings for NATO warplanes. There was a rainstorm and strong wind over the whole UK and my friend was busy explaining me that Boeings are very vulnerable to strong winds and wind is the scariest threat for Boeing pilots and so on. It wasn't
    • Sorry, I think this is happening to a number of Airlines:
      href=http://www.ctv.ca/servlet/ArticleNews/story/C TVNews/1091237095342_4/?hub=TopStories [www.ctv.ca]

      Probably just the CIA moving them all onto some big CIA super-computer.

  • Operating System (singular) (Score:5, Insightful)

    by Hypharse ( 633766 ) on Sunday August 01, 2004 @01:29PM (#9858329)
    "The operating system that drives the airline's flight plans went down."

    How in the world can they state that as singular. Surely they have a backup of some sort. Especially with all the supposed "increased security" around air flight, you are telling me that one system crash can knock out half of the major airlines? That's ridiculous. Have they not learned about redundancy?

    • Re:Operating System (singular) (Score:5, Funny)

      by Anonymous Coward on Sunday August 01, 2004 @01:33PM (#9858347)
      Yeah, Have they not learned about redundancy?
      • Yeah, Have they not learned about redundancy?

        Well duh! Your average "aviation writer" is flat out telling the difference between an aeroplane and a hole in the ground. Asking them to write a story that involves aeroplanes *and* computers is just asking for trouble. Why, that's like asking a six year old to pat his head and rub his stomach at the same time!

        Basically the way these morons operate is that they latch on to any half-valid snippet of information - if their editor believes it, it must be true -

    • Well, one operating system can run on multiple machines. Google, f'rinstance, has thousands of machines running Linux, and that's one operating system. Probably there was some service running on all the systems that choked simultanously on some piece of bad data or they distributed a bad upgrade.

      Either way, somebody fucked up somewhere.
      • Yeah, but if Google loses a website in their results, nobody notices. If an air traffic controller loses a plane with a few hundred people on board and lets another plane fly close to or into it, you can be damn sure that people will notice.

    • Re:Operating System (singular) (Score:5, Funny)

      by njcoder ( 657816 ) on Sunday August 01, 2004 @01:41PM (#9858379)
      "Have they not learned about redundancy?"

      Yep, their so good, even the failure was replicated!

    • ... you are telling me that one system crash can knock out half of the major airlines?

      That's not what it says at all. American and U.S. Airways certainly don't count as half the major airlines in the United States. There are hundreds of airlines in the U.S. of A., and maybe a dozen qualify as "major." And by some measures, U.S. Airways doesn't count as a "major." So, no, you're completely wrong. Don't read things into the article that aren't there (assuming you RTFAs.)

    • Re:Operating System (singular) (Score:5, Informative)

      by Vlad_the_Inhaler ( 32958 ) on Sunday August 01, 2004 @01:52PM (#9858448)
      This is partially a question of cost, redundancy costs money and those airlines are rather short of the readies (although this crash will cost serious money).

      For any *normal* 'extreme situation', a reboot should help.

      Having just read that The operating system that drives the airline's flight plans went down, it might even be a Windows problem. A 'Flight Planning' application is a low volume application where you work out the optimum route for a plane based on the weather. That bit about the weather involves serious number crunching and the PC world has more of that kind of power to spare than the mainframe world. I helped write one of these apps 20-18 years ago and the central part has since been converted to run on PCs.
    • ... were any human beings killed or injured? ... were any human beings in danger of being killed or injured?

      It's hard to tell from the sketchy news stories, but it looks like AA and UA *do* have a backup plan and *are* executing it. The backup plan is a ground stop for 2-3 hours while they sort things out.

      If you want them to have a backup plan which involves providing full service with no interruptions, then you would have a ticket price to fund that.

    • Do you know the cost of redundency ? (Score:5, Interesting)

      by aepervius ( 535155 ) on Sunday August 01, 2004 @02:05PM (#9858525)
      Here around we studied it, for one major airline in EU. We wanted a "backup system" in case the main system went down. Total Cost, without maintenance, about *3 whole day* of traffic "benefits"... Yes, that much. Right now the project is still discussed but most of us thinks it is dead in the egg. Instead the "older" and "less powerfull" developpement system will be used in case of break down.

      Redundancy is OK, as long as it is not bleeding you dry.
    • You are thinking like a computer techie... I suspect this guy does not even know what a Computer Operating System is, he is more likely refering directly to the underpinning infrastructure that runs his airline.
    • Actually, my father is a retired captain from American, and I used to go into crew schedule. It used to be that flight plans were done on a mainframe. I doubt that it is still that way, but it was one computer back then.

  • Settle down, we've all seen this before... (Score:4, Funny)

    by the_seal ( 758154 ) on Sunday August 01, 2004 @01:34PM (#9858350)
    Airport BSOD [capgemini.nl]

  • Last thing you want to hear (Score:5, Funny)

    by xIcemanx ( 741672 ) on Sunday August 01, 2004 @01:34PM (#9858354)
    I'm guessing the last thing you want to hear on a plane now is the pilot saying, "What do you mean, fatal exception error?"

    >_ Why don't they swtich to Linux?

    • You probably won't hear it (Score:5, Informative)

      by Anonymous Coward on Sunday August 01, 2004 @02:08PM (#9858540)
      The systems that run the aircraft and the navigational and communication systems really are redundant. It's the law. It also means that usually there are two different ways to do something not just the same thing repeated twice.

      Example 1 - The pilot and co-pilot can't eat the same meal. That way, only one of them can get food poisoning.

      Example 2 - The hydraulic system fails and the wheels won't go down. There's a hand crank.

      Example 3 - The communication systems at every tower I have worked at have two separate backbones. There are two of absolutely everything. If that fails, there are emergency radios under the desk. If the emergency radios don't work ... We used to joke that the controllers would climb to the top of the tower and wave fire extinguishers to warn the planes away. (I think it was a joke.)

      Example 4 - You can't fly very far over open water in a single engine aircraft.

      It used to be frustrating working on systems older than I was but we never had to worry about surprises.

      Of course all of this redundancy is very expensive. You spend the money where people's lives are at stake. On the other hand, if the worst problem is that some planes will be late, perhaps you don't spend the big bucks.
      • Example 1 - The pilot and co-pilot can't eat the same meal. That way, only one of them can get food poisoning.

        Yeah just as well they have that rule, just imagine the arguments if the co-pilot was allowed to eat the captain's lunch. Do they have rules that stop them both sitting in the same seat as well?

        Example 2 - The hydraulic system fails and the wheels won't go down. There's a hand crank.

        Its the same thing with software, no shortage of cranks.

      • Example 3 - The communication systems at every tower I have worked at have two separate backbones. There are two of absolutely everything. If that fails, there are emergency radios under the desk. If the emergency radios don't work ... We used to joke that the controllers would climb to the top of the tower and wave fire extinguishers to warn the planes away. (I think it was a joke.)

        It's not a joke. They have light guns (think God's Own Spotlight here) instead of fire extinguishers, and they use them from
      • If the emergency radios don't work ... We used to joke that the controllers would climb to the top of the tower and wave fire extinguishers to warn the planes away.

        I though that was what hand lights [nasa.gov] were for.

    • Re:Last thing you want to hear (Score:5, Funny)

      by Brandybuck ( 704397 ) on Sunday August 01, 2004 @02:13PM (#9858560) Homepage Journal
      Q: How far can the plane fly after a fatal exception error?

      A: All the way to the scene of the crash. Hell, it will probably beat the paramedics there by half an hour!

  • BSOL (Score:4, Funny)

    by Anonymous Coward on Sunday August 01, 2004 @01:39PM (#9858369)
    Blue screen of life. Because US Air cancelled the flight and we were forced to fly on a competent airline.

  • EDS (Score:3, Interesting)

    by sql*kitten ( 1359 ) * on Sunday August 01, 2004 @01:40PM (#9858374)
    My experience with EDS is that problem is most likely to have actually been operator error. These people, and CSC are the absolute bottom of the barrel as far as outsourced data centres go. Yes IBM GS costs more, but there's a good reason for that! I'd sooner use Accenture than EDS, and that's saying something.

    • Re:EDS (Score:3, Insightful)

      by Detritus ( 11846 )
      Staffing quality is rather strongly correlated to what the customer is willing to pay for labor. If the customer only cares about the bottom line, they get what they deserve.

      • Re:EDS (Score:3, Insightful)

        by sql*kitten ( 1359 ) *
        If the customer only cares about the bottom line, they get what they deserve.

        Penny wise, pound foolish. Always the way, these days.

  • EDS? Quelle surprise. (Score:5, Interesting)

    by leathered ( 780018 ) on Sunday August 01, 2004 @01:40PM (#9858376)
    Sorry, have to rant where I see EDS mentioned.

    EDS, in cahoots with the UK govenment, have wasted millions of pounds of taxpayers money on failed IT projects. Notable ones include the Inland Revenue (UK IRS), Child Support Agency (£50M over budget and still not working) and an email and directory service for the NHS (withdrew at last minute allowing C&W to steal at a much inflated price).

    Though the blame cannot completely be laid at the door of EDS, the government has been guilty of sloppy auditing and the worst being the willingness to hand over extra money when EDS has come around with the begging bowl.

    • Re:EDS? Quelle surprise. (Score:4, Insightful)

      by oddRaisin ( 139439 ) on Sunday August 01, 2004 @02:44PM (#9858739)
      If we're going to lay blame, let's make sure we're spreading it evenly. A lot of contracts, and especially government ones, suffer from extreme scope creep. I have seen projects that started out with a 20 page description grow to over 150 pages by the end of the project.

      EDS and other large IT vendors try their best to discourage scope creep by making changes-after-the-fact billable for time and materials, instead of a negotiated cost. This makes the project go over budget. If the clients knew what they wanted at the begining, instead of wasting time and money doing engineering on the fly during the project, then the costs wouldn't be so high.

      Don't be so quick to slag EDS about the outage either. There are lots of factors out there that could have contributed. I have worked on projects where the clients say the servers are mission critical, yet can't be bothered to shell out money to upgrade from ultra-1 and ultra-5s, let alone pay for an HA solution. The technical people keep providing the justification and making the requests, but it's the project managers and accountants that really determine what kind of solution is feasible.
      • If the clients knew what they wanted at the begining...

        Isn't this what Systems Analysts are supposed to do?

      • I have seen projects that started out with a 20 page description grow to over 150 pages by the end of the project.

        What's the record for government contracts/descriptions?

        German Toll Collect (a country-wide, per-kilometer road use toll from trucks) had an 17000 page contract with the goverment.

        The system should have been running since mid 2003, it still doesn't work.

    • Re:EDS? Quelle surprise. (Score:4, Insightful)

      by demachina ( 71715 ) on Sunday August 01, 2004 @03:54PM (#9859073)
      Don't think you can say much better or anything difference about CSC or the rest of the small cadre of IT companies who specialize in winning government contracts around the globe. They've had their share of multibillion dollar fiascos too.

      The problem with these companies is they specialize in winning big contracts. They put their best people on the proposals. They don't specialize in delivering great systems. Their best people probably move to the next RFP and they mostly fill the contracts with warm bodies.

      They can get away with it because its pretty rare for them to actually be punished for poor performance. If they get blacklisted by the agency that awarded the contract the agency ends up just replacing EDS with CSC or vice versa and the results don't get any better. I'd be interested if someone could cite a huge government IT contract that actually worked well. At some point governments need to figure out this methodology doesn't work and try something new.

  • Steve Balmer (Score:3, Funny)

    by Stevyn ( 691306 ) on Sunday August 01, 2004 @01:41PM (#9858378)
    You know he's going to convince them not to switch to linux. First he's going to get on a plane...oh wait.

  • Operating system? (Score:3, Informative)

    by SlamMan ( 221834 ) on Sunday August 01, 2004 @01:41PM (#9858381)
    Only one of the articles mention said anything about an "operating system." The rest called it a system problem. That does not necessarily mean an OS, or anything related to it. I think katu's reporter jumped to a conclusion.
  • Isn't it stated somewhere that a cetain OS, which is forever fair game in this community, should not be used for 'Mission Critical' situations?

  • I thought everyone knew (Score:5, Funny)

    by Xerp ( 768138 ) on Sunday August 01, 2004 @01:42PM (#9858389) Journal
    NEVER open Windows in an airplane!
  • Again another example of EDS shoddiness, why anyone would give money to EDS for anything is beyond me, they deliver inferior service at outrageous prices. M

  • Wild speculation (Score:4, Insightful)

    by MisterP ( 156738 ) * on Sunday August 01, 2004 @01:46PM (#9858419)
    They aren't telling the whole story.

    I come from Solaris/Veritas/Oracle and Redhat/Oracle RAC environments. One single system going down cannot take out the service. Database HA is somewhat complicated and expensive, but it's not rocket science, regardless of platform.

    I find it very difficult to believe that they would have any single points of failure in a system of that importance. Blaming MS is the easy way out.

    • Re:Wild speculation (Score:3, Insightful)

      by treat ( 84622 )
      Standard database clusters use shared or replicated database storage. If you trash the database - a human error - the cluster is useless.

    • Re:Wild speculation (Score:5, Interesting)

      by Archibald Buttle ( 536586 ) <`steve_sims7' `at' `yahoo.co.uk'> on Sunday August 01, 2004 @03:30PM (#9858961)
      Blaming MS is the easy way out.

      I just read all the stories that were linked to this article.

      None of them blamed Microsoft. In fact the only blame pushed in their direction was your comment...

      The articles did say that there was a problem with the operating system. Now we don't know who exactly said this, or what they said precisely, so it is quite possible that this isn't entirely accurate reporting.

      I find it very difficult to believe that they would have any single points of failure in a system of that importance.

      I agree it's unlikely, but it is possible that there is a single point of failure in their system. There are a great deal of shoddily engineered systems in use today.

  • Not Windows, Unix (Score:5, Informative)

    by JohnQPublic ( 158027 ) on Sunday August 01, 2004 @01:48PM (#9858425)
    This is undoubtedly a problem with Sabre, which EDS runs on behalf of Sabre Holdings. Both American Airlines and US Airways use Sabre for much of their operations.

    Sabre started it's life as an American Airlines internal system (SABER, slight spelling difference), running on a rare operating system (PARS, later called ACP and currently TPF) on IBM mainframes. In the last few years Sabre completed a lengthy migration to HP Unix on Non-Stop (i.e. ex-Tandem) hardware. The mainframe systems were rock solid, but software talent was hard to come by, so they decided the time had come to switch.

    Sorry, no Microsoft to blame here!

    • Re:Not Windows, Unix (Score:3, Interesting)

      by justins ( 80659 )
      This is undoubtedly a problem with Sabre, which EDS runs on behalf of Sabre Holdings. Both American Airlines and US Airways use Sabre for much of their operations.

      They use the same system for flight operations and for reservations? I've seen Sabre in use at the travel agent's office, somehow I would have thought this problem involved a different system...

      • More info about Sabre than you ever wanted... (Score:4, Informative)

        by airbatica ( 743048 ) on Sunday August 01, 2004 @06:04PM (#9859685)
        Sabre is a multitude of software products, for lack of a better definition. They include RES, DECS, TIM, BMAS and a couple of others that I can't remember.

        All Sabre applications are text mode, no GUI whatsoever... think CLI from hell, with no command history if you fat finger an entry.

        The system that went down was probably DECS (Dispatch Environment Control System), which is the system used by both American and USAir for generating flight plans, load planning, weight and balance, and various other flight operations functions.

        RES is the Reservations system, which covers the spectrum from building reservations and selling tickets, to customer checkin, boarding and god knows what else. IIRC, it will even do car rentals and hotels.

        TIM is also called Timatic. Its used for accessing information from the US State Department regarding internation travel to any country, from any country in the world. It covers entry and exit requirements, documentation, and pretty much anything you could want to know.

        I don't remember what BMAS stands for, but it is a lost bag tracking and reporting system. When AA or US looses your luggage, this is what they use to find it.

        Sabre is used by a whole variety of airlines and travel agencies, and is customised in modules to each particular user's needs.

        Now you are probably wondering how I know all this... I work for a major airline that uses a majority of the systems listed above, with the exception of the Dispatch system. We were not affected by whatever snafu took down that portion of Sabre :)

    • Probably Sabre Holdings, rest probably wrong (Score:5, Interesting)

      by Markus Registrada ( 642224 ) on Sunday August 01, 2004 @03:12PM (#9858871)
      First, they didn't "complete a migration". They're still deep in the middle of it, and will be for years to come.

      Second, this failure isn't in the Sabre reservations system, it's in some ancillary product, so who knows? Maybe they have no intention of switching it to Unix.

      Third, he didn't say so, but the migration isn't just to Unix. It's also migration to MySQL! (Hahahahahahahaha. Then again, coming from TPF, coded in assembly language for 4Kword pages, and a hierarchical database, that might seem pretty advanced.) Sabre had to fund a MySQL port to 64 bits, and a new "stored procedures" feature.

    • I concur, though there are a number of middleware apps around Sabre that could have conceivably had an issue. However, those apps that I've been made aware of are almost universally Java on Solaris.
    • Mainframes will be the death of us (Due to a lack of talent)! The last major delay in the UK air traffic system was their NAS mainframe going down after a failed patch, and taking three hours (on a Friday morning!) to get back in service. (you can google for NATS being grilled by parliament)

  • Guess IT Does Matter (Score:2, Funny)

    by Anonymous Coward
    well, i guess IT Does Matter after all....
  • Any attempt to retrieve information from them (flight data, schedules, FOIA requests) will result in total, immediate, and irreversable loss of data! [publicintegrity.org]

  • here's what happened (Score:3, Funny)

    by Scythr0x0rs ( 801943 ) * on Sunday August 01, 2004 @01:59PM (#9858488)
    "It looks like you are flying an aeroplane, Would you like help?" YES!

  • What *REALLY* happened... (Score:5, Funny)

    by catdevnull ( 531283 ) on Sunday August 01, 2004 @01:59PM (#9858490)
    At about 4:30 a.m., the outsourced SysAdmin was setting up to do routine patches to Windows 2003 server nodes. But just before, he decided to check his e-mail with Outlook and he opened an important message from his system administrator advising him that his e-mail would be de-activated if he didn't open the important attachment. I think we all know what happened after that...

  • Not Smart Enough (Score:3, Interesting)

    by PingPongBoy ( 303994 ) on Sunday August 01, 2004 @02:00PM (#9858495)
    It seems that computer failures are not very graceful. In a large business if an employee or even the chairman of the board is sick, the business still runs. However, failure of the central computer means no one knows how to make anything run.

    Perhaps the efficiencies of a computerized business offset the cost of short downtimes, and the business is able to grow to the complexity that it isn't worth running without the computer. A 2 or 3 hour stoppage once in a blue moon (that was last month, and it looked big) might not be worth working around.

    All the same I'm hesitant to let computer failures stand in the way of normality. Major infrastructure may be interrupted by nature but it can be scary for it to be stopped by computer problems. Who knows how long the system will be down? Who knows how much damage to information went unnoticed? Who knows what errors still exist?

    Increasing computerization causes increasing paranoia. Guard yourself prophylactically? Ask hard questions before entering relationships with big business? Insist on financial compensation against computer delays?

    Computer systems need to be built with more safeguards (redundancy, logging, checkpoints, backups), isolation of failure, data accessibility during failure (example: Windows safe mode) even for end users, etc.

  • This isn't what you think (Score:3, Interesting)

    by xenophrak ( 457095 ) on Sunday August 01, 2004 @02:19PM (#9858601)

    Even though this sounds dire, I have a feeling that this does nothing to compromise airline safety.

    From the sounds of it, the flight planning system went down. This is a ground-system only, often a terminal next to the ticket checking counter. The purpose is to file flight plans, check weather airport conditions, etc. It is not an onboard system. This would not have likely decreased passenger safety.

    The reason that the FAA got involved was because AA decided to ground the planes because the pilots most likely couldn't file flight plans electronically. If left to the filing flight plans the old way, it would have delayed things more and caused more headaches to just wait out the system outage.

    However, when any business runs and depends on a particular piece of software to generate revenue and to provide a service, I would be more inclined to host such a system on something like a mainframe or at least a big Unix server.

  • I found the root of the problem (Score:5, Funny)

    by Anonymous Coward on Sunday August 01, 2004 @02:27PM (#9858632)
    There is a line of code that raised the problem but is commented in Punjabi, I think it says "fuck this $3/hour job".
  • Every time I see one of these articles, I read the fine print just to be sure it wasn't one of my bugs that brought the system down. Looks like I'm probably safe this time.

  • What's going on here? (Score:3, Interesting)

    by iantri ( 687643 ) <iantri&gmx,net> on Sunday August 01, 2004 @03:29PM (#9858952) Homepage
    Air Canada experienced numerous delays [www.ctv.ca] yesterday, too...

    Hmm... what's going on here?

  • ..turn anything in a positive way.

    "Customers won't necessarily miss their connections," he said, "because everything was stopped."

  • This [nasa.gov], ladies and gentlemen, is a flight plan. Now how the hell you gonna die because some FAA form can't get filled out right? All it was was a paperwork requirement. Planes still fly, pilots still know how to land them rubber side down.

  • Not "OS" (Score:5, Informative)

    by Master of Transhuman ( 597628 ) on Sunday August 01, 2004 @05:22PM (#9859497) Homepage
    When they said "operating system", they meant "operations system" - not the OS.

    See this quote from one of the articles:

    Wagner said a database malfunctioned that "basically runs every aspect of our client operations -- aircraft dispatch, crew scheduling (and) reporting weight, passenger load, balance."

    This system is hosted by EDS, who only said it was a "systems issue".

    So there's no evidence it was an OS problem. It could have been anything - OS, Oracle/DB2/SQL Server database, application code, upgrade, whatever.

    Nothing to conclude here except that somebody screwed up - and even that isn't certain - could have been a bad memory board someplace, who knows.

    Not having a backup is even irrelevant, since the "backup" might have taken three hours to bring up, when you're dealing with a production system like this. "Failover" is what you want, and they should have had, but if something got screwed there, it could still have been three hours.

    Shouldn't have happened, but crap like this happens all the time because nobody can do their damn jobs.

  • Same Thing Happened to NorthWest (Score:3, Interesting)

    by Salis ( 52373 ) on Sunday August 01, 2004 @05:46PM (#9859599) Journal
    Their computer system went down for at least *3* hours in Minneapolis, shutting down the entire terminal. They couldn't check flight plans, ticket information, scheduling, logistics, etc. No planes in, no planes out.

    You'd think they'd have redundancy and backups, but they probably don't. That requires some planning beyond the immediate need of the company and, even if it's more profitable to invest in backups, long term planning simply isn't considered as much.

    This happens to my University all the time. The power goes out in one building for a few hours and services across the entire University are disrupted completely. This building happens to house most of the license servers for important software, but no one would _ever_ think of putting a backup license server in another building _just in case_. No, that'd be thinking ahead.

Slashdot Top Deals

I've noticed several design suggestions in your code.

Close