Forgot your password?
typodupeerror
Cloud Microsoft

Microsoft's Azure Cloud Suffers Major Downtime 210

Posted by Unknown Lamer
from the higher-availability-through-cloud-computing dept.
New submitter dcraid writes with a quote from El Reg: "Microsoft's cloudy platform, Windows Azure, is experiencing a major outage: at the time of writing, its service management system had been down for about seven hours worldwide. A customer described the problem to The Register as an 'admin nightmare' and said they couldn't understand how such an important system could go down. 'This should never happen,' said our source. 'The system should be redundant and outages should be confined to some data centres only.'" The Azure service dashboard has regular updates on the situation. According to their update feed the situation should have been resolved a few hours ago but has instead gotten worse: "We continue to work through the issues that are blocking the restoration of service management for some customers in North Central US, South Central US and North Europe sub-regions. Further updates will be published to keep you apprised of the situation. We apologize for any inconvenience this causes our customers." To be fair, other cloud providers have had similar issues before.
This discussion has been archived. No new comments can be posted.

Microsoft's Azure Cloud Suffers Major Downtime

Comments Filter:
  • But Remember - (Score:5, Insightful)

    by Ralph Spoilsport (673134) on Wednesday February 29, 2012 @11:52AM (#39197901) Journal
    Your data's safe in the Cloud.

    Until it isn't.

  • Eggs? (Score:5, Insightful)

    by OzPeter (195038) on Wednesday February 29, 2012 @11:53AM (#39197917)

    Basket?

    Or how about "Never outsource your core functionality?

  • by gral (697468) <kscarr73@NOspaM.gmail.com> on Wednesday February 29, 2012 @11:56AM (#39197965) Homepage
    The companies I deal with tend to say things like, we want to go with a company like this so we can can get "Support". Which usually means, so we can blame them if something goes wrong.
  • by masternerdguy (2468142) on Wednesday February 29, 2012 @11:58AM (#39197985)
    Also remember the cloud is just the 21st century spin of the dummy terminal-mainframe model.
  • by characterZer0 (138196) on Wednesday February 29, 2012 @12:01PM (#39198025)

    Cluster at the application level and have nodes at different providers. If your volume is too high for that, you are big enough to host your own stuff.

  • by Barsteward (969998) on Wednesday February 29, 2012 @12:15PM (#39198183)
    Stop talking sense, its no use here on /.
  • by vlm (69642) on Wednesday February 29, 2012 @12:23PM (#39198263)

    After an outage this long, it takes a LOOONG time to earn your way back to five nines (which works out to 5.5 minutes of downtime per year).

    Only 84 years per the article, and growing at a rate of a year every 5 minutes.

    Thats probably about how long it would take me to trust MS in an enterprise environment.

  • Ah, the cloud... (Score:5, Insightful)

    by ErichTheRed (39327) on Wednesday February 29, 2012 @12:25PM (#39198289)

    It's funny how those of us who bring up issues of data security and service resiliency are dismissed as just trying to protect our jobs.

    Like so many other things, the actual technical underpinnings of "the cloud" are great, and have been standard fare for years. Virtual machines + flexible networking are a godsend for systems guys tasked with getting capacity for a new project up and going yesterday. I love being able to build and rip down entire test environments just to try something out...that used to mean a rack of physical servers, switchgear, etc. tied up while it was being used. That's why everyone's slowly coming around to the "private/hybrid cloud" model, which is really just code for "VMs + network capacity + something to tie it all together + maybe some external hosting".

    The problem is that "the cloud" is very badly misunderstood. As sson as a CIO sees "virtual, on-demand capacity without those pesky physical on-site machines and IT staff, for a fixed cost per compute-hour" everything else takes a back seat. Then, it's "why do we need IT staff on-site, everything's being taken care of in the cloud." Public clouds like Amazon or Azure are great for startups who can't really afford their own data centers, or even bigger businesses to offload some of the nonessential stuff. When you start looking at hosting everything though, the marketing hype of the cloud sometimes distracts people from realities that they have to contend with.

    Also, I'm not saying that businesses who go the private cloud or traditional hosting/outsourcing route won't have downtime -- they will. However, having onsite staff and infrastructure means you can work those staff until they fix the problem, and you have control over them. Most sane outsourcing contracts have SLAs in them stating that the vendor will expend X amount of effort to fix your problems. Cloud provider agreements, unless specifically mentioned otherwise, are "as is, where is, best effort restoration with no warranty." OK, maybe some providers will give you an SLA, but all that does is buy you free service at a later date if they violate it...it doesn't bring your application back online. You still have no choice but to sit and wait around for the provider to fix whatever's wrong...just ask Amazon EC2 customers about what happened during their last outage...

    Companies need to draw sane boundaries around hosted systems, and decide what is critical and what can be offloaded. Do I care about a set of development/test machines that get used once a month? Probably a lot less than the critical database/application servers that run my core business. Comfort level, cost per minute of downtime vs. cost of dedicated resources and other factors need to be carefully considered before jumping into the cloud with both feet.

  • by hawguy (1600213) on Wednesday February 29, 2012 @12:26PM (#39198297)

    One of the selling points of using cloud services was that it would be more reliable than managing your own hardware/software. But to date, every single big player has suffered major downtime. If I would be hesitant to believe the sales pitch.

    But still, for most companies that are good candidates for cloud offerings, even 8 hours of downtime once a year is probably better than they can guarantee using their own infrastructure. Companies in this range tend to not have redundant servers, offsite backups, disaster recovery sites, etc. Larger companies that can build redundant infrastructure (and staff it properly) are probably better off staying away from the cloud since they can guarantee any level of uptime and redunancy they want to pay for.

    Of course, when a small company Admin spills a cup of coffee in the Exchange server and they are down for 5 days while building a replacement server, it doesn't make the news so you never hear about it...while when a large cloud provider has a 2 hour outage, it's all over the news.

  • Re:Eggs? (Score:5, Insightful)

    by Sir_Sri (199544) on Wednesday February 29, 2012 @12:35PM (#39198423)

    Ah, so there's the question. How much would it cost for you to run a system with 'no' downtime? I'm at a university, some of our labs (not so much in comp sci but generally) have fairly specific requirements about say not losing power, because it would damage/destroy equipment or running experiments.

    But IT is more than just power. In almost 4 years here every year we've had several days of downtime for our main undergraduate server (the one undergrads are supposed to use for various things, and that handles their logins and file storage), and several on the separate but arguably more important staff server, which is supposed does the same thing, but that includes all of our grant applications.

    Causes of our server outages (I'm not an IT guy, this is just what they've told us that I can remember): Power failures. Yes we have battery backups, but they're only good for so long, and since none of our equipment suffers permanent damage without power this isn't high priority. Networking. We only have two redundant pipes. That, for home use for example, or most businesses is pretty good. For our pipes one goes to a host to the west, one to the east. I'm not specifically familiar with what failed that took our networking offline for 7 or 8 hours but it affected both pipes. Storage: bad raid controller on the main fileserver. This has a few cascading effects. If you don't realizing it's garbling data it ends up distributing that garble off to the backups or clones. When it crashes (which doesn't take that long after the controller starts getting messy) you may have several backups that need to be repaired. We can't do much to the file system while it's being repaired or rebuilt (which, afaik you should be able to do on most professional grade setups, but for whatever reason our linux guys can't get it to behave). Added fun: When the system comes back up, if you tried to access your e-mail while the file system was garbled you probably still can't. And you get no error message about it. It just spits back nothing, as though you have no new mail. The system is 'up' but doesn't work and you have to go into your directory and delete some files that most people have never heard of. It's not hard to do, but because you have no idea that there's a problem the less technically inclined (or just ESL) people in building full of computer scientists don't always fix it immediately. The net effect is that if the storage controller gets messed up, we're down for 3 or 4 days if not longer.

    And that's just one university department. We have a relatively decent amount of money, and several full time staff for these things. But we probably can't match any cloud services uptime, even with 7 or 8 hours of downtime regularly, not even close. It's not a trivial calculation, even a 50 or 60 employee outfit will probably have trouble matching Amazon or Azure uptime with a full time IT guy. There's probably a cross over point where you have enough employees to support big enterprise IT infrastructure and manpower, but only support it badly (there's not enough money for proper replication or whatever), and then eventually you get big enough that you just run everything in house anyway because there's definitely no cost advantage to hiring someone. For us, I think we have 5 or 6 IT staff, if we could toss 3 of them, + all of their equipment, you're looking at somewhere around 350, 400k/year to spend on a support contract. I'm guessing, but don't know, if you can get a cloud service for ~20 TB of reasonably reliable file and e-mail storage for less than 350k/year from these guys.

    The big place I see people right now (as a sort of flavour of the month) using cloud service as an augment to burst capacity needs. That's a whole other analysis.

  • Cloud ain't so bad (Score:5, Insightful)

    by Martz (861209) on Wednesday February 29, 2012 @12:36PM (#39198431)

    I wrote a comment on slashdot a while back which questioned the sensibleness of running services in the cloud. I used to be a sceptic.

    Since then I've used Rackspace Cloud and found that it's actually a very good idea, for certain things.

    The benefits of using a cloud system are scalability and no commitment- it's not about reliability or higher availability - but you do get a little win in those areas.

    To give some examples, I was recently able to play around with mysql clustering. I followed a mysql clustering howto [reliablepenguin.com] and played around with it, setup a mysql cluster with load balancers. Once I was finished geeking about, I saved the VMs to the file storage and deleted the cloud instances. Total cost a £/$2-3 maximum. I hadn't previously been able to do this, I would have had to rent a dedicated server which would serve websites, email etc. I couldn't really use the dedicated server to play with new technology in case it had a negative impact on the live systems. I did have development box for a while, but it essentially doubled my costs without making any more money, just offering some protecting.

    Now I have staging/development instances in the cloud - and no commitments to them - I don't have to worry about a £250 monthly bill or sign a 12 month contract to get my own box. I can fire up some resources, use them, and throw it away when I'm done.

    The upshot is that I can play around with other peoples cool open source software without risk or buggering something up on my live box, and the costs are insignificant since I'm only renting it per hour. I can try something new, if it works great - it might go/stay in production. If not, delete it and move onto the next cool thing.

    If I need high availability, I would use Rackspace, Amazon, Azure, and I'd ensure that I have a plan to deal with a major outage with any of the providers. Each have APIs, so in theory I could create new instances automagically and failover between different cloud providers with a quick DNS change, while keep costs low.

    To recap, the cloud isn't all about high availability - no matter what the marketing says. It's about scaling systems and running resources for small amounts of time, and is perfectly suited to services which have peak demand (ticket sales for example).

  • by icebraining (1313345) on Wednesday February 29, 2012 @12:45PM (#39198573) Homepage

    Except those dumb terminals were, well, dumb, while nowadays the "terminals" are essentially the same as the "mainframe" but slower. So you can have hybrid configurations were a dedicated machines handles the base load and spins up remote resources on demand to handle peaks. If those resources are unavailable, the dedicated machine can still do the job, just with some performance degradation.

    A good example would be a script on your laptop that started an EC2 instance running distcc to reduce your compilation time from hours to minutes. If the instance can't be loaded, you could still compile, it just takes more time.

  • by dave420 (699308) on Wednesday February 29, 2012 @12:57PM (#39198769)
    Except this time you can add as many mainframes you wanted, dynamically. And access them over the internet. And serve content to millions of people over said internet. That wasn't possible with this clichéd "mainframes!!!!!1" nonsense. Yes, you are using a remote computer. That's the only similarity. The current terminals are far from dumb, and the server being connected to is vastly different to the mainframes of old.
  • by Surt (22457) on Wednesday February 29, 2012 @01:01PM (#39198815) Homepage Journal

    If only even a single cloud service were actually built this way, it'd be great!

  • by zooblethorpe (686757) on Wednesday February 29, 2012 @04:23PM (#39201633)

    I've always had to laugh at the name "Office 365" -- the fact this happened on Leap Day amuses me to no end.

    In light of Excel's horribly buggy code of handling Leap Day, I have to wonder if Microsoft's problems here might not be because it's Leap Day? Whaddaya bet Azure comes back up all fine and dandy once the date rolls over to 1 March instead of 29 February? I'm actually serious about this conjecture, this is not just an attempt at humor.

    On a different angle, does anyone else find it amusingly ironic that this service is named Azure, and now it's blue-screened? They've only gone up one letter -- now it's the ASOD.

    Cheers,

  • by sensationull (889870) on Wednesday February 29, 2012 @05:42PM (#39202573)

    OMG, you mean instead of having the mainframe at your building you have it on teh internetz and instead of owning you're own stack of them you rent usage on a much larger stack owned by someone else. You are so right, that is SOOOOOOOO! different. Yes there is more redundancy thanks to better infrastructure that allows for stacks of 'mainframes' in different locations but the vast change is basicly the business model.

    If you are really that amazed by this 'new, unpresidented' tech perhaps you should go and work in sales as their KoolAid obviouly works quite well on you.

    It's not exactly the same implementation the the model is very simmilar, just a bit updated.

God help those who do not help themselves. -- Wilson Mizner

Working...