Follow Slashdot blog updates by subscribing to our blog RSS feed

 


Forgot your password?
Close
typodupeerror
×
The Internet

AT&T vs MCI on Network Outages 62

Posted by Hemos from the let's-all-play-the-blame-game dept.
James Ensor writes " Cnn.com is running an interesting article comparing MCI Worldlnet's recent frame relay outage to an outage that AT&T had last year, comparing their method's of handling the situation. Good Reading. " It's fun to watch major corporations blunder about. Unless it's your connection that's down.
This discussion has been archived. No new comments can be posted.

AT&T vs MCI on Network Outages More

AT&T vs MCI on Network Outages

Comments Filter:
  • It finally happened. I knew one day I'd browse on over to Slashdot and find a naughty story about the company I work for. Well, I can't say they weren't asking for it.

    Yeah, the point about all the companies within MCI Worldcom rings true. Like the MCI local network, MCI long distance network, Worldcom long distnace network, and the Worldcom local network (which is actually the MFS local network and the Brooks fiber local network). Then you've got a layer on top of that, like UUNet. Not all that straightforward. They are working on these issues, though.

    Responsibility issues? They are there. You should see the hoops I have to go through just to get someone in the same building to investigate a problem within our own LAN. MCI highly emphasises "the procedure".

    But the story that Slashdot should be covering isn't the frame network outage. It is MCI's agreement to sell off their IT department to EDS. Most everyone assumed it has gone through. Guess what? They made the announcement 6+ months ago, and the deal (along with IT) is hanging in limbo.

    Will we work for EDS? MCI? Do we get our current stock options? Benefits? Policies? Management is saying absolutely nothing. Rumors are flying that the negotiations are going bad and neither side wants to walk away because of a hefty $$ penalty. Even an arbitrator hasn't helped. (That's right, an arbitrator just to reach an agreement. Heaven help both sides over the next five years.)

    The entire IT department of a major corporation that can't even say what conditions they will be working under next month. How smart is that?

    The main problem is that EDS is a very bad fit for a company like MCI Worldcom. EDS is great for an organization that is low-growth, low-innovation, and reliability situations aren't critical. Say, a local government. But a phone company? Oh boy.

    I can't say I know an employee that is enthusiastic about the merger. The only management that claims to love the idea is from the VP level and up.
  • it is short for people. u know just leave some letters out. learned that from the jews.
  • We switched our ISP's from MCI to Telwest, MCI left our T1 on for 18 months before disconnecting it. (Then tried to bill us!)
  • AT&T (wireless services) does not like to blame people for a problem, just fix it quickly and properly.

    Then do tons of documentation so it never happens again!

    I work daily with the technical support team. Their job is to solve problems, not pass the buck. And if they cant fix it, it goes up the chain of support, And yes all the way to Frank Ianna.

    Our goal is no more than 5 minutes of unplanned outage a year. Thats our motto at the office. (Damn good one also)

  • AT&T. We were using MCI for our T1 services and it seemed like everyweek they had some router in Atlanta that was on fire, had grown legs and walked off or had been somehow run over by a bus. Honest to god. They said a bus hit their relay station. We have been using AT&T now for almost 5 months and haven't had a hiccup, they aren't the fastest to get out here and install the stuff but they definately have the better of the two services.
  • If the CBT can't plan ahead and shell out a little extra cash so they can be multihomed, they deserve whatever crap they get. I wonder what happens to them when NT crashes. Does the whole place go belly up or do they actually have some kind of HA setup.
  • i too have seen this at several companies and it is very sad to see adults act like 5 year olds around a broken cookie jar.
    bottom line: lazy people that can't prevent disasters can only find scape goats. stay away from them or they'll pick YOU!

  • Management is not outsourced (Score:1)

    by Anonymous Coward
    That CNN article has a number of significant errors. MCI Worldcom does not outsource the engineering of the Lucent frame relay network. Operations is handled primarily out of the 4th floor of the Infomart in Dallas. There are technicians located in several locations, but they primarily just take care of turning up new customer ports, and troubleshooting individual customer problems (failed local loops, etc). Backup for Dallas is located in Cary, North Carolina (which also serves as Operations for the "other" frame relay network).

    I agree MCI Worldcom should take the blame. They should have focused more on the customer's needs and backed out the software to get the network back up. It was clearly the fault of upper management for listening to the vendor and his "assumptions" and "recommendations". The customer should be faulted for blind faith, not the vendor.

    If the sales reps had information and decided not to contact their customers, I'm not sure if that's a management problem or a weasely sales sales person problem. Either way, there wasn't very much information available because not very much was known or understood about the problem. For the longest time (the first week or so)Lucent even claimed it wasn't a result of the software upgrade!

  • Same here. I work(ed) (still part time when they need me) for a small ISP (no T3's there anytime soon) that used Sprintlink for backbone and Sprint for local service. First, their local guys kick ass. After all the local telco jokes, these guys were a pleasant surprise. They knew how to do it and could deliver a channelized T1 or PRI a week within ordering, 2 weeks in the worse case. Another ISP I have had contact with in GTE land has to order 3-4 months ahead and the usual reaction is T-what?

    Sprintlink has been very reliable. We had a little trouble with not enough bandwidth leaving the local POP we connect to, but they got it figured out eventually. In fact, we were going to replace them with someone cheaper, signed the contracts, got the new feed in, and after testing it, are trying to get out of the contracts. They sucked. Sprintlink is well worth the extra money.

  • Just like the company I used to work for. Customer has a problem, x'fer them to someone else. Even though our department most likely could finger the solution out faster than the "correct person". Kind of like "The buck stops over there, besides it's not my responsibiltiy."
    It seems like in the race to save a dollar companies have forgotten that most folks will pay a little extra for good service. Perhaps they should teach that concept in business school. Wait, I graduated from one and they didn't teach that. To damn busy doing SWOT analysis, Gannet charts and critical path analysis to worry about User Experience. IBM and AT&T haven't been around forever and a day letting their customers go away mad.
  • My man, you are a much braver person than I could ever be.

    I thought working for a large companies would insure stability in your career but with the woes of MCI (does our frame relay and one T-1 from UUNet) and SGI (go back to Silicon Graphics) who needs that type of instability?

    On the other hand, my current small company is trying to write up policies and procedures so I guess I will be looking for another job soon.


    ChozSun [e-mail] [mailto]
  • Where does Sprint fit in to all of this? We've been buying several T1s and ISDN lines from them for a couple of years and have had almost no downtime. Even when we do, a quick call to their TAC either gets it fixed or gets me an explanation (talking to the actual techs who work on things .. not some brain dead support rep). In fact, I have a full DS3 feed on order from them right now. I don't know everyone else's experience, but we've been very happy with their service.

  • I have a lot of friends that are AT&T employees in R&D. On day 8 of the MCI outage, I mentioned the MCI outage to them. They hadn't heard about it and were damned pissed about it also. After April 1998, when they saw AT&T outage on the front page, etc. and all the heads rolling around them, they couldn't believe that MCI wasn't being eaten alive. 8 days! Then MCI goes and blames it on Lucent. That really got these guys going since Lucent all but shares buildings with them (actually they did up until 3 months ago).

    Thank god one of the journalist crowd has a clue about this and published. Very impressive.

  • I was reading an article about the outage on news.com on day 9 - when I got one of those "Hi, are you happy with your current long distance service?" calls. It was from Qwest, a company that I haven't heard much dirt about yet. Yet.
    My old housemate had moved to Europe and they offered decent international rates, so for once I said 'yes' to one of those telemarketers that I usually hate being bothered by.
    I did tell them that the only real reason that I was switching was because MCI really messed up. Of course, I was talking to a 3rd party telemarketer who probably has 1000 resumes our trying to get a real job and really doesn't care about telecommunications in the least bit.

    But it made me feel better.
  • Yeah, "significant outlay for R&D", that would explain the LT Winmodem.
    Heheheheheheh. Well, what can I say? They also brought us UNIX, Plan 9, C and C++. Give 'em a few breaks, eh?

    And besides, there's even a beta of a driver for the thing, if you happen to be running OS/2 ;-)
  • You know it funny, but I agree heartily. I managed to win a consulting contract with a company by using a line I got from my first boss, "Never underestimate the bandwidth of a truck full of tapes..."

    Chris

  • AT&T Worldnet is AT&T's Internet Service (One of the top ranked national ISPs, too). Perhaps there's where the misinformation is coming from.


    j-a-w-a-d------------------------------
    replace ,'s in e-mail address with .'s.
  • After a company grows beyond a certain size, it gets increasingly tough to just "wing it"...

    Maybe you mean something more sinister by "policies and procedures"?
  • Anyone notice, that while talking about frame relay, they have a Centronics (SCSI/Printer) cable
    picture on the article?
  • Did AT&T have a monopoly at one time and learned from it (the hard way)? PPL and Companies should learn again that "The Customer is the King", as we Germans sometimes say.
  • What's most interesting about the whole story is that in a LA Times "Innovation" column a few years back (which I can't locate), AT&T was accused of being unable to do anything well other than provide long distance telephone service.

    This rang somewhat true, and in the years since, I watched AT&T live up to this accusation. Until now...

    I'm wondering if I was a bit too eager to write them off. Maybe AT&T actually does have a future...

  • ...this sort of thing is likely to continue in this day and age of mega-mergers. AT&T did the Right Thing(tm) and I'm pleasantly impressed. I guess it is possible for a 600 pound gorilla to handle a situation intelligently...
  • They have a "just enough" attitude to their services. Just give your customers as little as possible and continue to overcharge them. Outages are one thing, how come they refuse to upgrade overloaded networks?

    Putting blame on others is also quite common.

    I get that impression from many American companies. An embedded company called TERN gave me similar BS at a previous job.

    Their development tools were antiquated, the board was defective and they refused to replace it. Transferring new code kept giving CRC errors, and sometimes it would get through, but with errors. They gave me a similar runaround, claiming that all of our computers, including my Dell Dimension XPS 200, had "nonstandard" serial ports that couldn't keep up with 57.6k. Yeah right.

    I finally got to the "CEO" (small company) who told me that GM had the very same problem with 100 different new computers, and that all of those computers were bad.

    Yeah, right.

    We hauled the boards out of both of the projects we were working on and replaced them with cool, reliable ZWorld [zworld.com] boards.

    Development time shrunk immensely, and reliability was great. Their tech support was also extremely good, they would email me code snippets and everything. They also encouraged messing around with their libraries, I thought that was excellent and found them well written and easy to hack.

    Hmmm... Parallels the referenced article quite well. Goes to show, if you're not satisfied, walk. It makes all the difference.
  • Myself and a few thousand other people were completely left in the lurch when MCI sold its dial up service to Cable and Wireless...Neither company had a full accounting of their subscribers, but that didn't keep both sides from debiting money from my checking account for services never rendered! I had to close my bank account to stop it!

    What's really sad is the freakin' Chicago Board of Trade got the same clueless response I did! It was always some other group's problem when I called. My account number started with a J and whoever was on the phone didn't handle J accounts. I must have talked to 10 different finance groups between MCI and C&W before I found somebody who knew what was going on...

    The business weasels in charge probably wouldn't have heard about the outage if the stock hadn't tanked that day.

    MCI sucks, pass it on...
  • Situtation is being handled by blameing everybody.
    Yes indeedy.. the quickest way out of a situation is to make sure you don't get the flak for it..

    Of course, that means that the "smoking gun" left the gunpowder traces on your hand(s).. so, in the end [if not before] we all know you did it anyways.. so why bother hiding it?
  • AT&T's way is defintely the way a company should handle problems and you know that next time AT&T will be able to solve the problem quickly. No one likes to be blamed for a problem and blaming someone that you are in buisness with will only alienate them from. Unfortunately I have encounted too many companies with this attitude here in the valley and I was very suprised. After a while it became clear to me that in most companies blame needs to go to someone and woe to the person who gets up and says "sorry that was my fault I f%@ed up". You either get fired or get a bad performance review because no one around you admits to any wrongdoing then you must be only incompentant monkey working there.
    just my 0.02$
  • that the quote at the bottom of the page is:

    "One reason why George Washington Is held in such veneration: He never blamed his problems On the former Administration." -- George O. Ludcke

    Serendipitous or nice touch?

  • Reminds me of the way Ebay kept making up excuses for their down time: it's not our fault, blame Solaris!

    I always wondered how they expected to maintain a good relationship with Sun's engineers, after they pointed the finger at them and called them every name in the book.
    --

  • It's actually MCIWorldCom, not MCIWorldNet.

    Note that this is the same company who still handles payroll via paper and Federal Express.

  • Paper's a shame, but if you want true high
    bandwidth (like NASA needs sometimes for instance)
    you go with FedEx.

    Came up in a job interview once -- interviewer
    asked me how I would transfer terabytes of info
    from their NY office to their SF office as
    quickly as possible (this was a good real
    problem they were dealing with at the moment).
    They had a few T1's which were nearly saturated,
    but a pretty good internal backup system (you've
    got to if your throwing around that much data).
    I told them to put the data on tape (clearly as
    dense as they have available for best results)
    and overnight FedEx them. Cheap, fast, and
    hopefully reliable.
    (They made me a good offer but I went elsewhere
    --don't know if they FedEx-ed their data after
    all or not).
  • How the hell would a sales force successfully deliver info. to the customers by email if the whole problem was that their network connections were down?
  • One thing that this article points out, indirectly, is that honesty and responsibility pay. AT&T, by taking its share of the blame and being open about its problems, probably gained quite a few customers; MCI, by keeping quiet and shrugging off the blame, looks like it could lose some major clients.

    We can draw a lesson from this. Most of the source of anger against Microsoft is the fact that they close the ranks and engage in finger-pointing when there's a problem, and attack the competition with FUD instead of concentrating on a better product. The Linux world, which is for the most part inherently honest (thanks to the Open Source paradigm), earns the reputation of a better product.

    Another lesson to draw from this: it is better to be open about the strengths and weaknesses of Linux than to just plain Microsoft bash. But then again, this is already in the Linux Advocacy HOWTO.

  • I know that QWest provides LD service for a lot of schools and colleges here in the Northeast. I guess ECCI gets contracts with individual instituions and then bundles the stuff together to get good QWest rates. From what I hear, they're pretty reliable (at least better than CampusLink, who just does individual school stuff and where you have to just *hope* stuff works).
  • This sort of thing just makes me feel better about trusting the big bad former monopolies with their specialties :)

    Just goes to show one reason why AT&T and IBM have stuck around this long after having their monopolies broken up -- they realized that they could no longer compete with the same old "kill the competition" practices, and went ahead and started providing service. Of course, I also like IBM and (what used to be Bell Labs) Lucent for their significant outlay for R&D. That sort of thing can't be over-estimated in the value it gives a company that can afford it.

    What is the meaning of this ramble? Not much, except to maybe provide some food for thought towards what Microsoft might be like it gets its ass kicked by the DoJ :)
  • That is a very good idea. You could of course also have told them to have some slaves put it on ticker tape and throw it out the window. Ha Ha Ha
  • They've been complaining about MCI for quite a while now, and viewing the situation from afar, their bitching is justified.

    Redundancy and QoS? Cripes, the half-assed (or full assed) ISP I belong to is dual-homed, and probably has no problem keeping things running because of it. MCI seems to have no QoS policy.

    I think the Chicago Board of Trade should send a wake-up call to MCI and change providers, possibly to AT&T. THAT would get their attention. If it doesn't, it should signal the other big subscribers to MCI to find other sources of bandwitdth.
  • Maybe if the net monkeys at CBT knew what an AS and BGP4 was and, more importantly, how best to use such techniques/technology, they wouldn't have had this problem. Instead, they stick all their eggs in one basket and hope that nothing goes wrong (after all, it's MCI, right?).

    If you really care, you don't put your life in someone else's hands. You plan for your primary provider to go tango upsilon. You plan for the day when the grid goes south. You plan for the day when everyone decides to SELL SELL SELL and flood your feeds with orders.

    One thing I've learned about finger pointing; when you point a finger at someone, three more point back at you. When you trust a big carrier to run your network because all you want to hire are knuckle dragging idiots, then you have no one to blame but yourself when things go wrong and you can't fix them.

    I wouldn't expect any less from the CBT crew - they think rebooting the NT systems every evening to maintain reliability during the day is actually reasonable. Puh-leeze :-P

  • My former employer ordered three T1 lines from these people. Two were totally phone trunk lines, the third was 12 channels voice, 12 channels data.

    1) It took them two months to get it setup and installed! All the while, we were chewing on Sprint, trying to get them to keep us on for a little while longer (we'd already dumped and reactivated the Ts twice).

    2) When we finally did get them installed and working (which, BTW, we had to do ourselves. MCI/WC swore that the tech they sent would be able to configure a Definity G3 switch. he didn't. I learned a lot about phone switch administration that week), the two voice T1s would lose channels at complete random intervals, sometimes dropping to as few as 2 channels per line.

    3) The third T1 had all 12 voice channels......and no data. It took over 3 months before we had an Internet connection on that line. Well, 2 months, 27 days, over 400' of ethernet wire (212' run installed and replaced), and one bridge later.

    Fortunately for the ADSL users around my home, we have GTE running the lines (yes, I know, most of you hate GTE, but they've been wonderful for me) and only 2 providers subscribe to MCI's backbone. One (mine) colocated their Cisco with GTE at the frame relay network interconnection point, so my ADSL pipe is only 1 hop away from either Level3 or UUNet (I know UUNet is part of MCI, so we just used Level3 for those 10 days).

  • BTW my post was not meant sarcastic. I think to transport terrabytes from one location to the other you should really not use the internet or 56k modems ;-)

Slashdot Top Deals

The moon is made of green cheese. -- John Heywood

Close