Blackberry Network is Down

Brian writes "According to CNET and others, "A system failure at Research In Motion has affected BlackBerry users in the Western Hemisphere, a news channel reported on its Web site late on Tuesday. The infrastructure failed on Tuesday night, and e-mails were not being delivered to the handheld devices.""

Better headline needed

Common, this was an easy one!

"Blackberry Blackout"

Black Berry Down

Blackberry down, we got a blackberry down.

Leave No Email Behind

Re:Better headline needed

Does anyone have a mobile link for the website. I can't seem to connect to it on my blackberry.

Re:Better headline needed

Blackberry Jam.

OH NO!

My thumbs are twitching!

Re:OH NO!

First Post!

------
Sent from my BlackBerry Wireless Handheld

Crap!

Now I have to go in to work and explain that I dont control the Blackberry network........Cmon RIM!!

I feel a disturbance in the net...

...as though a million voices just cried out and were suddenly silenced.

Re:I feel a disturbance in the net...

Voices? Silenced?

Much you have to learn, young PDAwan.

Re:I feel a disturbance in the net...

Much you have to learn, young PDAwan.

Dear Sir, please send payment for:

• 1: New Coffee
• 1: New Monitor

to: 123 Fake street, Fakeville, NY, 12345. Thank you.

Man, this isn't good for RIM...

Not to mention the service interruption, but imagine the backlog once everyone realizes their crackberries are working again... Frankly, I'm scared.

Re:Man, this isn't good for RIM...

They have about a half hour to fix it before the stock market opens. From what I've seen, they've already taken a massive hit. I wonder how much of their gain since Oct 2006 they'll lose (They jumped from around $79 to over $130 in the 4th quarter last year).

They're already down close to $3.60 in pre-market alone. Ouch.

A hundred thousand angry users plus thousands of angry investors? Someone's got a case of the, uh.. Wednesdays.

There's some other coverage on this....

.... In the following locations:

http://www.theglobeandmail.com/servlet/story/RTGAM .20070418.wblackberry18/BNStory/Business/home [theglobeandmail.com]
http://www.wnbc.com/news/12339359/detail.html [wnbc.com]

But I can verify that their network is up (sort of) and Engaget.com confirms this:

http://gizmodo.com/gadgets/cellphones/blackberry-o utage-update-its-kinda-up-253214.php [gizmodo.com]

So YMMV.

must resist Great Disturbance In The Force joke...

Because millions of voices really ARE crying out in pain. Man, the cooler we make things, and the more that entire business cultures get built around this stuff, the more fragile it is. Just think of the war stories we're going to hear about people who've come to utterly depends on their Crackberries having lost a deal, not heard that a critical server was down, not realized that a surgery had been rescheduled, and so on. I wouldn't make a living if people didn't depend on fancy networked technologies, but it sure does feel like a house of cards, some days.

It IS a house of cards

Reason? It's cheaper than bricks.

Seriously, that's what's gonna break our neck sooner or later. We strip systems of their redundancies to make them cost less, we use cheaper components and the lowest bidder, we downgrade specs to the bare minimum because price makes right.

Technology already starts learning from nature, copying structures and models from millenia proven concepts. I think business could learn a thing from them too. Because nature has down what business wants to achive: Maximum output for minimum input. There is no such thing as waste and surplus in nature's makeup, if there was, it would be used for more output instead. So why do we have 2 kidneys, why is our brain able to adapt to damage, if it wasn't for the simple fact that this proved to be the more successful way in the long run.

But as long as companies are run by managers who care more about their next quarter report than the company itself, this won't fly.

Re:It IS a house of cards

Technology already starts learning from nature, copying structures and models from millenia proven concepts. I think business could learn a thing from them too. Because nature has down what business wants to achive: Maximum output for minimum input. There is no such thing as waste and surplus in nature's makeup, if there was, it would be used for more output instead. So why do we have 2 kidneys, why is our brain able to adapt to damage, if it wasn't for the simple fact that this proved to be the more successful way in the long run.

You must be new here. (To life that is...)

You will realize the fallacy of your statement soon after your 50th birthday. Nature isn't all that it's cracked up to be. Those bouncy little disks in your back will start taking on the appearance and flexibility of cardboard. Your prostate (assuming you are male) will enlarge and back your bladder up into your nostrils. Your uterus (assuming your female) will simultaneously enlarge, flop over and quit working. Your eyes will go bad. Various other bits will quit working while parts of your body that you didn't know existed will start creating problems.

While modern medicine can approach some of the problems, any repairs will be more like soldering a few new capacitors on to a Pentium II motherboard (and charging for an eight way Opteron system) than anything really useful. Oh, and you want a warranty?

Death and taxes to you sir!

Re:It IS a house of cards

Disclaimer: I am 27.

You do have a good point about things going out when you get older, but that's not nature's fault. Because if it were not for all our complicated medicines, we'd still have about a 45 year lifespan, and we'd be dead before a lot of those things started to happen. So Nature does have it worked out to the extent that it thinks we should be living... all the other things that happen stem from our ability to extend life through knowledge.

Re:It IS a house of cards

You mean... just ... maybe ... a few million managers could die? May I get my hopes up?

BES users potentially not affected?

For what its worth, my employer (in Washington D.C.) has their own Blackberry Enterprise Server (an on-site server that interfaces directly with corporate mail systems), and it appears to be unaffected by this outage.

Re:BES users potentially not affected?

RIM lost their primary NOC last night. Your BES transmits data to their NOC including mailflow routing and SRP information. This issue affects the Americas entirely and is currently fixed for Sprint and Nextel. GPRS/EEDGE providers are still unable to contact back to RIM at the present time. So, if you are a Sprint or Nextel shop, this would explain why you are not experiencing the issue at this time. However, everyone was experiencing it from 6:30PM CDT to about 3:00AM CDT.

Re:BES users potentially not affected?

RIM lost their primary NOC last night. Your BES transmits data to their NOC including mailflow routing and SRP information. This issue affects the Americas entirely and is currently fixed for Sprint and Nextel. GPRS/EEDGE providers are still unable to contact back to RIM at the present time.

I'll LOL if RIM is FOOBAR and can't GBTW but my PHB won't ROFL that the BOFH at RIM is AFK. LBC, 213.

Re:BES users potentially not affected?

NOC = Network Operations Center
RIM = Research In Motion
BES = Blackberry Enterprise Server
SRP = Server Routing Protocol
GPRS = General Packet Radio Service
EDGE = Enhanced Data GSM Environment
IHBT, HAND

Not very reassuring.

A single point of failure can bring down the entire network? Not very reassuring, especially considering Blackberry is predominately a business tool.

Dan East

Re:Not very reassuring.

especially considering Blackberry is predominately a CRITICAL business tool.

Contrary to popular belief. E-mail is NOT a critical business tool. You don't believe me? Then compare e-mail with the following:

Supply Chain Systems? Critical.

Customer Order/Customer Relationship Systems? Critical.

Manufacturing systems? Critical.

Payroll? Absolutely the single MOST critical application at most enterprises (especially during pay runs).

But, e-mail is not critical. When I've been involved in storage meetings, e-mail is always a Tier Three [wikipedia.org] application. In other words, while it is a useful tool, there are other ways of contacting people in an emergency. You know, like by telephone.

Re:Not very reassuring.

Wow, you must be one of the users that is freaking out over Blackberry today.

Supply chain is worthless if crm can't communicate with them, ditto for manufacturing, payroll is worthless if they can't communicate with accounting or HR.

Most of the systems I describe have a DR and/or failover capability because they are considered critical to the operations of the business. Personally, I have NEVER seen an e-mail system that has had DR or Failover. Beyond DR, ask an even more important question: In the event of disaster, how quickly does e-mail come back up on-line? Is it the first system recovered? Or are there 35 other systems that are recovered before e-mail? That should tell you how important e-mail is in comparison to your company's other systems.

Now, let's talk about using e-mail as an interface: Does the communication you describe have to occur via e-mail? I have been working in IT for 15 years and I have never seen a place that depends upon an e-mail to interface between any of the systems I described. For example, if I am running a Supply Chain system, I can keep the supply chain moving for hours without worrying about ANY e-mail communication with external systems. Well-designed Supply Chain systems use EDI (not e-mail) for communication. And, EDI is usually built with redundancy and DR capability If they don't use EDI, they depend upon someone manually entering the information into a screen, or even a flat file. If I absolutely MUST contact someone, I just call their phone. As for Payroll, payroll runs at even small companies are not dependent upon e-mail. The timekeeping systems interact with the Payroll systems via EDI or some sort of file transfer that is NOT dependent upon Exchange/Domino.

Hell, even if I am in hardware support and my e-mail server crashes, I should be smart enough to log directly into my messaging console to keep an eye on the status of my other servers. Most monitoring tools use e-mail to send messages to Operations. My operations department had better be able to log into their consoles to check the systems real-time and not be waiting for the e-mail system to recover.

Also note, for those things the phone is not an option, it must be in writing.

There are a few options here: If you are running a normal PO approval process, then there is always an expected time-lag in approvals. My manager cannot be expected to drop everything to approve my new business-card order system. And, if there is an emergency, well-designed systems have an override that allows for exceptions.

Try to keep in mind that many, many businesses do not reside in one office, or one building, or one postcode, or one country, before you make a silly comment like "e-mail is not critical".

Firstly, I work for a Fortune 500 company. We are spread all over the world.

Secondly, There is nothing silly about my statement. E-mail is NOT critical. People THINK it's critical because it is the system they have the most 'face-time' with. But, face-time is not equal to criticality. If your organization has built interfaces or business processes that depend upon someone approving something via e-mail, then you have a very poorly designed system. You can easily have interfaces move directly between systems without ANY human interaction. You can build custom web-pages to allow approvers the ability to make their approval directly in the purchasing system.

Cold-turkey for Crackberry-heads

Well, it was bound to happen with a centralized/heirarchical service model. Perhaps RIM will learn and go decentralized. Perhaps not.

More interesting will be the addicts reaction. Some people really hang on the devices and get addicted to their Crackberries. I wonder how they will adjust (most people will do just fine) and what lawsuits will result. Or if the plantiffs are too worred about simply having their service cancelled!

Re:Cold-turkey for Crackberry-heads

Lawsuits? Surely any network-based buisness with a legal department worth a lick of salt would include provision in their terms and agreements of services that cover such instances of blackouts, loss of service, or even financial collapse of the company leaving addicts with their network device without a network.

Can't be true..

Can't be true... I would have got an email telling me if it was......

more like all day tuesday

This wasn't really Tuesday night methinks. My boss who is Deaf and uses a blackberry like the dickens for contacting outside the office said she couldn't send from it all Tuesday day, starting in the morning.

Just count it like a Snow Day...

... and tell everyone to stay home.

Two Reactions

I can predict at least two possible reactions from end-users.

First: jumping out of a building due to the terror and pain of sudden withdrawal. It happens to heroin addicts, it can happen here too, folks.

Second: people wandering the streets of major cities bright-eyed and staring in open wonder, as though they were waking up from a long dream. Joining hand in hand, they frolic in the parks or whatever greenspace they can find chanting "Free at Last, Free at Last..." The clouds part, and an auspicious rainbow graces the sky. Oh, and I suppose there are other reactions: incoherent rage at no one in particular (ever chat with a cold-turkey smoking quitter?), unjustified rage at corporate IT for letting this happen, curling up in a fetal position in the corner, uncontrollable thumb twitching (almost like phantom limb pain).

Then there's another reaction: simply shrugging and going back to computer-based email and cellphones.

Not quite 'Western Hemisphere'

It's fine here in the UK, and the rest of Europe. As far as I can tell it's the US server, which 'only' serves the USA and Canada. As mentioned by others though, one point of failure destroying such a large portion of the network is bad planning at the very least!

Uh, system reset...

"Officials with RIM said they are trying to reset the system and told NewsChannel4 that they are concerned that the backlog of data, which will rush through when it comes back on line, could cause a bigger problem"

When in doubt, reboot!

Oh, nbw it makes sense!

I got in trouble with my wife last night for not reading her emails to me, and I could not figure out why synchronization wasn't working. Thank you, Slashdot! You saved my marriage!!!

Western Hemisphere

Did anyone read Western hemisphere and think WTF?

You don't see anything on the server

And I don't want to carry one, first I heard is when a user asked me about it. Sure enough, my boss is asking me why their blackberries aren't working and I'm in a meeting trying to show them how the system works... CURSE YOU RIM!

They could at least tell a time they plan on having it working again...

Quote much?

There were so many double-quotes in that write up that I thought it was a """paean to VMS""" [compaq.com]. It's nice to see that Python continues the practice [python.org].

Its funny

'cuz mine still works like a champ. Damn near everyone else in the office is calling and whining that they can't send or receive any emails. I tried rebooting the server a few times, rebooting other servers, rebooting the blackberries, and then I find this.

I gotta start checking teh internets before I start working...

More Info On Outage and Status

I was on RIM's rolling conference call last night and received some additional information. It seems that somehow they lost connectivity in their NOC. When they failed over to a co-lo they couldn't get the SRP communication up and running, causing all BES to fail in their connection to RIM. Fast forward a few hours and SRP is back up, but they cannot get critical components for email delivery to connect to their DB. Fast forward a few more hours and they get this up and running also. Currently, Sprint and Nextel are up and running, but the GPRS/EDGE service providers are still not receiving consistent mail flow.

one word response to this tragedy

Yippee!

The DoD emailed our Blackberries to tell us.....

Good ole' government. Last night at 11:37 PM the DoD distributed an email to all Blackberries informing them that the Blackberry service was down.

just like imus

I heard the blackberry mainframe was about to spill the beans on 9/11, thus blowing the lid off of civilization.. but it was taken down at the last minute, mysteriously.

Eedundancies and dependencies

A single point of failure is always a bad thing. Especially if a lot depends on it. But it can be some cascading effect too, for example power plants running on top output that couldn't swallow the sudden additional load when some parts of the network blew up.

It's a lack of redundancies. Redundancies cost money, and we want everything as cheaply as possible. So no redundancy. But hey, it "works". Usually. The question is, though, can we afford the blackout?

Imagine communication breaking down. No cell, no net, no data transfer, nothing. You could hear commerce grind to a halt. Nothing could be scheduled, nothing could be delivered on time, we'd simply break down. And that scenario isn't as impossible as it seems, because telcos don't have a lot of redundancies in their networks anymore either.

But we're depending on them. Often enough with our very lives. Yet we're not willing to pay the price.

But even if we did, would it be invested? I mean, afaik it's not like RIM made some kind of promise that the service would work. And as long as you can't hold them responsible for the loss, of money and maybe even life, they certainly won't add anything that costs more than it has to.

Damnit man, I need details!

It'll be interesting to see just how much detail we get concerning what exactly failed and why the current level of redundancy didn't kick in.

I mean, it's not like the power supply failed on an NT 4 server (you know, the one with the post-it taped in place that says "East Coast B-berry server, DO NOT POWER OFF!!!"), it's not like somebody accidentally drove a nail trough some coax in the wall at RIM's HQ, it's not like somebody accidentally typed "rm -r *" at the wrong prompt. There has to be some serious "Thank God I'm not the one stuck cleaning up that mess" stuff going on here.

Funny unrelated story. We had an exec looking at one of the blackberry's. He put in back in the hard case and was fumbling around with it and saw the "RIM" on the back. Then he asked, in the innocence that only an exec can have, "So, how does one go about getting one of these 'RIM' jobs anyway?" When I am canonized as a saint, one of my miracles will be "not laughing at the VP who asked how to get a rim job."

For a communications company

RIM is pretty bad at communicating with their end users. They do not have a network status page. When you call in to TSupport, there is no message indicating a general interruption in service. This is totally unacceptable. I spent two hours on hold last night after checking over a client's BES and not finding any issues locally. That hold time could have easily been reduced to a couple of minutes if RIM had a system in place to notify users of problems. Even Time Warner, which has terrible customer service, has service outage notices as part of the call tree when you call in for support. I would guess that this is a conscious choice on RIM's part, to try and appear as if they never have problems. Bad idea. Customer frustration sets in big time when you don't talk to them during outages.

RIM: Get better at communicating with your client base or they will go elsewhere.

philo

Single points of failure

I don't understand how the ENTIRE network can be down. Hasn't anyone learned from the design of email?

But then again I can't see what Blackberry gives you that you can't get with an IMAP server anyway.

Oh, no

Now what's a busy executive supposed to do when you're giving them a presentation? Actually pay attention to you?

What about users with Blackberry Servers

Does this affect us? We haven't had any issues but I am wondering if it is because we have the enterprise server.

Remember POP over TCP/IP ?

In the meantime, my Treo is working fine. Why? Because it relies on the POP protocol used over a TCP/IP network. Traditional, boring, standards-based and reliable.

Yes, my TCP/IP is provided over GPRS, but I hope that my next Treo (linux-based?) will offer a Wi-Fi connection as well.

Pascal

It's back up...

... 9:52am EST (Rochester, NY)

mine has been working for the past 24 hours....

then again it's a treo and not dependent on a provider with a single point-of-failure.

Mobile outlook 1, crackburry 0.

New service outage updates at ZDNet

This is Russell Shaw. I do the BlackBerry beat blog for ZDNet. I've just called all the carriers, and RIM, and have just posted the latest at: http://blogs.zdnet.com/blackberry/?p=135 [zdnet.com]

RIM says service back for "most" customers

From the statement: "Root cause is currently under review, but service for most customers was restored overnight and RIM is closely monitoring systems in order to maintain normal service levels."

http://www.networkworld.com/community/?q=node/1416 9 [networkworld.com]

Good. May it be permanent.

Maybe people will actually pay attention during meetings today. It always stuns me when my team has meetings to discuss, say, a plan to realign billions of dollars of procurement funds, and some senior jerkoff is too busy shooting off emails to pay attention to the rather more important matter at hand. I always joke that we need to slip in a random bit of classified info so that everyone will be required to check all their devices at the door.

This just in!

Karl Rove in ER being treated for shock following Crackberry failure. Nation's government falls. Peace breaks out, we run a surplus, Crawford is consumed by tornados that miraculously miss the protestors.

Wait, am I awake now? Damn.

I'm already tired of hearing about this

People are ridiculous. "Oh no, I can't receive emails while I'm in my car!" Can't these people find anything useful to do with themselves in the interim? Barring very, very specific and anamolous exceptions, if I have an employee who basically shuts down because of a Blackberry outage, I think I need to find another employee. It's like, wow guys, it's amazing anyone got anything done back in the Dark Ages of 2002 when we didn't have Blackberries!

Grow up, Blackberry users, and find something to do.

Exchange Activesync is better

I would prefer Microsoft's Exchange Activesync better than Blackberry. Both configuration wise (EAS is out of the box in Exchange 2003/2007) and in usability.

BlackBerry vs BCSS

In addition to a BES are running a Sprint/Seven product known as Business Connect Server Solution which offers the same sort of middleware solution as a BES does but for a client (BizConn) that can run on PPC/PalmOS/and some Java-based handhelds. We have a saying in the office - "If RIM is down, it makes headlines. If BizConn is down, it must be Friday". My own experience is that RIM's managed network is more reliable and better supported. Sprint's continued mismanagement of the BCSS back-end has caused us no shortage of grief. They refuse to provide an SLA for the service, which has forced our team to dial support back to 8x5, M-F for the app. They push client updates with little notice and no advance download available for testing. Their support line offers perpetual catch-22. They refuse to issue ticket numbers for incidents and then request ticket numbers on subsequent callbacks. I can see the rational for the arguments that RIM should decentralize, but I would much rather deal with them in times such as these than the telecom proper. Then again we also run Exchange and offer ActiveSync. Don't ask why we offer 3 flavors. Please don't ask.

RIM Outage Mail List.

There's a mail list setup at http://www.c2security.org/mailman/listinfo/bb-outa ge/ [c2security.org] to notify people of this. BB-Outage is a mail list for people to notify others if RIM's service is down in a particular area. if RIM is down in China, US, A particular state, city, etc, for people then post the area to the list so others are notified. It's probably best to subscribe from an address that's not associated with your Blackberry since if RIM's service is down, you won't get the e-mail anyhow

Does this effect SMS ability?

I was planning on purchasing a Blackberry shortly. I know the e-mails sent have to go through the RIM servers, but I guess this would not effect SMS or phone calls on the newer Blackberries right?

Good, Another excuse!

I've avoided crackberries and similar products like the plague since they came out.My pda has a cable that connects it to my cell phone if I really need to get on the net for something... I picked up a treo a while ago and immediatly put it back down on the shelf. If I want to make a phone call, I'll use something small that resembles a phone. if I want to surf the net or send email, I'll use a computer, and if I want to set up a meeting on my planner, I'll use a palm pilot that is designed to be an agenda. Combine too much together and you have a device that's too large for it's main purpose, to small for it's secondary purposes, and generaly not equiped to handle any of it's jobs properly. Now with blackberries proving they don't have a solid network, it's just one more excuse for me to keep avoiding them. My desk phone forwards to my cell, my voicemail sends a SMS message to my cell, urgent emails send a sms message to my cell... servers page me by sending a SMS message to my cell... I don't really need any more instant information... if it's a critical issue call me... Any important server issues that are detected by the servers can easily be summarized into a SMS message, which has been around for a while and has been relatively stable. Actually 90% of the server messages I receive fall into ("Application log full", "Power fault, UPS active", "Power restored", "Server X is rebooting" or "Server X is MIA")

Re:Where is the redundancy???

The redundancy is right there, in your post, which was duplicated by earlier ones.