Verizon's NYC 911 System Shutdown 346
Dead Nancy writes "A combination of human error and software that didn't anticipate it brought down New York City's 911 emergency line for several hours on Friday night."
This discussion has been archived.
No new comments can be posted.
Verizon's NYC 911 System Shutdown
Related Links Top of the: day, week, month.
I THINK THEY SHOULD CONTINUE the policy of not giving a Nobel Prize for paneling. -- Jack Handley, The New Mexican, 1988.
Quick! (Score:4, Funny)
Re:Quick! (Score:3, Funny)
Re:Quick! (Score:5, Insightful)
Remember, everytime a stupid person dies the average intelligence of the world goes up.
Won't somebody please think about the children!? (Score:3, Insightful)
Re:Quick! (Score:5, Funny)
912 = "Real" emergency number from the Stonecutters episode
So.. (Score:5, Funny)
Re:So.. (Score:2)
Re:So.. (Score:5, Funny)
Re:So.. (Score:5, Funny)
Yeah, it went something like this: "911? Can you hear me now?"
I've had that happen! (Score:5, Interesting)
Civil Defense Sirens (Score:5, Insightful)
311 (Score:5, Informative)
Re:311 (Score:2, Interesting)
I know about 411, but that's not a non-emergency number.
Any other *11 numbers we should know about (and what do they do/where are they available?)
511 (Score:5, Informative)
That number is handy, but I have to admit that the time I called the pizza place that I knew had caller id and asked them what my phone number was makes a lot better story!
NOTE: I just tried this on my home for and it worked. My phone company is CenturyTel.
Re:511 (Score:3, Informative)
Here in California, 511 (so the signs/radio ads say, I haven't tried it) will give you travel information - road conditions, traffic accident reports, bus schedules, carpooling information, etc.
-- Joe
Re:511 (Score:3, Informative)
Now, you just call the power company at 1 888 POWER-ON and it repeats the number back to you
Some other *11s around here...
611 = repair
711 = TTY/TDD operator from payphones
Re:511 (Score:2)
Re:311 (Score:5, Informative)
In NYC 311 is used for Non Emergency calls. Noise, broken lights, etc. It has taken a tremendous load off of the 911 system. People used to call 911 for everything, I think the call volume is down 30- 40% since 311 went into effect.
Re:311 (Score:3, Funny)
Sounds to me from this story that they've got call volume down 100% now. Good job!
Re:311 (Score:5, Informative)
Re:311 (Score:3, Interesting)
I recently sat on hold with 911 for literally 20 minutes after I watched a vehicular hit-and-run accident from my car, where the
Luckily I was close enough to get the a-hole'
Re:311 (Score:4, Informative)
WTF? How would you feel if *your* car was the one damaged in a hit-and-run?
Re:Mods on crack (Score:5, Informative)
Monopolies and software (Score:5, Interesting)
Re:Monopolies and software (Score:3, Insightful)
Re:Monopolies and software (Score:2, Informative)
Re:Monopolies and software (Score:5, Interesting)
- 6 Nobel Prizes in Physics shared by 11
scientists
- 9 U.S. Medals of Science
- 7 U.S. Medals of Technology
- 1 Draper Prize
- 6 Marconi International Fellowship Awards
- 7 C&C Prizes shared by 12 scientists and
engineers
- 27 IEEE Medal of Honor winners
Now I don't think monopolies are a good idea in the general case, however AT&T's results were at least halfway decent.Re:Monopolies and software (Score:4, Interesting)
Bad Example (Score:5, Funny)
AT&T was far from perfect (Score:5, Informative)
Yet at the same time, daytime Long Distance cost over a dollar a minute from NY to California. Phones and telephone equipment had to be rented from the phone company, so technological development in many areas (faxes, answering systems, business telephony) came at a snail's pace. Also, if you think DSL rollout was slow and overpriced under the Baby Bells, just imagine what it would have been like under Ma Bell. All things have their price, and this was a high one.
Furthermore, AT&T had their problems as well. On at least one occasion, they had massive network failures due to a combination of-- guess what-- human error and software failure.
Re:Monopolies and software (Score:4, Informative)
I remember the "good ol' days" of Ma Bell's monopoly, before the big breakup. I'd take exception to the use of the phrase "*real* quality of service".
The only good thing I recall about those days was that most Western Electric phones were virtually bulletproof, the telco equivalent of an IBM Type M keyboard. But that's about it.
For starters, since the telco owned everything on the network, adding an extension phone was a violation of their terms of service, and they'd come down hard on you if they found one in use (and don't think they didn't check up on people; they did). The sound quality was vastly inferior to what we now have: long distance sounded like long distance, but even local calls could sometimes be rendered unintelligible by the monopoly's antiquated switching system. Service in rural areas pretty much sucked hind tit; even in the late '70s it was party lines or nothing in certain towns in upstate New York.
Even worse, Ma Bell's responsiveness to consumer complaints was a national joke. Remember Lily Tomlin's character Ernestine? One ringy-dingy... That was a caricature, of course, but one grounded in truth.
After divestiture, things really changed for the better overall. The relaxation of restrictions on what could be placed on the network meant a boom in devices like answering machines, fax machines, and modems. Had the old pre-1984 restrictions been in place, what do you think the effect would have been on BBSs and dial-up access to the Internet? Imagine having to pay extra in order to have a modem connected to your phone line. Sound quality improved largely due to technological advances, but had the monopoly still been in place, would there have been any incentive to upgrade the telco network?
Yes, AT&T had been on the cutting edge of computer science and electronic engineering for decades. But had the break-up not taken place, we'd still be using a phone system worthy of the movie Brazil.
k.
Re:Monopolies and software (Score:3, Insightful)
I ran a 10 state wide call center and the data processing branch of our company and dealing with AT&T was not easy even though we had nearly 100 lines with them. The customer service representives though at AT&T were outstanding, however t
The problem (Score:5, Informative)
When you allow competition those that attempt to compete are forced to either charge less than it costs to supply the service or charge more. If they charge less but can't get the customer base they go out of business. If they charge more people tend not to switch. And if you don't charge enough less, nobody cares enough to switch.
Cox saves us all of a buck or two over Qwest on phone service. We never bothered to switch until we switched to Cox for high speed internet.
And of course the only reason Cox had the money to implement phone service is because they're the monopoly on cable service.
In cases like this it's actually better for the government to force the monopoly to act in the best interest of the people than to allow competition which just gives people the false impression that it'll lead to cheaper prices and better service.
Competition in these cases are almost always forced to either cut corners to survive or charge more.
Ben
Re:Monopolies and software (Score:3, Funny)
Monopolies dont run the phone system (Score:3, Interesting)
This is yet another example of why the breakup was a bad thing.
Re:Monopolies and software (Score:4, Interesting)
Nice guys. I wish I had a heart attack just so I could file a lawsuit. That's the only way things change nowadays.
Re:Monopolies and software (Score:3, Insightful)
This failure required two failures in a row - one human, one software. After this failure, both systems will be fixed, so this failure won't happen again on this system. Furthermore, it won't happen again on any system those involved in cleaning up work on - we learn by o
NYTimes (Score:3, Informative)
username: slashdot2003
password: slashdot2003
Re:NYTimes (Score:5, Insightful)
Re:Doesn't work? My mistake (Score:5, Informative)
Re:Doesn't work? My mistake (Score:2)
Re:Don't mod me up please! (Score:2, Funny)
It was a heist of great porportions. (Score:4, Funny)
But you fail to realize that a big heist probably took place. I saw George Clooney and Matt Damon do it to Las Vegas. They let off this big ass EMP and shut down the power to the whole city! Long story short, they stole a shitload of money and got away with it.
Don't let these stories fool you, that is exactly what happened here.
DDOS 911 (Score:5, Interesting)
Here a few years ago there was a sting of robberies where the thieves called a whole lot of people and convinced them to "test" the emergency response system at a specified time a few days later.
All of a sudden there were hundreds of simultaneous calls reporting accidents, fires, muggings, heart attacks, rapes, robberies, etc. The thieves robbed two banks and a big-box store while the police were tied up.
Re:It was a heist of great porportions. (Score:4, Funny)
Feh. (Score:5, Insightful)
Re:Feh. (Score:3, Interesting)
So, they've just announced to all the terrorists in the world - this problem still exists, is going to remain there and if any terrorist organisation can get one
What *kind* of human error? (Score:2, Interesting)
You're right. (Score:5, Insightful)
Now imagine what a genuinely malevolent person could accomplish. Perhaps a single individual shouldn't be capable of disabling such a critical system.
Re:You're right. (Score:5, Insightful)
Full text (what happened to google partner links?) (Score:5, Informative)
The emergency system broke down about 7:20 p.m. after a Verizon engineer who was making service changes to a bank's telephone numbers in Brooklyn inadvertently included numbers that are used to carry 911 calls, city and telephone company officials said. The numbers were close in sequence, the officials said.
The 911 calls then ended up being rerouted to the bank's phone system, and callers heard a busy signal. City and Verizon officials said that while the backup system in place for 911 was functioning properly, it failed to pick up the calls because it was designed to catch a technical error, not a human error that would be interpreted as simply a change of instruction.
Daniel Diaz Zapata, a Verizon spokesman, said the telephone company would now require a second person to double-check any entry of data that could affect the 911 system, and said the company planned a thorough review of its procedures that would be documented in a report to the city within a few days.
"We determined that a human error resulted in the accidental rerouting of phone calls during a procedure to upgrade service for a corporate client," Mr. Zapata said. "We have immediately altered our processes to ensure this type of situation does not reoccur. We have assured the city that we took immediate steps to make sure this doesn't happen again."
Citing privacy concerns, Mr. Zapata declined to identify the Verizon engineer, except to say that he was a veteran of the company. Mr. Zapata said it was unlikely that disciplinary action would be taken against him.
Police and fire officials said yesterday that they had no reports of injuries during the 911 failure. Fire officials said that about 60 firefighters responded to a major fire, at 3301 Foster Avenue in Brooklyn, which was called in at 8:49 p.m. by someone using a fire alarm box on the street. There were no injuries in the fire.
Paul J. Browne, the Police Department's deputy commissioner for public information, said the department immediately adopted emergency procedures, like requiring e officers on patrol to turn on their flashing lights so people could find them easily and increasing staffing at precinct station houses to answer phone calls. But he said there was no reported increase in crime.
"This didn't present an opportunity for the criminally minded - like the blackout did - because probably most people were unaware that it was out of service," he said.
However, several City Council members expressed anger that the 911 system could have been so easily disabled, and called for creating a more effective backup procedure.
"It's an emergency wakeup call," said Councilman Peter F. Vallone Jr., the chairman of the Public Safety Committee, who plans to hold a hearing about the incident. "We don't have an adequate backup system for 911, which is more important than ever as we fight the war against terrorism."
Gino P. Menchini, the commissioner of the Department of Information Technology and Telecommunications, said city officials were working with Verizon to ensure that the emergency system's numbers were clearly identified, and that its software and equipment were protected from similar human errors.
But Mr. Menchini emphasized that the emergency system already had many built-in safeguards, such as the ability to route 911 calls through either of two central offices and their 911 answering centers. "The bottom line is, 911 works very well, and it's worked very well for a long time," Mr. Menchini said.
Several emergency services experts agreed yesterday with Mr. Menchini, saying that New York 911 system compared favorably with those in other large cities and that an error like the one made by Verizon could not necessaril
What is 911s number again? (Score:2, Funny)
Google brings good to all (Score:5, Informative)
Modding Non-Reg Links (Score:2)
Instead mod non-reg links and article texts as FUNNY if they are not posted AC, this brings them to the top and adds nothing to the posters karma score.
You got the wrong number... (Score:3, Funny)
-Chief Wiggum
Fear Fear Fear!! (Score:2)
I don't have anything against making an emergency system more reliable or robust, but I just hate when everyone says that "something needs to get done to improve this" when something bad happened. In this case, if politicians knew of this ahead of time, why was something not done about it then??? My guess is that it's just a politically correct statement to make but the politicians just have no clue on
I wonder whose switches they use? (Score:4, Informative)
Re:I wonder whose switches they use? (Score:2)
But, it's still *human* error that led to the problems, not the type of switch.
Not actually a 911 problem (Score:4, Informative)
Shouldn't the interface for the system prevent you from accidentally modifying similar but unrelated numbers when you're modifying a set of numbers?
Of Human Error and Metasystems (Score:4, Insightful)
The idea of having a second person "double-check" is nice in theory, but I will wager that the second person will let errors through too. If the first person is careful, the second person is faced with a long list of matching, correct entries to check. The second person soon becomes fatigued and keeps hitting the "OK" button even when there is a discrepancy. Unless the second person is offered an outsized reward (and the first person is penalized by an even greater amount), its to easy to become apathetic or non-vigilant. (Also, the double-checking process assumes that the original set of command directives was correct).
The real solution is a meta system that logs any changes to the system (like a config change), monitors dependencies of that change, and cross-checks them during exceptions. When an exception occurs, such as a bunch of 911 busy signals, the system would trace through the code and config files and correlate the fact that the onset of busy-911 calls corresponded with the insertion of the erroneous numbers. The system would then either roll-back the changes that caused a fault or alert someone of the list of likely culprits.
Re:Of Human Error and Metasystems (Score:3, Insightful)
Consider:
The changes you have entered will cause 911 calls to be routed to Citibank (or wherever) . Are you sure you want to commit the changes?
Re:Of Human Error and Metasystems (simulation) (Score:2)
Absolutely! A competent network simulation would help predict the impact of a change and catch errors before they are committed. A monitoring system that correlated changes in system behavior against changes in configuration provides added safety in the likely event that the simulation is imperfect or the technician enters values different from that s
This explains the can you hear me now? commercials (Score:2)
...We're sorry... (Score:3, Funny)
"gaaaaackkkk... Get this thing off of me!!!"
"... that number is temporarily out of service."
......grrrrrrrrrrrrrr......
......grrrrrrrrrrrrrr......
"Look you F&*%#$ chihuaha!! I'm sorry I took your chalupa! I'm sorry god damnit!!!"
"Please check the number and try your call again again."
"NOOOOOOOOOOOOOOOOOOOO!!!!!!!!!!"
"If you need assistance please call your operator."
*gurrrgle*
*click*
Better One (Score:2)
You want a real 911 call, try this recording. [98online.com] It is an oldie but a goodie.
Here's a nice "Well, duh" moment. (Score:5, Funny)
Probably because folks were trying to use 911 to report them.
That's Verizon for you, folks! (Score:2, Insightful)
WTF?!?!!!1111 (Score:3, Insightful)
In any case, New York is the suxx0rz because city services don't work.
Re:WTF?!?!!!1111 (Score:3, Insightful)
Media density. All fourty-nine television stations mentioned it, all seven-thousand talkback radio shows had an hour long discussion about it. And don't forget the newspapers, although they managed to give a more digested version of what happened instead of the minute-by-minute update of (mis)information what the television stations did.
Re:WTF?!?!!!1111 (Score:3, Funny)
Hey, people in New York are just as smart as people anywhere else!
Chris Mattern
More Than One "Human Error" (Score:2, Informative)
> didn't anticipate...
Several "human errors", one of them be excessive centralization.
In other news... (Score:2, Funny)
"We figured out that with our new phone systems, completely powered by Microsoft Software these problems will be a thing of the past"
The backup didn't fail... (Score:4, Insightful)
A Verizon tech who was re-routing a customer's numbers accidently made a numerical error that ended up re-routing lines that were meant to go to 911 to a bank. Therefore, the backup system never got a chance to kick in, people were being routed to a very poor selection for a primary destination.
The safety valve that I'm sure is being installed now is requiring a higher degree of password to change the routing instructions for the 911 lines... because this tech should not have been able to mess with them, and didn't mean to, he just typoed the numbers he was supposed to type in. He at least should have seen a "You're trying to reroute 911! Are you sure you want to do that? N" prompt.
Analysis of a Flawed System (Score:5, Insightful)
1. False redundancy: Although the NYC system has a backup central offices and call centers, it apparently routes all calls from the affected area through a single Verizon subsystem. Their system is fully redundant except where its not.
2. Organizational silos in a coupled system: The City claimed that its 911 system was fine because "an error like the one made by Verizon could not necessarily have been prevented because it was not a flaw in the 911 system itself." Yet the Verizon circuits, systems, and procedures are an integral part of the 911 system. The City (and Verizon) maintain a fiction that they are independent entities when, in fact, they are tightly coupled. This division of responsibility is fine for playing the CYA Blame Game, but does not create a robust system.
3. User Interface Flaws I don't know what kind of user interface that technician was using, but it obviously has some terrible flaws if it did not warn him of the implications of the data entries. I also suspect that he was manually retyping some numbers off a computer print-out when he should have had some mechanism to download a set of proofread, verified, double-checked entries.
I don't fault NYC or Verizon in particular, they are probably no worse that anyone else. I only get angry that these types of structural insecurities are probably more widespread than anyone realizes.
Re:Analysis of a Flawed System (Score:3, Interesting)
No, the calls always have multiple paths they can travel. The problem came when all of the routers were given the same mis-information. No number of redundant routers can protect from that.
911 is a Joke (Score:2)
(For those that don't get the subject it was a Public Enemy song, which was later covered by Duran Duran.)
Subtle Terrorism? (Score:3, Insightful)
Eh? Did NYPD and NYFD need the 911 system to find out about the WTC strikes? Terrorism isn't about killing people, it's about getting publicity.
The councilman can rest assured that the terrorists will helpfully keep their activities high-profile enough that 911 notification will not be required.
[Sheesh, why does *everything* have to be about terrorism these days.]
Re:Subtle Terrorism? (Score:3, Insightful)
Well... depends on what kind of terrorism you are talking about. If you are talking about politically motivated terrorism like the IRA, FARC, Hizbollah, then yes, publicity is part, if not all, of the goal. But if you are talking about terrorism like the Al Qaeda variety, notice that they almost never claim responsibility for their attacks, not for the African embassy bombings, not for WTC (ASSUMING, of course, that they are indeed beh
911 was slashdotted. (Score:4, Funny)
Score tied between POTS & VOIP (Score:2, Funny)
The unemployed Ohio power plant operator... (Score:3, Funny)
Get used to it (Score:5, Interesting)
Having worked in private line provisioning and maintenance at Ma Bell I can say without equivocation that this is a direct result of the breakup and not really 'human error'.
I've seen the exact same thing being done at all the locals and the long distance companies.
Manpower is being drawn down, redundancies eliminated, and a talent and brain drain that causes errors like this.
The reason is always given that automation is allowing the company to maximize the remaining workforce and competition makes is neccesary. BS.
Best Practices are -gone- everything is driven by sales and bean counters. Engineers, Technicians and Managers who complain are moved, removed or eliminated from the loop because facts are not going to be allowed to get in the way.
It used to take weeks to get a misdirected line corrected in some instances.
The fault was blamed on too many layers, and union incompetence.
Now with all the improvements brought about by divestiture and competition it is a near imposibility unless it affects a major source of income or government.
This type of error was prevented by human redundancy and a workforce able to put the breaks on before the damage was done because they could stand on the strength of regulations and the union and tell the idiot boss in charge that things were wrong.
Get used to it, rapid reorder will be the order of the day.
Permission to Modify (Score:3, Insightful)
As far as the software is concerned, I'm glad it's getting fixed. Sounds like your typical permissions problem to me. Some guy out in the field shouldn't be able to redirect the phone number for 911. Just like some e-mail attachment I run shouldn't be able to modify HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Curr entVersion\Run.
Seriously, this is something that should be top priority in all kinds of software. Even if the person at the keyboard should be able to modify stuff, doesn't mean they want to. And by default, they should not have the ability to modify certain data.
Let's learn from this and crack down on insecure code. PLEASE!!
911 circus (Score:5, Informative)
Databases at telcos contain what they think is accurate data about how the telephone switches are configured. Telcos may occassionaly audit these two systems to see how well they match up. Then again, they may not do such audits. In modern soft switches, the DB controls the switch, so they do match exactly. But most of the world (including the U.S.) still use legacy telephone switches that are not well integrated with the customer DBs.
Every so often the telcos query their DBs to create 911 update reports. Those reports are passed on to whoever maintains the 911 systems. Then the 911 systems are updated. Maybe.
Overtime, inconsistencies between the telco's system and the 911 system build up. Every so often (once a year or so, maybe) the 911 system gets purged and reloaded from the telco's system. Between reloads, it is not uncommon for a police department to call the 911 system maintainer or the phone company or both (often the 911 maintainer and the telco are completely unrelated entities) to let them know that there were 911 calls last night from such and such phone numbers which had missing or erroneous address info. Steps may or may not be taken to manually correct the info for those individual phone numbers.
You might be surprised to see the percentage of 911 calls that come in with bogus subscriber data.
911 has been a mess for many years and it hasn't been a secret. Eventually, some homeland security committee is going to pass a lot of legislation to address this. The legislation will cost a lot of money, and impose silly requirements. It will likely be drafted by the clowns who have created the bureaucratic 911 system of today. That will only make things worse.
A handful of good IT folks could clean things up very quickly. But that will never happen.
Re:Old News (Score:2, Insightful)
Re:Old News (Score:4, Interesting)
Re:Old News (Score:5, Interesting)
The news was modded down (Score:4, Interesting)
===--===
Re:Old News (Score:4, Funny)
(Are you new here?)
Re:Old News (Score:5, Insightful)
Technically, any news on Slashdot is old news since it has been reported already.
Re:interesting name... (Score:2, Funny)
And someone else was partially pregnant.
Re:interesting name... (Score:5, Interesting)
My wife,daughter and I happened to be wandering around Brooklyn during the outage. We noticed an increase in the number of uniformed officers on the street. This made us slightly nervous in a 9/11 (not a 911) sort of way. It wasn't until we read Saturday's NYT that we realized what was up. To increase visibility during the outage, NYPD had ordered all plainclothes officers to put on uniforms.
As we strolled through some of the city's busiest commercial sectors, everybody as oblivious as we were.
Re:Yeah, (Score:2)
But you could probably fry a neighborhood.
Re:Yeah, (Score:2)
Re:Yeah, (Score:2, Insightful)
Like most things in that book, they either don't work, or will probably cause you to kill yourself if you tried it.
The phone company equipment would obviously have surge detection/supression systems in place to isolate any box that had something like that done to it.
What'd be really funny is if a guy hooked one up and then went home and answered his phone...
Re:Yeah, (Score:4, Interesting)
Re:Yeah, (Score:5, Insightful)
</troll> please.
And, ummm, stop reading Phrack from 1985.
First off, these days, most lines are served off 'digital loop carriers (DLCs),' which take the analog lines from your home, and multiplex it onto high-capacity lines (often running over fiber-optic SONET loops) back to the central office.
Even if your 220 volts made it back to the DLC (which is fairly unlikely, considering 220 VAC at any dangerous ampreage will probably overheat and melt the copper, anyway), the worst you'd do would be to burn out the service area the DLC is handling.
And even if your unlikely scenerio of getting 220 VAC back to a central office, and through the fuses, and the main distribution frame, and even if you hit the switch, you wouldn't affect anything more than that local exchange. Central offices aren't "daisey chained" down copper lines.
(and yes, I do work for a telco)
Re:Real world example (Score:3, Interesting)
Re:Well, how about this, then... (Score:3, Insightful)
Re:we KNOW what it was running (Score:3, Funny)