Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror
Check out the new SourceForge HTML5 internet speed test! No Flash necessary and runs on all devices. ×
Communications

Verizon's NYC 911 System Shutdown 346

Dead Nancy writes "A combination of human error and software that didn't anticipate it brought down New York City's 911 emergency line for several hours on Friday night."
This discussion has been archived. No new comments can be posted.

Verizon's NYC 911 System Shutdown

Comments Filter:
  • Quick! (Score:4, Funny)

    by mindaktiviti ( 630001 ) on Sunday March 28, 2004 @06:18PM (#8698504)
    "Quick! Give me the number for 911!" - Homer J. Simpson
  • So.. (Score:5, Funny)

    by yetdog ( 760930 ) on Sunday March 28, 2004 @06:19PM (#8698506)
    Do you think someone called 911 to report the emergency outage?
    • Yeah, I tried to call 911 to report the outage, but couldn't get through for some reason...
    • Re:So.. (Score:5, Funny)

      by dattaway ( 3088 ) on Sunday March 28, 2004 @06:40PM (#8698678) Homepage Journal
      You must work in tech spport...
    • Re:So.. (Score:5, Funny)

      by awtbfb ( 586638 ) on Sunday March 28, 2004 @07:56PM (#8699171)
      Do you think someone called 911 to report the emergency outage?

      Yeah, it went something like this: "911? Can you hear me now?"
      • by p51d007 ( 656414 ) on Sunday March 28, 2004 @08:39PM (#8699406)
        I worked 911 for 14 years........here in the midwest, we have tornados around this time of year. The warning sirens usually go off to warn the public. Ok, I'm sitting at my console, waiting for people to call about wind damage etc......and the line rings....911, do you have an emergency?....yeah, can you tell me what the sirens are going off for?.........it's a shame that the line is recorded....there have been times, when we get idiot calls like that, that you really really want to say....well, I'm sorry to tell you this, but the russians have changed their minds, the bombs will be here in 10 minutes. And then, hang up on them LOL.... Heck, we've had calls from people that ask us where Bass Pro Shops is.....(Springfield Missouri)
  • 311 (Score:5, Informative)

    by Frying Ferret ( 557022 ) on Sunday March 28, 2004 @06:20PM (#8698523)
    well the calls were routed to 311, so the calls got answered, just maybe not as quickly. Yes this sucks, but the calls didn't go nowhere.
    • Re:311 (Score:2, Interesting)

      by Anonymous Coward
      Am I the only one who has never heard of 311?

      I know about 411, but that's not a non-emergency number.

      Any other *11 numbers we should know about (and what do they do/where are they available?)
      • 511 (Score:5, Informative)

        by Stalemate ( 105992 ) on Sunday March 28, 2004 @06:45PM (#8698718)
        Last I checked 511 will tell you your own phone number. It sounds silly, but it came in handy for me in college when they would often forget to tell us the phone numbers for our dorm rooms.

        That number is handy, but I have to admit that the time I called the pizza place that I knew had caller id and asked them what my phone number was makes a lot better story!

        NOTE: I just tried this on my home for and it worked. My phone company is CenturyTel.
        • Re:511 (Score:3, Informative)

          by The Vulture ( 248871 )
          It differs from area to area.

          Here in California, 511 (so the signs/radio ads say, I haven't tried it) will give you travel information - road conditions, traffic accident reports, bus schedules, carpooling information, etc.

          -- Joe
        • Re:511 (Score:3, Informative)

          by hysma ( 546540 )
          The number 211 use to work in here in BC, Canada with Telus. They changed that a few years ago though, supposedly because phreakers were using it to identify the phone lines they got ahold of.

          Now, you just call the power company at 1 888 POWER-ON and it repeats the number back to you :)

          Some other *11s around here...

          611 = repair
          711 = TTY/TDD operator from payphones
        • In the Portland, OR area it used to be 611, but that changed when they implemented 10-digit dialing. I'm pretty sure that 611 still works in 541.
      • Re:311 (Score:5, Informative)

        by Nurseman ( 161297 ) <nursemanNO@SPAMgmail.com> on Sunday March 28, 2004 @07:04PM (#8698840) Homepage Journal
        Am I the only one who has never heard of 311?

        In NYC 311 is used for Non Emergency calls. Noise, broken lights, etc. It has taken a tremendous load off of the 911 system. People used to call 911 for everything, I think the call volume is down 30- 40% since 311 went into effect.

        • Re:311 (Score:3, Funny)

          by Anonymous Coward
          I think the call volume is down 30- 40% since 311 went into effect.

          Sounds to me from this story that they've got call volume down 100% now. Good job!

    • Re:311 (Score:5, Informative)

      by IO ERROR ( 128968 ) <error@ioe[ ]r.us ['rro' in gap]> on Sunday March 28, 2004 @06:44PM (#8698713) Homepage Journal
      Only 80 calls went to 311 during the two-and-a-half hour outage. Countless (possibly hundreds) others just got a fast busy signal, recording or continuous ringing with no answer. My guess is the people who got 311 actually CALLED 311 because they couldn't get through to 911.
    • Re:311 (Score:3, Interesting)

      Thanks for bringing up 311. I'm amazed at how few people even know about 311. It seems the public's adoption of 911 has to be 90%+, yet I bet the adoption of 311 has to be 10% or less.

      I recently sat on hold with 911 for literally 20 minutes after I watched a vehicular hit-and-run accident from my car, where the :cough: gentleman fleeing the scene looked around for a good 30 seconds to see if anyone noticed before making a break for it. (Read: he sped away.)

      Luckily I was close enough to get the a-hole'
  • by Anonymous Coward on Sunday March 28, 2004 @06:20PM (#8698524)
    I think this is a shining demonstration of why monopolies shouldn't be allowed to run the phone system. Speaking of monopolies, I wonder what the 'software' concerned was?
    • by Anonymous Coward
      So instead when you call 911 you have a random chance of the software working? Yes, lets put small understaffed local companies in charge of the emergency phone system.
    • by the eric conspiracy ( 20178 ) on Sunday March 28, 2004 @06:40PM (#8698681)
      When the phone companies were truly a regulated monopoly (AT&T) you got *real* quality of service, plus an R&D organization that invented the laser, transistor and was awarded:
      • 6 Nobel Prizes in Physics shared by 11 scientists
      • 9 U.S. Medals of Science
      • 7 U.S. Medals of Technology
      • 1 Draper Prize
      • 6 Marconi International Fellowship Awards
      • 7 C&C Prizes shared by 12 scientists and engineers
      • 27 IEEE Medal of Honor winners
      Now I don't think monopolies are a good idea in the general case, however AT&T's results were at least halfway decent.
      • by T-Ranger ( 10520 ) <jeffwNO@SPAMchebucto.ns.ca> on Sunday March 28, 2004 @07:55PM (#8699162) Homepage
        Man, this is slashdot. Point out some usefull things:

        • UNIX
        • C
        • C++ (well, ok, thats not usefull)
        • awk
        • Plan 9
      • by dachshund ( 300733 ) on Sunday March 28, 2004 @08:05PM (#8699222)
        Now I don't think monopolies are a good idea in the general case, however AT&T's results were at least halfway decent.

        Yet at the same time, daytime Long Distance cost over a dollar a minute from NY to California. Phones and telephone equipment had to be rented from the phone company, so technological development in many areas (faxes, answering systems, business telephony) came at a snail's pace. Also, if you think DSL rollout was slow and overpriced under the Baby Bells, just imagine what it would have been like under Ma Bell. All things have their price, and this was a high one.

        Furthermore, AT&T had their problems as well. On at least one occasion, they had massive network failures due to a combination of-- guess what-- human error and software failure.

      • by ktakki ( 64573 ) on Sunday March 28, 2004 @11:39PM (#8700442) Homepage Journal
        When the phone companies were truly a regulated monopoly (AT&T) you got *real* quality of service...

        I remember the "good ol' days" of Ma Bell's monopoly, before the big breakup. I'd take exception to the use of the phrase "*real* quality of service".

        The only good thing I recall about those days was that most Western Electric phones were virtually bulletproof, the telco equivalent of an IBM Type M keyboard. But that's about it.

        For starters, since the telco owned everything on the network, adding an extension phone was a violation of their terms of service, and they'd come down hard on you if they found one in use (and don't think they didn't check up on people; they did). The sound quality was vastly inferior to what we now have: long distance sounded like long distance, but even local calls could sometimes be rendered unintelligible by the monopoly's antiquated switching system. Service in rural areas pretty much sucked hind tit; even in the late '70s it was party lines or nothing in certain towns in upstate New York.

        Even worse, Ma Bell's responsiveness to consumer complaints was a national joke. Remember Lily Tomlin's character Ernestine? One ringy-dingy... That was a caricature, of course, but one grounded in truth.

        After divestiture, things really changed for the better overall. The relaxation of restrictions on what could be placed on the network meant a boom in devices like answering machines, fax machines, and modems. Had the old pre-1984 restrictions been in place, what do you think the effect would have been on BBSs and dial-up access to the Internet? Imagine having to pay extra in order to have a modem connected to your phone line. Sound quality improved largely due to technological advances, but had the monopoly still been in place, would there have been any incentive to upgrade the telco network?

        Yes, AT&T had been on the cutting edge of computer science and electronic engineering for decades. But had the break-up not taken place, we'd still be using a phone system worthy of the movie Brazil.

        k.
      • by Sanat ( 702 )
        God forbid that you ever have a problem with a data line (higher quality line in those times) for the tech at AT&T would simply say that it was your equipment causing the problem and that their line check verified it. It was a monumental lesson in frustration.

        I ran a 10 state wide call center and the data processing branch of our company and dealing with AT&T was not easy even though we had nearly 100 lines with them. The customer service representives though at AT&T were outstanding, however t
    • The problem (Score:5, Informative)

      by KalvinB ( 205500 ) on Sunday March 28, 2004 @06:42PM (#8698689) Homepage
      is that there is a very finite number of customers for things like electricity and phone service. You need a very large customer base in order to be able to charge a reasonable amount.

      When you allow competition those that attempt to compete are forced to either charge less than it costs to supply the service or charge more. If they charge less but can't get the customer base they go out of business. If they charge more people tend not to switch. And if you don't charge enough less, nobody cares enough to switch.

      Cox saves us all of a buck or two over Qwest on phone service. We never bothered to switch until we switched to Cox for high speed internet.

      And of course the only reason Cox had the money to implement phone service is because they're the monopoly on cable service.

      In cases like this it's actually better for the government to force the monopoly to act in the best interest of the people than to allow competition which just gives people the false impression that it'll lead to cheaper prices and better service.

      Competition in these cases are almost always forced to either cut corners to survive or charge more.

      Ben
    • We should outsource 911.
    • Thanks to the government mandated breakup and unfair reprisals against AT&T, the current phone system in the USA is **NOT** a monopoly.

      This is yet another example of why the breakup was a bad thing.
    • by RealityMogul ( 663835 ) on Sunday March 28, 2004 @08:09PM (#8699247)
      My Verizon phone service went out twice in the last month. Line was completely dead. I work 8-5 M-F. I called them when I got home which was around 6:00PM, I went to a payphone and informed them of the problem. Their response was, we'll send a technician out tomorrow. I asked what I was suppose to do if there was a medical emergency and I needed to dial 911. The person on the other end just went completely silent until I had to ask if she was still there. Obviously its too much to get a tech to come out after 4pm unless there's a major outage.

      Nice guys. I wish I had a heart attack just so I could file a lawsuit. That's the only way things change nowadays.
    • I don't agree. An eventuality occurred which the original software designers hadn't thought of. This happens all the time in embedded systems - the world is not only more complicated than we think, it is more complicated than we can think.

      This failure required two failures in a row - one human, one software. After this failure, both systems will be fixed, so this failure won't happen again on this system. Furthermore, it won't happen again on any system those involved in cleaning up work on - we learn by o
  • NYTimes (Score:3, Informative)

    by LordK3nn3th ( 715352 ) on Sunday March 28, 2004 @06:20PM (#8698525)
    For those who don't want to register:

    username: slashdot2003
    password: slashdot2003
  • by DroopyStonx ( 683090 ) on Sunday March 28, 2004 @06:20PM (#8698526)
    You read stores like this and think, "Oh, it was just error."

    But you fail to realize that a big heist probably took place. I saw George Clooney and Matt Damon do it to Las Vegas. They let off this big ass EMP and shut down the power to the whole city! Long story short, they stole a shitload of money and got away with it.

    Don't let these stories fool you, that is exactly what happened here.
    • DDOS 911 (Score:5, Interesting)

      by temojen ( 678985 ) on Sunday March 28, 2004 @06:56PM (#8698795) Journal

      Here a few years ago there was a sting of robberies where the thieves called a whole lot of people and convinced them to "test" the emergency response system at a specified time a few days later.

      All of a sudden there were hundreds of simultaneous calls reporting accidents, fires, muggings, heart attacks, rapes, robberies, etc. The thieves robbed two banks and a big-box store while the police were tied up.

    • by Dirtside ( 91468 ) on Sunday March 28, 2004 @08:37PM (#8699398) Journal
      You read stores like this and think, "Oh, it was just error."

      But you fail to realize that a big heist probably took place. I saw George Clooney and Matt Damon do it to Las Vegas. They let off this big ass EMP and shut down the power to the whole city! Long story short, they stole a shitload of money and got away with it.

      Don't let these stories fool you, that is exactly what happened here.

      So I guess the people behind it are errorists, then?
  • Feh. (Score:5, Insightful)

    by James A. M. Joyce ( 764379 ) on Sunday March 28, 2004 @06:21PM (#8698535) Journal
    And to think everyone was worrying about the terrorists, fer chrissakes.
    • Re:Feh. (Score:3, Interesting)

      by dackroyd ( 468778 )
      Wow. You're right - shurely the Department of Homeland security was meant to have made sure that vital infrastructure in the US is robust in the face of terrorist action, not just people being dumb.

      a Verizon spokesman, said the telephone company would now require a second person to double-check any entry of data that could affect the 911 system

      So, they've just announced to all the terrorists in the world - this problem still exists, is going to remain there and if any terrorist organisation can get one

  • Was it a "what does this little red button do?", or a misconfiguration somewhere by a sleep-deprived sysop at 3am?
  • by oldosadmin ( 759103 ) on Sunday March 28, 2004 @06:27PM (#8698581) Homepage
    Verizon began taking steps yesterday to better protect New York City's 911 emergency line after a data error by an employee brought down the system in Brooklyn, Queens and Staten Island for about two hours on Friday night, city and Verizon officials said.

    The emergency system broke down about 7:20 p.m. after a Verizon engineer who was making service changes to a bank's telephone numbers in Brooklyn inadvertently included numbers that are used to carry 911 calls, city and telephone company officials said. The numbers were close in sequence, the officials said.

    The 911 calls then ended up being rerouted to the bank's phone system, and callers heard a busy signal. City and Verizon officials said that while the backup system in place for 911 was functioning properly, it failed to pick up the calls because it was designed to catch a technical error, not a human error that would be interpreted as simply a change of instruction.

    Daniel Diaz Zapata, a Verizon spokesman, said the telephone company would now require a second person to double-check any entry of data that could affect the 911 system, and said the company planned a thorough review of its procedures that would be documented in a report to the city within a few days.

    "We determined that a human error resulted in the accidental rerouting of phone calls during a procedure to upgrade service for a corporate client," Mr. Zapata said. "We have immediately altered our processes to ensure this type of situation does not reoccur. We have assured the city that we took immediate steps to make sure this doesn't happen again."

    Citing privacy concerns, Mr. Zapata declined to identify the Verizon engineer, except to say that he was a veteran of the company. Mr. Zapata said it was unlikely that disciplinary action would be taken against him.

    Police and fire officials said yesterday that they had no reports of injuries during the 911 failure. Fire officials said that about 60 firefighters responded to a major fire, at 3301 Foster Avenue in Brooklyn, which was called in at 8:49 p.m. by someone using a fire alarm box on the street. There were no injuries in the fire.

    Paul J. Browne, the Police Department's deputy commissioner for public information, said the department immediately adopted emergency procedures, like requiring e officers on patrol to turn on their flashing lights so people could find them easily and increasing staffing at precinct station houses to answer phone calls. But he said there was no reported increase in crime.

    "This didn't present an opportunity for the criminally minded - like the blackout did - because probably most people were unaware that it was out of service," he said.

    However, several City Council members expressed anger that the 911 system could have been so easily disabled, and called for creating a more effective backup procedure.

    "It's an emergency wakeup call," said Councilman Peter F. Vallone Jr., the chairman of the Public Safety Committee, who plans to hold a hearing about the incident. "We don't have an adequate backup system for 911, which is more important than ever as we fight the war against terrorism."

    Gino P. Menchini, the commissioner of the Department of Information Technology and Telecommunications, said city officials were working with Verizon to ensure that the emergency system's numbers were clearly identified, and that its software and equipment were protected from similar human errors.

    But Mr. Menchini emphasized that the emergency system already had many built-in safeguards, such as the ability to route 911 calls through either of two central offices and their 911 answering centers. "The bottom line is, 911 works very well, and it's worked very well for a long time," Mr. Menchini said.

    Several emergency services experts agreed yesterday with Mr. Menchini, saying that New York 911 system compared favorably with those in other large cities and that an error like the one made by Verizon could not necessaril
  • Hey, I can't find the eleven....
  • by dan_polt ( 692266 ) on Sunday March 28, 2004 @06:29PM (#8698599)
    Non-reg link: here [nytimes.com]
    • Please, this is karma whoring (perhaps unintentially), don't mod a non-reg link comment Informative.

      Instead mod non-reg links and article texts as FUNNY if they are not posted AC, this brings them to the top and adds nothing to the posters karma score.
  • by Anonymous Coward on Sunday March 28, 2004 @06:30PM (#8698606)
    "You got the wrong number, this is nine one...two"
    -Chief Wiggum
  • "Meanwhile, politicians are speaking out saying improvements need to be made to the emergency system."

    I don't have anything against making an emergency system more reliable or robust, but I just hate when everyone says that "something needs to get done to improve this" when something bad happened. In this case, if politicians knew of this ahead of time, why was something not done about it then??? My guess is that it's just a politically correct statement to make but the politicians just have no clue on

  • by luckytroll ( 68214 ) on Sunday March 28, 2004 @06:34PM (#8698638) Homepage
    I once built an HA cluster that had a role in the 911 system, at a major telco switch vendor. Apparently the fines for a vendor bungle in this are in the millions of dollars per minute of downtime. I dont know which switches these are, but such penalties could affect that vendor in fines and their bottom line. It looks like Verizon shot itself in the foot, but keep an eye out for a dip in share prices for some of the switch vendors in case the blame gets spread around.
  • by iabervon ( 1971 ) on Sunday March 28, 2004 @06:38PM (#8698660) Homepage Journal
    The 911 system seems to have continued to work correctly, but the regular numbers that calls to 911 get directed to were redirected to a bank. So the issue is really that there aren't safeguards against the wrong phone numbers getting changed accidentally, and the phone lines used by the 911 system aren't immune.

    Shouldn't the interface for the system prevent you from accidentally modifying similar but unrelated numbers when you're modifying a set of numbers?
  • by G4from128k ( 686170 ) on Sunday March 28, 2004 @06:38PM (#8698666)
    This type of error is a classic problem of the computer assuming that the human was right. We create machines to give us power, but use that power to cavalierly.

    The idea of having a second person "double-check" is nice in theory, but I will wager that the second person will let errors through too. If the first person is careful, the second person is faced with a long list of matching, correct entries to check. The second person soon becomes fatigued and keeps hitting the "OK" button even when there is a discrepancy. Unless the second person is offered an outsized reward (and the first person is penalized by an even greater amount), its to easy to become apathetic or non-vigilant. (Also, the double-checking process assumes that the original set of command directives was correct).

    The real solution is a meta system that logs any changes to the system (like a config change), monitors dependencies of that change, and cross-checks them during exceptions. When an exception occurs, such as a bunch of 911 busy signals, the system would trace through the code and config files and correlate the fact that the onset of busy-911 calls corresponded with the insertion of the erroneous numbers. The system would then either roll-back the changes that caused a fault or alert someone of the list of likely culprits.
    • I was thinking more of a system that would catch this sort of error in advance, rather than after the damage had already been done by a data-entry error.

      Consider:

      The changes you have entered will cause 911 calls to be routed to Citibank (or wherever) . Are you sure you want to commit the changes?

      • I was thinking more of a system that would catch this sort of error in advance, rather than after the damage had already been done by a data-entry error.

        Absolutely! A competent network simulation would help predict the impact of a change and catch errors before they are committed. A monitoring system that correlated changes in system behavior against changes in configuration provides added safety in the likely event that the simulation is imperfect or the technician enters values different from that s
  • I always thought it was just an annoying phrase we were having drilled into our heads, but now we now the guy really has problems with Verizon service.
  • by l0ungeb0y ( 442022 ) on Sunday March 28, 2004 @06:44PM (#8698710) Homepage Journal

    "gaaaaackkkk... Get this thing off of me!!!"


    "... that number is temporarily out of service."

    ......grrrrrrrrrrrrrr......

    "Look you F&*%#$ chihuaha!! I'm sorry I took your chalupa! I'm sorry god damnit!!!"
    ......grrrrrrrrrrrrrr......


    "Please check the number and try your call again again."

    "NOOOOOOOOOOOOOOOOOOOO!!!!!!!!!!"


    "If you need assistance please call your operator."


    *gurrrgle*

    *click*

  • Police and fire officials said yesterday that they had no reports of injuries during the 911 failure.

    Probably because folks were trying to use 911 to report them.

  • by Anonymous Coward
    If you ask anyone from IT who has to deal with Verizon, they'll all tell you that Verizon and incompetence go together.

  • WTF?!?!!!1111 (Score:3, Insightful)

    by rice_burners_suck ( 243660 ) on Sunday March 28, 2004 @06:52PM (#8698761)
    Strange. First, human and computer error caused the entire power grid in that area of the world to go down for quite a few hours. Now, the 911 system had a similar (although much smaller, I suppose) event. Why are so many things of this nature happening in New York? Does it have anything to do with the city's density?

    In any case, New York is the suxx0rz because city services don't work.

    • Re:WTF?!?!!!1111 (Score:3, Insightful)

      by MavEtJu ( 241979 )
      Does it have anything to do with the city's density?

      Media density. All fourty-nine television stations mentioned it, all seven-thousand talkback radio shows had an hour long discussion about it. And don't forget the newspapers, although they managed to give a more digested version of what happened instead of the minute-by-minute update of (mis)information what the television stations did.
    • > Does it have anything to do with the city's density?

      Hey, people in New York are just as smart as people anywhere else!

      Chris Mattern
  • > A combination of human error and software that
    > didn't anticipate...

    Several "human errors", one of them be excessive centralization.
  • ... DIebold anounced their plans to replace Verizon at handling the 911 system...

    "We figured out that with our new phone systems, completely powered by Microsoft Software these problems will be a thing of the past"

  • by LostCluster ( 625375 ) * on Sunday March 28, 2004 @07:04PM (#8698841)
    The political types saying that they don't have a good enough backup 911 system failed to understand the root cause of this failure.

    A Verizon tech who was re-routing a customer's numbers accidently made a numerical error that ended up re-routing lines that were meant to go to 911 to a bank. Therefore, the backup system never got a chance to kick in, people were being routed to a very poor selection for a primary destination.

    The safety valve that I'm sure is being installed now is requiring a higher degree of password to change the routing instructions for the 911 lines... because this tech should not have been able to mess with them, and didn't mean to, he just typoed the numbers he was supposed to type in. He at least should have seen a "You're trying to reroute 911! Are you sure you want to do that? N" prompt.
  • by G4from128k ( 686170 ) on Sunday March 28, 2004 @07:08PM (#8698866)
    As I read the article, it is obvious that NYC's system is fraught with deep flaws in its design and management. These include:

    1. False redundancy: Although the NYC system has a backup central offices and call centers, it apparently routes all calls from the affected area through a single Verizon subsystem. Their system is fully redundant except where its not.

    2. Organizational silos in a coupled system: The City claimed that its 911 system was fine because "an error like the one made by Verizon could not necessarily have been prevented because it was not a flaw in the 911 system itself." Yet the Verizon circuits, systems, and procedures are an integral part of the 911 system. The City (and Verizon) maintain a fiction that they are independent entities when, in fact, they are tightly coupled. This division of responsibility is fine for playing the CYA Blame Game, but does not create a robust system.

    3. User Interface Flaws I don't know what kind of user interface that technician was using, but it obviously has some terrible flaws if it did not warn him of the implications of the data entries. I also suspect that he was manually retyping some numbers off a computer print-out when he should have had some mechanism to download a set of proofread, verified, double-checked entries.

    I don't fault NYC or Verizon in particular, they are probably no worse that anyone else. I only get angry that these types of structural insecurities are probably more widespread than anyone realizes.
    • 1. False redundancy: Although the NYC system has a backup central offices and call centers, it apparently routes all calls from the affected area through a single Verizon subsystem. Their system is fully redundant except where its not.

      No, the calls always have multiple paths they can travel. The problem came when all of the routers were given the same mis-information. No number of redundant routers can protect from that.
  • Verizon says calls were rerouted during a procedure to upgrade service for a corporate client.
    "Now to know that the 911 system is that vulnerable to these routine types of disruptions is a very serious concern," said Councilman Vincent Gentile.
    Don't worry councilman Gentile, they've got an upgrade planned that will fix it. D'Oh!

    (For those that don't get the subject it was a Public Enemy song, which was later covered by Duran Duran.)
  • Subtle Terrorism? (Score:3, Insightful)

    by FTL ( 112112 ) * <slashdot&neil,fraser,name> on Sunday March 28, 2004 @07:13PM (#8698899) Homepage
    "We don't have an adequate backup system for 911, which is more important than ever as we fight the war against terrorism." -- Councilman Peter F. Vallone Jr.

    Eh? Did NYPD and NYFD need the 911 system to find out about the WTC strikes? Terrorism isn't about killing people, it's about getting publicity.

    The councilman can rest assured that the terrorists will helpfully keep their activities high-profile enough that 911 notification will not be required.

    [Sheesh, why does *everything* have to be about terrorism these days.]

    • by Mao ( 12237 )
      Terrorism isn't bbout killing people, it's about getting publicity.

      Well... depends on what kind of terrorism you are talking about. If you are talking about politically motivated terrorism like the IRA, FARC, Hizbollah, then yes, publicity is part, if not all, of the goal. But if you are talking about terrorism like the Al Qaeda variety, notice that they almost never claim responsibility for their attacks, not for the African embassy bombings, not for WTC (ASSUMING, of course, that they are indeed beh

  • by CowardNeal ( 627678 ) on Sunday March 28, 2004 @07:18PM (#8698938)
    Blame you guys.
  • So reliable 911 service *isn't* an advantage that POTS has over VOIP?
  • by dark-br ( 473115 ) on Sunday March 28, 2004 @07:32PM (#8699010) Homepage
    ...just found a new job. Well, he's unemployed again anyways.

  • Get used to it (Score:5, Interesting)

    by oldstrat ( 87076 ) on Sunday March 28, 2004 @08:13PM (#8699276) Journal


    Having worked in private line provisioning and maintenance at Ma Bell I can say without equivocation that this is a direct result of the breakup and not really 'human error'.
    I've seen the exact same thing being done at all the locals and the long distance companies.
    Manpower is being drawn down, redundancies eliminated, and a talent and brain drain that causes errors like this.
    The reason is always given that automation is allowing the company to maximize the remaining workforce and competition makes is neccesary. BS.

    Best Practices are -gone- everything is driven by sales and bean counters. Engineers, Technicians and Managers who complain are moved, removed or eliminated from the loop because facts are not going to be allowed to get in the way.

    It used to take weeks to get a misdirected line corrected in some instances.
    The fault was blamed on too many layers, and union incompetence.
    Now with all the improvements brought about by divestiture and competition it is a near imposibility unless it affects a major source of income or government.

    This type of error was prevented by human redundancy and a workforce able to put the breaks on before the damage was done because they could stand on the strength of regulations and the union and tell the idiot boss in charge that things were wrong.

    Get used to it, rapid reorder will be the order of the day.

  • by thedillybar ( 677116 ) on Sunday March 28, 2004 @08:58PM (#8699513)
    Congratulations to the people running the show over there. I'm really impressed they implemented plan B so quickly and got the system back up and running so fast. No one likes a page on Friday night...

    As far as the software is concerned, I'm glad it's getting fixed. Sounds like your typical permissions problem to me. Some guy out in the field shouldn't be able to redirect the phone number for 911. Just like some e-mail attachment I run shouldn't be able to modify HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Curr entVersion\Run.

    Seriously, this is something that should be top priority in all kinds of software. Even if the person at the keyboard should be able to modify stuff, doesn't mean they want to. And by default, they should not have the ability to modify certain data.

    Let's learn from this and crack down on insecure code. PLEASE!!

  • 911 circus (Score:5, Informative)

    by Anonymous Coward on Monday March 29, 2004 @02:33AM (#8701093)
    People seem to think that the 911 system is directly connected to the systems that run the telephone network. This really isn't so.

    Databases at telcos contain what they think is accurate data about how the telephone switches are configured. Telcos may occassionaly audit these two systems to see how well they match up. Then again, they may not do such audits. In modern soft switches, the DB controls the switch, so they do match exactly. But most of the world (including the U.S.) still use legacy telephone switches that are not well integrated with the customer DBs.

    Every so often the telcos query their DBs to create 911 update reports. Those reports are passed on to whoever maintains the 911 systems. Then the 911 systems are updated. Maybe.

    Overtime, inconsistencies between the telco's system and the 911 system build up. Every so often (once a year or so, maybe) the 911 system gets purged and reloaded from the telco's system. Between reloads, it is not uncommon for a police department to call the 911 system maintainer or the phone company or both (often the 911 maintainer and the telco are completely unrelated entities) to let them know that there were 911 calls last night from such and such phone numbers which had missing or erroneous address info. Steps may or may not be taken to manually correct the info for those individual phone numbers.

    You might be surprised to see the percentage of 911 calls that come in with bogus subscriber data.

    911 has been a mess for many years and it hasn't been a secret. Eventually, some homeland security committee is going to pass a lot of legislation to address this. The legislation will cost a lot of money, and impose silly requirements. It will likely be drafted by the clowns who have created the bureaucratic 911 system of today. That will only make things worse.

    A handful of good IT folks could clean things up very quickly. But that will never happen.

Ever notice that even the busiest people are never too busy to tell you just how busy they are?

Working...