Forgot your password?
typodupeerror
The Internet

Vulnerability of Telco Switching Equipment 199

Posted by Hemos
from the weak-systems dept.
call -151 writes: "Interesting New York times article about the Sept 11th attacks' effect on the Verizon switches in lower Manhattan. Turns out there was a problem in that much of the network switching was in one building and it has taken a while to restore service. Sounds like there is lots of pondering about the vulnerability of the network, even when it is distributed across many physical locations. Of course the attacks are making lots of people rethink their vulnerabilities, but the estimate is for five years' work before there could be redundant paths for the lines into their switches in the one building, with no plans to spend the money to do it. Maybe someone should send them a few hundred thousand 'self-install' kits like they do with their DSL service ..."
This discussion has been archived. No new comments can be posted.

Vulnerability of Telco Switching Equipment

Comments Filter:
  • Oh darn, now we'll NEVER get fiber to the last mile. ;-)
  • geographic density (Score:4, Interesting)

    by shibut (208631) on Monday October 15, 2001 @05:24PM (#2433240)
    the problem is when you have a small metro area that is very dense and a high concumer of telco services. Even if you had redundant services, it setill makes economic sense (from the service point of view) to locate both (say) switches in the same area therefore, it would only somewhat help with an attack such as this.
    • I would be less bothered by this if VZ wasn't so insanely monopolistic.
    • by RobNich (85522)
      In larger areas, for instance Cincinnati, the Bell has a number of switches (I think it's 30). Each are connected with Sonet rings. I have toured the Cincinnati Bell NOC, where they have electronic maps of the rings and the network. Each CO may be on a number of rings. There are something like 300 rings in the area, all interconnecting different sets of COs, with plenty of overlap on each CO.

      For those of you not familiar with Sonet, it is a ring of nodes, with a fiber pair running in each direction (four fibers instead of the normal two). If a cut happens, traffic is instantly routed in the opposite direction, around the break.

      Cincinnati Bell uses their Sonet network for all voice, ATM, etc. LD carriers can connect to the network at any point (or multiple points).

      • by mclaugh (130321) on Tuesday October 16, 2001 @07:36AM (#2435414) Homepage
        Your argument would be valid if it even came close to the size and breadth of the VZ network at 140 West Street. Besides the facilities that existed in the Towers themselves, West Street was our sole serving facility for most of downtown. You can't compare 30 C.O.s in Cincy to one at West Street just because the NY Times article compared 140 West to Cincinnati in terms of data travelling through it. You can't make a direct comparison from an analogy like that. It's not as if West Street is the only C.O. in Manhattan- it's just the one that was damaged when a major catastrophe occurred next door.

        The biggest problem at West Street is not necessarily the damage, or the flooding in our basements- it's the fact that West Street does not have reliable records. Their switching facilities and relay racks were not mapped reliably, and as a result, they are expending just as much manpower to figure out what was going where as to run new cables out from the 4th floor.

        As for why there is so much damage to begin with, we were told that either a huge internal beam or the antenna on top the north tower pierced 140 West Street, causing most of the damage. The side of the building that faced the Towers was the side that most of the switching equipment was kept on. Combine that with the fact that the basement was flooded with water and diesel fuel, and you have a building that can't really support any kind of telephone service, at least immediately after the attacks.

        Besides the huge amount of data lines that are served out of West Street, most other low speed data lines were routed through there. So, if the little bodega near your apartment in the Bronx or Brooklyn stopped selling Lotto Tickets right after the attacks, that's why.

        Hhope ths helps.
        • I hate to contribute a 'mod this up!' post, but somebody please do so.

          I've been with telcos that struggled to merely operate daily under the weight of their own poor record-keeping. I can't imagine trying to rebuild a facility like this without state-of-the-art, zealously maintained records. Electronic backups of all node configurations and datafill would be a good start, but the miles and miles of old cabling running through that building just gives me nightmares...
  • Could someone post the NYTimes user and password?

    For some reason, even though the link is to partners.nytimes.com it still prompts.
  • priorities? (Score:2, Insightful)

    by InfiX (160201)
    Sure, I suppose Telco redundancy for protection would be helpful to safeguard against risks like this...but (a) who can forsee such an event? and (b) is protecting the Telco systems any kind of priority in relation to the neccessary defense of life and peace of mind?

    The main item to be gleaned from this I think is simply that there is widespread and not readily obvious impact in many sectors from this catastrophe. But reworking national infrastructures out of paranoia may be overdoing it...
  • by damiam (409504)
    For those who don't want to register with the NY Times, here's the article:

    October 15, 2001

    Attacks Expose Telephone's Soft Underbelly

    By SIMON ROMERO

    Joseph Pennell, the prolific illustrator who often depicted the cityscape of Lower Manhattan in his prints, called the New York Telephone Building "the most impressive modern building in the world" when it was completed in 1926.

    How antiquated it now seems.

    The 32-story structure at 140 West Street, one of the city's first Art Deco skyscrapers, is now owned by New York Telephone's descendant, Verizon Communications (news/quote ). And the heavy damage the building sustained on Sept. 11 underscores the vulnerability of communications networks operated by Verizon and other telephone companies ? sprawling systems that rely heavily on critical hubs.

    In the days after the Sept. 11 attacks, it became commonplace to comment on how well the Internet performed because it was designed to route traffic around damage. But the telephone network, including the dedicated data lines that are used by big corporations, financial institutions and others, does not have the Internet's self-detouring abilities.

    When they work, the telephone network's voice and data lines can be superior in quality and carrying capacity to the Internet. Yet when the telephone network is damaged, it cannot heal itself.

    And while Verizon has worked almost around the clock the last month to restore operations at 140 West Street and service to its customers, the company has indicated that significantly reducing the building's network vulnerabilities would require more time or money than Verizon is willing to expend.

    Verizon's building was near the north tower of the World Trade Center and next door to 7 World Trade Center, which collapsed several hours after the attacks. Falling rubble and steel girders tore into 140 West Street, which housed one of the nation's busiest telephone central office switching stations. When fully operable, it serves a customer base comparable in number with all the telephone lines in a city the size of Cincinnati.

    After electric power for the building was interrupted, service was temporarily disrupted for more than 300,000 telephone lines and 3.6 million high-capacity data circuits, many serving the New York Stock Exchange, large financial institutions and other companies in lower Manhattan. A gaping hole was torn in a seventh-floor exterior wall, exposing and damaging huge communications switches dedicated to the information needs of the banking company J. P. Morgan Chase.

    In the last month, Verizon has labored to restore service or provide new service for customers that have moved to other parts of the city or to New Jersey. Virtually all of the fiber optic lines and copper strands that had wound their way under the streets and sidewalks and into 140 West Street are being replaced. Some circuits have been rerouted to other Verizon central offices in Lower Manhattan.

    "The ideas we previously had about diversifying our networks have become much more important," Lawrence T. Babbio Jr., Verizon's vice chairman, said in an interview last week as he led a small group of journalists on a tour of 140 West Street.

    Until last month, the most obvious reasons for network disruptions were natural disasters like hurricanes or floods. Now, though, Verizon and other telephone companies must worry about the possibility of physical attacks on their installations. Mr. Babbio warned last week that significant harm could be done to the nation's communications system if terrorists destroyed the 50 or 100 most important central offices.

    Verizon, which is the dominant telephone company on the Eastern seaboard and operates in 30 states overall, is seeking to increase security at its central offices, where it is required by federal law to lease network access to its competitors. After Mr. Babbio issued his warning last week, competitors said they would resist tighter security measures if it made it more difficult for them to conduct operations within Verizon's central offices.

    Beyond physically shielding their switching centers, phone companies can protect their communications networks from direct attacks or peripheral damage from nearby attacks by routing voice and data traffic to other parts of their own networks or those of other companies.

    But Mr. Babbio said that it would take Verizon five years to build alternate pathways for all the telephone lines that wind their way into and out of the New York Telephone building. And Verizon has no plans to do so.

    The reason may be a simple cost- benefit analysis. Despite its primacy to Lower Manhattan's communications network, the central office at 140 West Street accounted for less than 1 percent of the traffic on Verizon's nationwide network.

    "So much of the activity on networks takes place at dispersed locations," said Roy A. Maxion, a system scientist at Carnegie Mellon University. "But the fact remains that we're vulnerable even after putting redundancy systems in place due to the physical nature of connecting to our networks. The issue should be what level of risk you're willing to live with."

    Assuming they are willing to spend the money, business customers can achieve redundancy, or surplus and backup capacity, by running cables to several different central offices or, in some cases, by using several different communications carriers. Several of Verizon's competitors, in fact, have benefited from the disruptions by signing up new customers in Lower Manhattan.

    "Identifying potential failures in networks is not easy," said Joe Flach, vice president of the Eagle Rock Alliance, a consulting company that provides advice on disaster planning. "The most important thing to avoid is putting all of your eggs in one basket."

    Only after Sept. 11 did executives from the financial services industry in Lower Manhattan come to realize just how many of its eggs were in that one 75-year-old building.

    Mr. Babbio recalled having to explain the situation at a meeting in Midtown Manhattan on Wednesday, Sept. 12, at the Park Avenue offices of the investment bank Bear, Stearns. Executives and government officials present included Richard A. Grasso, chairman of the New York Stock Exchange; Harvey L. Pitt, chairman of the Securities and Exchange Commission; Richard S. Fuld, chief executive of Lehman Brothers (news/quote); John A. Thain, a president of Goldman Sachs (news/quote); and Peter R. Fisher, under secretary for domestic finance at the Treasury Department.

    The group was not happy when Mr. Babbio said how long it might take to restore basic service. Mr. Grasso had been hoping to reopen the stock exchange on Thursday or Friday. The following Monday now seemed ambitious.

    "It was not an easy meeting," recalled Mr. Babbio, who spoke with the group immediately after visiting the disaster site, where his clothes had picked up the odor of smoke and ash. "I smelled awful after coming back from downtown. No one wanted to sit next to me."
  • Wireless? (Score:3, Interesting)

    by elroyjenkins (221758) <elroyjenkins@datapimp. o r g> on Monday October 15, 2001 @05:30PM (#2433272)
    Would this problem be easier to solve with a large wireless network? Considering the coverage of antennas these days, we could have some major overlappage for a fraction of the comparable cost.


    • Re:Wireless? (Score:3, Informative)

      by BrookHarty (9119)
      All the wireless "basestations" have frame relay connections into one building. Same point of failure.
      • Re:Wireless? (Score:2, Interesting)

        by cyberformer (257332)
        Not the Ricochet network, switched off in August but still in place. It proved so resilient that the City of New York temporarily reactivated it, for use by search and salvage workers.
  • by DavidJA (323792) on Monday October 15, 2001 @05:34PM (#2433300)

    ... Just because I have havinging to rego for the NYTimes site.

    Attacks Expose Telephone's Soft Underbelly

    By SIMON ROMERO

    oseph Pennell, the prolific illustrator who often depicted the cityscape of Lower Manhattan in his prints, called the New York Telephone Building "the most impressive modern building in the world" when it was completed in 1926.

    How antiquated it now seems.

    The 32-story structure at 140 West Street, one of the city's first Art Deco skyscrapers, is now owned by New York Telephone's descendant, Verizon Communications (news/quote). And the heavy damage the building sustained on Sept. 11 underscores the vulnerability of communications networks operated by Verizon and other telephone companies -- sprawling systems that rely heavily on critical hubs.

    In the days after the Sept. 11 attacks, it became commonplace to comment on how well the Internet performed because it was designed to route traffic around damage. But the telephone network, including the dedicated data lines that are used by big corporations, financial institutions and others, does not have the Internet's self-detouring abilities.

    When they work, the telephone network's voice and data lines can be superior in quality and carrying capacity to the Internet. Yet when the telephone network is damaged, it cannot heal itself.

    And while Verizon has worked almost around the clock the last month to restore operations at 140 West Street and service to its customers, the company has indicated that significantly reducing the building's network vulnerabilities would require more time or money than Verizon is willing to expend.

    Domingo Mones/Verizon
    Falling steel girders pierced the exterior of 140 West Street.

    The Security: Rivals Worry About Access as Verizon Seeks Buffer (October 12, 2001)

    Get Stock Quotes
    Look Up Symbols

    Portfolio | Company Research
    U.S. Markets | Int. Markets
    Mutual Funds | Bank Rates
    Commodities & Currencies

    Verizon's building was near the north tower of the World Trade Center and next door to 7 World Trade Center, which collapsed several hours after the attacks. Falling rubble and steel girders tore into 140 West Street, which housed one of the nation's busiest telephone central office switching stations. When fully operable, it serves a customer base comparable in number with all the telephone lines in a city the size of Cincinnati.

    After electric power for the building was interrupted, service was temporarily disrupted for more than 300,000 telephone lines and 3.6 million high-capacity data circuits, many serving the New York Stock Exchange, large financial institutions and other companies in lower Manhattan. A gaping hole was torn in a seventh-floor exterior wall, exposing and damaging huge communications switches dedicated to the information needs of the banking company J. P. Morgan Chase.

    In the last month, Verizon has labored to restore service or provide new service for customers that have moved to other parts of the city or to New Jersey. Virtually all of the fiber optic lines and copper strands that had wound their way under the streets and sidewalks and into 140 West Street are being replaced. Some circuits have been rerouted to other Verizon central offices in Lower Manhattan.

    "The ideas we previously had about diversifying our networks have become much more important," Lawrence T. Babbio Jr., Verizon's vice chairman, said in an interview last week as he led a small group of journalists on a tour of 140 West Street.

    Until last month, the most obvious reasons for network disruptions were natural disasters like hurricanes or floods. Now, though, Verizon and other telephone companies must worry about the possibility of physical attacks on their installations. Mr. Babbio warned last week that significant harm could be done to the nation's communications system if terrorists destroyed the 50 or 100 most important central offices.

    Verizon, which is the dominant telephone company on the Eastern seaboard and operates in 30 states overall, is seeking to increase security at its central offices, where it is required by federal law to lease network access to its competitors. After Mr. Babbio issued his warning last week, competitors said they would resist tighter security measures if it made it more difficult for them to conduct operations within Verizon's central offices.

    Beyond physically shielding their switching centers, phone companies can protect their communications networks from direct attacks or peripheral damage from nearby attacks by routing voice and data traffic to other parts of their own networks or those of other companies.

    But Mr. Babbio said that it would take Verizon five years to build alternate pathways for all the telephone lines that wind their way into and out of the New York Telephone building. And Verizon has no plans to do so.

    The reason may be a simple cost- benefit analysis. Despite its primacy to Lower Manhattan's communications network, the central office at 140 West Street accounted for less than 1 percent of the traffic on Verizon's nationwide network.

    "So much of the activity on networks takes place at dispersed locations," said Roy A. Maxion, a system scientist at Carnegie Mellon University. "But the fact remains that we're vulnerable even after putting redundancy systems in place due to the physical nature of connecting to our networks. The issue should be what level of risk you're willing to live with."

    Assuming they are willing to spend the money, business customers can achieve redundancy, or surplus and backup capacity, by running cables to several different central offices or, in some cases, by using several different communications carriers. Several of Verizon's competitors, in fact, have benefited from the disruptions by signing up new customers in Lower Manhattan.

    "Identifying potential failures in networks is not easy," said Joe Flach, vice president of the Eagle Rock Alliance, a consulting company that provides advice on disaster planning. "The most important thing to avoid is putting all of your eggs in one basket."

    Only after Sept. 11 did executives from the financial services industry in Lower Manhattan come to realize just how many of its eggs were in that one 75-year-old building.

    Mr. Babbio recalled having to explain the situation at a meeting in Midtown Manhattan on Wednesday, Sept. 12, at the Park Avenue offices of the investment bank Bear, Stearns. Executives and government officials present included Richard A. Grasso, chairman of the New York Stock Exchange; Harvey L. Pitt, chairman of the Securities and Exchange Commission; Richard S. Fuld, chief executive of Lehman Brothers (news/quote); John A. Thain, a president of Goldman Sachs (news/quote); and Peter R. Fisher, under secretary for domestic finance at the Treasury Department.

    The group was not happy when Mr. Babbio said how long it might take to restore basic service. Mr. Grasso had been hoping to reopen the stock exchange on Thursday or Friday. The following Monday now seemed ambitious.

    "It was not an easy meeting," recalled Mr. Babbio, who spoke with the group immediately after visiting the disaster site, where his clothes had picked up the odor of smoke and ash. "I smelled awful after coming back from downtown. No one wanted to sit next to me."

  • ... (Score:2, Redundant)

    by !ramirez (106823)
    The problem is that the telephone network isn't a routed/multiaccess network like the majority of the Internet is. You still have upwards of 10K users (lines) terminating into one telco building/closet/whatever. This simply isn't going to change; telcos, being the legacy providers that they are, simply don't have the capital (or incentive) to go and redesign a service like this from the ground up, when it performs 99.99% of the time, catastrophe or not.
  • Trying to set up truely redundant telco access can be really hard to get in practice. Sure, anyone can buy separate T-1 (or whatever) lines from two different carriers, but given how frequently equipment and capacity is leased and co-located throughout all the big players, it is just about impossible to guarantee that those two lines don't share a single point of failure somewhere.
    • Trying to set up truely redundant telco access can be really hard to get in practice.

      In theory I don't think it's that hard - at least if it was planned for from the beginning. Maybe this technique can be used for new estates.

      One way in the suburbs is to actually set the phone network up like ISPs. That is, every 1000 homes or so is connected via the local loop to what is essentialy a multiplexing box. Every Multiplexing box is connected to two different exchanges via fiber running in two different directions.

      This method can actually be cost effective because insted of running 100,000 pairs of cable to the exchange, you are running 1000 pairs to a local box.

      Disclamer: I know jack shit about phone network design, the above is just a little logical thinking.

    • How true. One fiber cut seems to take out a whole area.

      Speaking of Fiber cuts, Verizon was fixing a major fiber connection in a van. After 4 hours of fixing the fiber bundle, they realized they had the cable running in the side of the van, and out the back. It was easier and cheaper to cut the van corner and pull the bundle out.

  • by bstrahm (241685) on Monday October 15, 2001 @05:35PM (#2433308) Homepage
    So they are saying that if you take out a large telephone networks Central Office, people connected to this office will suffer lost connections. Infact some long distance connectivity will suffer as well.

    Why does this suprise anyone. Hmmm let me see, if you take out your ISP, all of the sudden you will loose connectivity to the internet unless you pay A LOT of money to have a second line put in. Even then the chance that both of those lines run through some common area is pretty high.

    Things are easy to engineer with fully redundancy, what isn't easy is to do it cheaply enough that people will still be willing to pay for it.
    • Is a suburb out side of Chicago, which suffered from a fire in the CO.

      Phone service was cutoff for a large swath of suburbs in the area, they couldn't contact anyone else, and this went on for weeks.

      yes, it was only one CO, it affected thousands not served by that CO, but whose traffic was routed through there. In a heirachal network, when you loose a node, you lose everybody below you, which can be really bad if it is a high enough node. In this case, it was the one CO where the LD carriers connected to the network.
    • It may be too hard/too expensive to engineer a system with full redundancy, but what about partial redundancy? For example, suppose that you have 1 (very costly to run) link between two points. So, rather than running a second big pipe, put in a smaller, cheaper pipe that offers partial service, as opposed to no service when the link goes down. There. Partial redundancy. It seems to me that that might be a workable solution.
    • by Sonicboom (141577) on Tuesday October 16, 2001 @01:05AM (#2434853) Journal
      My brother is a Verizon installation manager downtown, and he told me one thing that isn't being publicised about the WTC tragedy.

      When the towers collapsed, hat large antenna that was ontop of one of the towers pierced the Verizon bldg. on 140 West St. and travelled through the wall, down through several floors, through the basement into the cable vault, which is 2 stories deep there. It proceeded to annihilate a few racks of cable in the vault before coming to a hault lodged into the floor of the cable vault. As a former Outside plant tech for verizon (lineman) who used to pull cables into vaults - I can vouch that this one event alone caused considerable ammounts of damage. Go look at http://newscenter.verizon.com/wtc/ to take a look at the damage done to the 140 West St. Central office.
      There was over 30 feet of rubble covering the outside service holes to feed cables into the vault too... the switches were also pretty much destroyed from the debris, the antenna, and water damage from broken pipes and the sprinkler system. The vault flooded from broken pipes, sprinklers, and the water used by the NYFD.

      With all things considered, Verizon got circuits rerouted and are restoring them in a rather timely fashion. There is redundancy in the WTC area via SONET rings and other things, which helped get limited service back up as quick as it did... but Slashdotters must realize that MILLIONS of circuits were annihilated during that attack, including CO's in the basements of the WTC too.

      Those old telco buildings built during the Bell System years are tough!!! They're built strong!
      They weren't made to have 110 stories dropped on them tho... no buildings are. A tragedy like this is hard to be prepared for... .
    • Hmmm let me see, if you take out your ISP, all of the sudden you will loose connectivity to the internet unless you pay A LOT of money to have a second line put in.

      That Hmmm almost put me into a trance of agreement, but the implications are way offbase. The internet was designed for redundancy. The designers intentionally set out to eliminate single points of failure and make distributed control. We are supposed to have many ISPs, many lines supporting a network of peer machines. It is NOT supposed to work like the phone company with ONE single service provider in a single venerable building. No one but assholes (the greedhead in the middle) would want a world with one or two ISPs dominating an ocean of powerless consumers of information. We have those and they are called TVs.

      The price of this redundency is not as great as you make it out to be. I could have cable, DSL, wireless and a normal modem all working at the same time if it were that important to me. The powers that be seem to be assholes, however. They continue to spew lies to build the future of digital rights management and publishing control they think they can master. My hopes are now firmly on wireless. It's easy to destroy a central telephone office. It's harder to destroy a distributed cable network. It would be almost impossible to destroy a cable network linked by wireless at thousands of points.

    • The telephone system is designed so that the larger system is able to survive a disaster of (most) any group of parts. This is like saying (for example) that cutting off your right pinky finger is not going to affect the left pinky... On the other hand (if you'll excuse the pun), your right hand will hurt like hell, and your dexterity will be somewhat impaired (if only by all the bandages necessary to staunch the wound).

      Similarly, I would expect to find that the largest national impact of the WTC disaster on the phone system was all of the people calling into and out of New York with (or for) news on survival (or lack thereof) of friends, family and colleagues.

  • i just returned from a week long training trip in lower manhattan the and best western that i was staying at didn't have telephone capability. i just thought it was interesting to see all of the surrounding businesses including the hotel itself usin cell phones. imagine the tumors...
  • Telco switches and networks are the most reliable. 99.999% (5-9's) uptime. Better than IP, cable, wireless... Just ask the dorkwads trying to get VOIP to work...
    • Not really. SS7 hardware has some buggy software, and after fixing a HLR/VLR (databases) and rebooting it, you just dropped your fivenine uptime.

      Nobody really has fivenine, you can fake fivenine, if you exclude your maintence windows.
      • 5 nines are required for wireline telco hardware. You might expect less than that other applications, but if you're talking about telco hardware made by the big companies (Alcatel, Lucent, Nortel), that kind of uptime is taken seriously. This equipment includes local exchanges, access tandems, long distance switches, and the SS7 network. So switches designed for wireline telco usage must meet the fewer than 5 minutes of downtime per year requirement.

        SS7 networks are some of the most reliable in the industry. They're designed to be completely redundant, with the specialized switches (called STPs) set up in mated pairs, located in different parts of the country in the event of a catastrophic disaster. HLRs are typically run in mated pairs as well, so if you're updating the software in one, you still won't lose that kind of service because the mate can take over any functions.
  • Think about it -- in countries where wired infrastructure is lacking (most third world countries, eg in Africa, for instance), mobile phone usage growth has FAR exceeded those in developed countries with wired infrastructure. There are many more mobiles in such countries than landlines.

    If it's shown that our telephone network could be vulrenable to attack in terms of central offices, etc with the potential for major disruption, might we see a radical shift towards wireless as the primary transport mode of telecom, rather than landlines? And/or satellite phones, if you really want to make them hard to get (it'll be a while before terrorists can shoot down satellites, I guess.)

    Yes, it will be expensive, but do you think such a thing just might happen?

    • I do not really know a whole lot about phone networks, wireless or otherwise. Let me say this though: Just because cell phones or satellite phones do not have wires does not change the fact that if you take out a node, everything connected (though the connection is wireless) is going to go down. If a cell phone tower goes down that was serving your area, you will probably have to go somewhere else to use your phone. (I am pretty sure not all celular networks are redundant).

      Also, do you think it would be easier for a determined enemy to disable the CO or a tower? The point of the article is that CO's are vulnerable, and since the phone network is hierarchical, everyone connected to the CO is vulnerable. Same is true for a microwave tower, though they are probably cheaper and easier to replace.
      • The one real point here is that by obsessively building ever larger central nodes in star configurations, instead of making many smaller central offices linked in a mesh, they greatly increase the area knocked out by a single disaster. If you lose the central office your house is connected to, of course you lose phone service. You also lose ISDN or ASDL internet, because that's the same darn bundle of wires. Possibly the nearest cell tower is on top of that CO, so it goes down too. This is pretty much unavoidable, short of doubling the cost of everything.

        But there is no reason that taking out one building should take out service over 500 densely populated square miles. They could have stuffed an automated CO into a closet every six blocks in NYC. Instead, they ran all the wires for miles to get to one building which was filled with multiple copies of the same automated switch. It probably cost more (because of all the extra length of copper) than a distributed network, but being monopolies local phone companies just pass those costs on to their customers, so why go to the trouble of changing your business structure every century?
        • But there is no reason that taking out one building should take out service over 500 densely populated square miles. They could have stuffed an automated CO into a closet every six blocks in NYC.

          Remember that telephones have been in use for over a century. Through various level's of technology. The wires themselves could be 30 years old and the routes they follow 80.
          Indeed you still see relics of long obsolete equiptment in numbering plans. (Indeed the NANP with it's 3-3-4 format is a good example). The last 4 digits originally refered to a a piece of hardware which could handle up to 10,000 numbers (You could do multiple numbers to one line easily enough electromechanically, but for one number to ring multiple lines you'd need something like 0001, 0002, 0003, etc.)
          As new technology came along what was generally done was to replace an old piece of kit with new hardware which worked in the same way. Radical changes (especially to numbering plans are uncommon, typically you get constant "tinkering".)
          Whilst whatever is built to replace the WTC might well have a distributed telephone switching system with hardware every few floors (and a fibre backbone) don't expect the whole of Manhatten to end up that way.
      • I had an experience along these lines. When Hurricane ??? hit New Jersey in 99, the flooding in Hackensack took out the switches in the basement of the Bell Atlantic building situated on the suggestively named River Road. All local phone service was out. So was my cell phone. My AT&T cell phone that I was roaming on in another part of the country. Seems AT&T rented space in the building...
  • by joenobody (72202) on Monday October 15, 2001 @05:39PM (#2433326)

    This makes perfect sense: the Internet did well because it relies on smart endpoints (computers) and unintelligent routes. The best routing, then, is equal speed routes from and to every endpoint and we see something approaching this with multiple routes connecting small groups of hosts.

    The phone company relies on dumb endpoints (phones) and a smart system in the middle. The best (simple) routing solution would be every phone connected by a line to a central switching station. In an urban area, this is exactly what we see- one or two central switching stations or point of failure.

    This really shouldn't be any surprise at all.

    As a side note, this is also why growth and development has been much faster than on the phone- to change the phone system you have to change one place - but no one will let you, because you might break it for every other customer. On the Internet I can tinker with one or two machines and everyone else is unaffected.

    • you're making a false comparison. you say "the internet" did well - maybe, but so did "the phone network". i live outside Manhattan, but in the NY Metro area, and i had no problems with my phone connectivity, cel or land-line. by contrast, my ISP went down shortly after the towers, when their upstream provider's emergency generators ran out of fuel.
      your comparison is (intentionaly, i suspect) unfair becuase you're comparing the fact that a small area of the phone network went down to the fact that the Internet overall continued to work. it's a stupid comparison.
      your "analysys" is also ignorant of the physical network underlying both the phone network and the Internet. the phone network is built on top of a series of actual, physical links. the Internet is built partly on top of this, partly with additional links. lost of my friends in Manhattan lost IP connectivity because - suprise! - their phone service, which they use for IP connectivity, wasn't working.
      sure, my IP connections to California were unaffected by the WTC going down. but i made phone calls from 15 min. outside the city, that day, all over the country, with no problem (other than into Manhattan).
      for all that talk of redundant routing on the internet, how many lines do most people have protruding from the back of their box? 1. how many ISPs do most people have? 1. how many upstream providers do most of those ISPs have? 1. all single points of failure.
      ask someone you know familiar with the net what would happen if someone took out MAE-East or MAE-West, among a handfull of other very important Internet sites. it'd be much easier, in fact, to make the Internet useless by taking out ten or so buildings than to take out the phone network by taking down 50.
      ...this is also why growth and development has been much faster than on the phone...
      uh, yeah, but it's also why my phone crashes so much less often (uh, never?) than my PC (rarely, 'cuase i run good stuff), and why my telephone company won't let me connect so much less often (once in my lifetime, while the line up my street was being worked on) than my ISP (once every month or two).
  • by Zen Mastuh (456254) on Monday October 15, 2001 @05:40PM (#2433329)

    Physical vulnerabilites (location, etc...) aren't the biggest worry.

    Not too long ago, Wired [wired.com] ran an article about the apparent h4x0ring of phone lines in and around Las Vegas. It seems that a certain escort service (prostitution is legal there) would stop receiving phone calls, especially on busy nights. The employees would call their number from another line, but the phone wouldn't ring. When the authorities came to investigate, the phones miraculously started working again. So the mobsters are in it with the telco employees or the cops or the h4x0rz. Anybody with a copy of phrack or 2600 can probably hijack a switch. This has been known for years. Perhaps there is a large-scale secret phone net that dries up when the telcos or feds try to dial in?

    Regardless, the telco infrastructure is hopelessly inadequate.

    • by rfc1394 (155777) <Paul@paul-robinson.us> on Monday October 15, 2001 @06:01PM (#2433425) Homepage Journal
      apparent h4x0ring of phone lines in and around Las Vegas. It seems that a certain escort service (prostitution is legal there) would stop receiving phone calls [deleted] authorities came to investigate, the phones miraculously started working again.
      Contrary to popular belief, prostitution is not legal in Las Vegas, Nevada. It is just as illegal as Chicago or Los Angeles. The rule is that rural Nevada counties (population under 50,000 I think) may permit it if they choose to do so; Nye County is one such place, about 80 miles from Lost Wages... Even if they wanted it in Las Vegas, the county is too large to have local option on this and so it's always been illegal there.
      Regardless, the telco infrastructure is hopelessly inadequate.
      That statement was probably just as true 20 years ago and it's probably gotten even worse since then.

      Paul Robinson <Postmaster@paul.washington.dc.us [mailto]>

  • Any time there is a signifigant change on the phonelines, DSL service can be interrupted. I can just imagine how badly this affected DSL customers in the area. And with the fragility of fiber line, I guess even more people were affected. Imagine what was to become of all the water/gas/electric lines running into the building. Our nation needs to build in redundancy for such things. Gas/electric/water lines have physical redundancy, they can be cut off at the last point. But for DSL/fiber, it doesn't automatically inform the server that the connection will be unavailable. The packets flow straight to hell.
    • I live in Queens, NY.
      My Eathlink DSL went out on 11th, and
      didn't start working till several days later.
      And it flakes out every day since then.

      -DVK
  • Should the world's major Stock Exchanges have redundancy?


    How long would the NYSE have been down if there had been a direct hit on it by terrorists?

    • The US stock market does, at least. At least 2 different sites that I know of maintain duplicate copies of all information generated during each trading day. If it had been hit, they probably would have shut it down for awhile for security reasons, but theoretically it could reopen the very next day.
    • Of all the exchanges, the NYSE is THE most vulnerable because it actually has a trading floor where brokers and specialists execute orders. (AMEX and NY Board of Trade, have floors, too...but nowhere NEAR the scale of the NYSE. The NASD owns Amex anyway...and the NASD's automated system is the antithesis of trading floors.)


      If something were to happen to 11 Wall St. (NYSE), trading would be halted indefinitely. Of course, they may have a redundant trading floor in one of their other buildings. But I wouldn't expect them to be able to handle the amount of volume they handle at 11 Wall St. That's why they have a ton of security outside. You can't walk near the building unless you have a NYSE ID, and you can't drive a car or truck very close either.

    • The biggest problem the stock exchanges has was that after 9/11 many of the folks who worked as traders for the big Wall St firms were at best without offices and at worst dead. Yes you have data backups but loosing 20% of your staff has got to be hard. And there was brockerage that had 3,500 people in one of the towers. Most of them got out but not all.

      In truth the hardest thing to replace is the people.
  • Unless the telco's are going to install two services to every house and every business, each one running in seperate duct, to a seperate exchange, with the backup exchange having fully-redundant backhaul circuits that wouldn't be affected by (lets say) someone flying a Boeing product into the primary exchange, the system is still going to be vunerable. As long as all the copper runs back to one building, if "something" happens to that building, theres going to be an awful lot of people with "NO DIALTONE".
    • But nobody will fly a Boeing product into the primary exchange, because we have defeated the Evil h4xx0rZ with the USA act! Now we can stop them from posting their '1337' encrypted messages to all their terrorist buddies!

      Just out of curiosity, did the USA Act make it illegal to fly a plane into a skyscraper, or is that still OK?
  • Cable (Score:1, Informative)

    by crumbz (41803)
    The most difficult part of installing a new switching station is managing the hundred of miles of copper and fiber that interconnect within the building. Combine that with identifing and splicing the incoming fiber, copper and coax and you have a task requiring ten (hundreds) of thousands of hours of labor. In addition, only so much work can be performed concurrently within a given area in the CO. It is a monumental task.
    Building a brand new CO is far easier than repairing or perfroming MAC work at an exisitng facility (ask any old Bellhead).
  • ...voice over IP.
    Perhaps now the strenths of VoIP will be shown instead of just the "wizz bang" of it all.
    Look at how quickly internet access was restored to the area via wireless.
    By seperating the network from the application, it becomes much more robust.
    • And, with VOIP riding over a single line to each residence, you've accomplished exactly what?

      Redundancy for the casual consumer is just not practical. In order to do it right, you need fully diverse cables and conduits to/from *each* residence, each entering the residence in different parts of the building, and terminating into different CO's. You want your phone costs to double? I don't.

      If you are a hospital, gov't office ( police, fire, ... ) you're phone service is on a priority restore. IE, anything that's not priority gets whacked until all critical service is restored.

      It dosn't matter whenter you use voice over cowboy neal, if you haven't provided 100% diversity to every piece of the path between you and the phone switch, you are susceptible to exactly this type of catastrophe when something happens to the piece that isn't fully redundant.

      For the business or really rich person who decides that they simply cannot afford to be down, even if a 757 hits their CO, you *can* get diversity. Be prepared to pay a lot of money for it, though, because it's not cheap. For the rest of us, between my POTS ( plain old telelphone service ) and my Cell, I'm comfortable that I've done pretty much all I can. Anything more and you're hitting the wall of diminishing returns for the money you're expending.

      Remember, buzzwords do not a problem solve.

      • And, with VOIP riding over a single line to each residence, you've accomplished exactly what?

        You misunderstood the point of the VOIP solution. It is not intended to provide redundancy to each point in the node. It prevents someone from knocking-out the central office and disabling millions of residences. Noone is suggesting that each home requires redundant lines coming into each side of the house - we aren't worried about someone cutting individual lines.

        • by Anonymous Coward
          Alas, it's at a point like this where the data guys and the voice guys show their true colours...

          While VoIP or VoP (Voice over Packet) in general is a fine idea, where is the box that converts the datastream that represents the voice back into voice? 99 times out of 100, in that very same CO where the legacy switch is.. Hell, with some of the newest Nortel DMS products, the VoP engine is actually PART of the switch.

          The voice network is far more reliable then the data network - the does NOT mean it is more redundant, just more stable - after all, when's the last time you had to reboot your phone?

          Without going to wide-scale wireless, which has some pretty interesting problems of it's own, there is no practical way to provide redunancy when large chunks of the infrastructure are smashed, burned, or otherwise rendered useless.

          Heath
  • What distribution? (Score:4, Interesting)

    by rfc1394 (155777) <Paul@paul-robinson.us> on Monday October 15, 2001 @05:51PM (#2433382) Homepage Journal
    Turns out there was a problem in that much of the network switching was in one building [deleted] lots of pondering about the vulnerability of the network, even when it is distributed across many physical locations. [deleted] estimate is for five years' work before there could be redundant paths [deleted] with no plans to spend the money to do it.
    Ever since the Hindsdale, Illinois fire in a telephone switch room, it has been or should be known that telephone companies routinely under-build and over-load equipment and only add it when they absolutely have to (or are possibly forced to by regulators once in a blue moon), and then complain that they need to raise rates to pay for it, as if they are supposed to be able to operate without equipment and that's not supposed to be part of the cost of service.

    If this equipment is that important - and we know it is from the cost to replace it - why isn't it even worth the cost of one clerk at minimum wage around the clock to be able to check on things there? Someone once pointed out that Illinois Bell Telephone ended up spending millions because of the fire, hundreds of times more than it would have cost to have have had a single person present on each of 3 shifts, to provide a 24/7 presence in that building for the next 100 years.

    Someone who claims that telephone service is distributed should look again; I've never found a telephone company that operated more than one central office for an area and in some cases trying to combine them in larger and ever larger buildings until the central office for an area might be 40 miles away, yet still continuing the previous rate structure - which may have been created 30, 40, or 50 years ago or more - so that a call to another phone connected to a different switch in the same building is a toll call because it's in a different rate center.

    If all the mergers and acquisitions of telephone companies by each other was supposed to benefit the consumer, why is phone service more expensive than ever?

    Paul Robinson < Postmaster@paul.washington.dc.us [mailto]

    • More info [ncl.ac.uk] on the Chicago C.O. fire, interesting stuff.

      A few notes:
      "Non-local telephone service was cut off for customers in an approximately 500 square mile area"

      The phone company employees tried to call the fire department, but of course the telephone lines did not work.

    • If all the mergers and acquisitions of telephone companies by each other was supposed to benefit the consumer, why is phone service more expensive than ever?

      Long distance service is much, much cheaper than before de-regulation. In the 1970's, it cost nearly $1/minute, so the cost has gone down nearly 10 times even though the $ is worth much less than it was. This reflects both cut-throat competition and technological advances (fiber instead of copper).

      Local service has gone up, perhaps with inflation, perhaps more than that. No competition, and no real technological advances since digital switching went in about 30 years ago. Hook-up charges are even more outrageous; in most cases, the house is already wired and all they have to do is tell the computer to turn the line on, but they'll charge a $70 "service call".

      Understand the difference between a competitive market and a regulated monopoly yet?
  • It's worse in T.O. (Score:2, Insightful)

    by OrenWolf (140914)
    There's a building in Toronto (151 Front Street West) that's known as a "Telco hotel", in that it contains most of the switching equipment for most of southern Ontario.

    What's interesting however is that the ISP's of the area have also moved into this building, due to it's prime location downtown and the proximity to Telco facilities.

    If someone were to drop a bomb on this building, phone service for half the province and Internet connectivity for a huge part of the Greater Toronto Area would be toast.

    It's one of those things that's oft-discussed as you take the elevator up into the building. Our only hope is to remain "under the radar" of Terrorists. :)
    • Yeah, but Toronto's built to be taken out. A friend and I were discussing it one day, after a cop got shot in the face on the 401. You could drop somewhere between 5 to 7 bridges that happen to go over the major access points in and around Toronto (lots of three digit numbers starting with '4'; two on the 401, two on the 407, one or two on the 410, 427 and 403) and Toronto would be shut off from the outside world. We figured a week, at MOST, before the city decended into food riots, looting, and old men with shotguns on the porch. I'll bet you could apply that to pretty much any major city and get teh same result.
      Their population centers are clustered ridiculously close to one another! These primitives are completely ignorant of space-war tactics!
  • Doesn't take much (Score:5, Interesting)

    by JediTrainer (314273) on Monday October 15, 2001 @05:59PM (#2433415)
    I don't suppose anyone else remembers the
    infamous fire [www.exn.ca] in a Bell Canada phone exchange in Toronto. This fire knocked out phones in much of the city for a couple of days as the crews scrambled to fix things. It was interesting trying to do business....

    In my company's case, we still had working Internet via ISDN, so we were still able to go about our business. Some cell phones weren't working, however.
    • Re:Doesn't take much (Score:3, Interesting)

      by Nos. (179609)
      I was actually on a tour of the telco facilities in Ottawa about 18 months after that happened. The tour guide explained to us what happened. It was a dropped tool, apparently a wrench, but it wasn't on the switch, but on a set of the batteries themselves.

      The batteries look like a bunch of oversized car batteries hooked together, but there is no insulation on the wires, just bare metal. As such, when the wrench dropped and connected +ve and -ve, sparks flew.

      Its surprising that they were able to stay running as long as they did since fully 1/2 of their batteries were toast after that incident. Its also surprising that after six months Ottawa's batteries still had no insulation on the connectors. Over 2 years later, our batteries on our switch (server 5 T1 lines) are still bare metal as well. Of course that's a much smaller setup, but it is serviced by Bell Canada :).

    • Oh, I remember that fire. I was working at a wireless provider at the time, and there was bedlam at our switch site... both our networks routed huge amounts of traffic via that ILEC CO and without it, our other routes were drowning.

      In the grand scheme, everyone survived it rather well, with little more than inconvenience for most (massive downtown bank HQ's were a notable exception). It was a really interesting peek into the vulnerabilities of the usually stable network.
  • by Anonymous Coward on Monday October 15, 2001 @06:04PM (#2433436)
    for reasons below.

    Might sound like a troll, but here goes. If you would like more specific info on the tech, reply and I'll reply to you.

    I work for a large regional telco in Canada.

    I routinely work on various switching and transport equipment. I think I'm finally somewhat qualified to post to slashdot on atleast one topic.

    Modern telco equipment is VERY expensive. Large transport shelves will range anywhere from $150 000 to $400 000 per shelf (Canadian, transport being the fibre equipment) There will be several of these shelves per Central Office.(found in every neighborhood) Cards to fill these shelves will range from $8000 to $70000. (they burn out WAY more than I like, usually at 3 in the morning) Switching equipment is even more expensive, the prevelent DMS technology from Nortel Networks is per capita is even more expensive. I would imagine their competitors prices are about the same, although don't quote me. You will have several of these shelves per office as well.

    As well, any good telco will have spare equipment on hot standby - major components at 1:1 and lesser at maybe 1 to 10 or 1 to 8 depending on manufacturing

    Incidently, you also need expensive people to program and maintain the equipment. A good example is a DMS technician who will get paid the same as an excellent UNIX admin. (and rightfully so, the DMS is a convuluted enviroment to work in)

    Each Office needs to be built to the highest standards, physical security, enviromental controls, backup battery plant and huge power systems to feed the equipment

    Outside Plant, (that being the fibre and copper cable), is expensive as well, and even more expensive to maintain, this is why you see very few redundant routes, possibly only within a city. Often there is only 1 redundant route, in the classic SONET ring configuration, and often both sides of the ring have to terminate in one physical location. (office building collapses, phones don't work)

    I don't know anything about the telco in manhatten, but I can imagine the catastrophe of losing a major office. If they were cutting corners on redundandcy, (which thankfully happens very seldom in Canada due to the regulations here) I could see major routing problems.

    For those of you who thing telephone networking is like IP routing, it's not even similar. It's a hiearchy, you cut off the head, it suffers. Many companys may only have 1 or 2 hosts (a host being the "CPU" of the network.) This is due to the expensive of running a host. Telco equip manufactures charge an arm and a leg and your first born, and the liscensing is microshod style draconian.

    What I'm saying after all that is - if you want total redundancy everywhere, it's going to cost more money for service. I don't know what the competition is like in Manhatten - but if you're not paying much for your cell phone, there might be a reason.

    Just a thought. Flame away.
    • I worked for a VoIP company until recently. The product we were using was _significantly_ cheaper than a DMS, and would provide ALL the same facilities as a 250(at class 4 level). Total price for a box capable of 7392 DS0s: $1M. Significantly cheaper than a DMS 100 or 250. Note that you'd have to provide your own channel banks, but that's normal. IIRC(and I'm probably wrong), the 250 doesn't have loop capability, so you'd have that cost anyway.

      The major problem with traditional telco is the single cable from the telephone to the switch, as well as the stupidity of the phone. MGCP/Megaco/SIP solve the stupidity problem, but don't resolve the one cable problem. At a higher level, SS7 is just plain stupid WRT routing and access. If interested, I'd be happy to elaborate on it.
    • Oh for heavens sake, stop fixating on karma. If you want to be a serious slashdotter and get that permanent bonus point, you have to take chances, and not worry about on occasional downmodding. That was true even before the upgrade, when half the moderators were trolls in drag. Now you've lost a chance to raise your karma four points. Next time you do a thoughtful, informative post, trust the moderators to see it.

      Hey Rob, maybe people shouldn't know what their precise karma is!

  • Either we embrace it and secure our future, or don't and remain forever vulnerable.
  • by DNS-and-BIND (461968) on Monday October 15, 2001 @06:11PM (#2433458) Homepage
    Old-time linemen and phone phreaks will tell you the same thing: a Telco's idea of a redundant circuit is two cables in the same conduit. About the only disaster-resistant construction Telcos undertake is replacing wind-blown-down telephone poles with underground cables.
  • September 11 has, as stated caused people to rethink their vulnerabilities. The recent terrorist attacks, I believe, proved that a centralized model for telephone switching is no longer viable.


    During my years at Bell Labs, we drew up a fast, redundant, distributed switching system. At the time, technology wasn't up to implementing it cost-effectively. But today, it could be done for cheap using Linux and the Linux Router Project. Nearly all switches in the US are already digital, and a changeover to a fail-safe, decentralized switching system operating along the lines of a packet-switching network would be trivial. I'm almost inclined to call the Telcos irresponsible for not having made the change already.

  • Perhaps a good approach for the general /. public would be to think of it as a large network. Right now, there is one large managed switch at the center, with 100s of thousands (probably millions) of computers connecting to that switch. Now, what's the best way of implementing redundancy at a separate location?
  • Have they done any simulations on the impact of different failure scenarios on the system?

  • it is probably the RIAA hacking into phones, looking for mp3's of songs used as cell phone rings.

    (ALL YOUR PHONE (and base stations) ARE {sound of one hand clapping}
    Owwww, that hurt.}

    Heh, or not.

    Moose.

    (not a troll, just an attempt at humor...if I fail, just ignore me, everyone else does) :\
  • We have been working on making our networks/servers diverse for a couple years now. There are some technical problems to overcome when you need to switch an entire building and its servers/networks to another across the country.

    All the big co's will start to become more diverse, and the people who come up with the new technologys (and own the patents) will become very rich. Very cool stuff coming out of R&D, wish I could go into detail.

    One of the cool features about making your network diverse, you can upgrade one location and switch to the upgraded services running new code. No downtime for maintence windows. (Ok a couple seconds while you switch routes)

    Here in Seattle, there is 1 building downtown where all or most the telcos have their Internet feeds, if that building was attacked or hit with an earthquake, Seattle would be without telephones for a month.

    You cant tell your stock holders "Umm, sorry, the networks down, be up in 30 days..." Well, I guess you CAN tell them. Just wont be working there much longer.
  • Survery Says: DUH (Score:2, Interesting)

    by fooguy (237418)
    As for a former New York Telephone/NYNEX/Bell Atlantic/Verizon employee, this is no surprise. Everytime there was heavy rains in lower NY State Long Island and Staten Island (516) could only get the operator - switching in and out of that area would shit.

    The large scale upgrades to digital switching in the early 90s happened (sadly) under the reigns of NYNEX - the cheapest RBOC in history (they still printed paychecks on NYTEL check stock).

    The biggest nightmare of the NYNEX/Bell Atlantic years was OSDI. After TOPS and TSPS, Operator Services contracted to get a new switchboard system called Operator Services Digital Integration, which didn't work. Only thanks to NYNEX Science and Technologies were they able to make it work.

    More horrors on my webpage:

    http://eisenschmidt.org/jweisen/bellatlantic.htm l
    • You mentioned heavy rains would mess up switching. Heavy rains screw up the lines here too, though we can usually get a voice call out. Data line keeps switching on and off though.

      Can you (or anyone) go into exactly why this happens?
      • I can tell you why it happens in Long Island and Staten Island:

        Neighborhoods connect to Central Offices, but there is really only one or two connects from the island(s) to the mainland, where their calls are switched back into the network.

        NYC also *still* has a lot of copper. Rain and flooding will cause far more problems with copper than fiber (i.e. a rural area).
  • It's not just Verizon that does this kind of thing. Last spring Houston flooded from Tropical Storm Allison. The Sprint routers went down and left them with Plan B, which turned out to be "Hope that Plan A Doesn't Fail." We had problems with phone service for ages. Furthermore, my university uses the Sprint routers in Houston for the Internet gateway, and wound up sharing out time on the Austin system with UT Austin. As far as I know, Sprint hasn't sunk a pile into their infrastructure to prevent a repeat occurrence...
  • by Anonymous Coward
    It was working perfectly (it was switching emergency calls) until 4pm sept 11th when it's batteries failed. All that with 110 floors piled on top of it. WOW.
  • by ONU CS Geek (323473) <<moc.liamg> <ta> <nosliw.m.nai>> on Monday October 15, 2001 @07:12PM (#2433701) Homepage
    Part of my University-Sponsored Employment means I work for Communications Services--dealing with the phones, computers, and backbones as needed to keep them up. What I've come to find out is that most Administration don't want to plan for emergency situations.
    We were looking at disaster planning. Since we use NEC Phone Switches [nec.com], we were taking a look at what would be the first thing to go. Take a fire...you could get a switch in a semi trailer sent up overnight (or something like that), but your Main Distribution Frame (MDF) would be crud--you'd have to re-splice every cable pair that you have in order to restore service to everyone; depending on how bad the fire is, you'd have to resplice your RDF's as well
    There are some things that we've thought of...like having a bit of redundancy in our wire plant, but the administration shoots us down every time we bring it up.
    I guess what I'm getting at is that there isn't a whole lot of redundancy with SS7. Get into things like Voice Over IP, you'll have some flexability, but if your switch gets royally hosed, you're going to be down unless you've got an extra one sitting in another building with a backup MDF that is current.
  • Redundancy would be great in the phone infrastructure, which their is to some extent, but with circuit based switching its extremely hard to achieve since their will always be at least one point of failure (e.g. switch, copper pair, etc.) Obviously with packet based switching it will always be more redundant since the packets can just be rerouted. Like I can get a Satellite connection and a land based T-1 circuit and if one should go down theoretically the other should pick up the load. The telephone network does this to some extent using SS7, but that only works at the higher levels at not at the actual locations where the CPE (customer premise equipment) is located.
  • It's inevitable (Score:5, Interesting)

    by jht (5006) on Monday October 15, 2001 @07:29PM (#2433802) Homepage Journal
    As long as you have lots of wire going back to an endpoint, the endpoint is vulnerable. Most CATV systems have the same weakness, too. About the only thing that isn't as vulnerable to a single point of attack is the power grid at the plant level, and that's because of grid interconnection (there were some interesting power grid-related articles in IEEE Spectrum a few months back). But at the local level, a few substations feed large portions of a city - in my city of 40,000 or so a single squirrel took out a large portion of the town earlier this year. And we have our own generating station here, too.

    In any tree-shaped network taking out the trunk takes down all the branches. Verizon is just doing what makes (in the pre-9/11 world) good economic sense in not having full redundancy, with multiple paths. What you might see someday in the not-too-distant future is a few areas (like Wall Street) get second switching stations further uptown, but really the best solution for a business that really never thought about the phone network is a dish pointed to a CLEC that isn't in the same CO as the primary circuits from the ILEC.

    If Winstar had remained viable they might well be seeing a big demand spike hit about now as corporate DR people realize their potential weakness.
  • by bryan1945 (301828) on Monday October 15, 2001 @07:31PM (#2433818) Journal
    I was/am in the middle of converting a federal agency in 26 Federal Bldg (about 6 blocks away from WTC) from analog to ISDN phones. We had half a floor converting on 9/13- needless to say, it has been postponed. 26 Fed has about 16000 phone lines, some ISDN, some analog. Analog service is being restored quicker, but almost no ISDN lines have been restored. Overall, Verizon is restoring about 200 lines a day in the building. 3 major problems with telecom after the attacks: 1) There were COs in the WTC and the Amex building, both of which are totally destroyed. 2) The Verizon CO building was damaged, including water and shock damage (I wonder how well an E5 switch handles water). 3) Several major trunk lines were cut to downtown Manhattan. Basically, too many COs were too close together, and every CO in the bottom half of Manhattan have their circuits maxed out, so numbers can only be restored when trunk lines are re-connected. This disaster has shown how vulnerable our infrastructure can be, especially in metropolitan areas.
    • > 1) There were COs in the WTC and the Amex
      > building, both of which are totally destroyed.

      Sorry, wrong here.
      AmEx building, aka 3WFC, sustained *some*
      structural damage but is not destroyed and probably will be useable within a year.
      (half of the building was owned by or company so i know for sure).

      -DVK
      • Ok, sorry about that, I was going by info I got from the telecom guys. When they said "the building was destroyed, I thought they meant the actual building, not just the CO. Silly kids.

        Good point out.
  • alas, you have to put the network points of presence where the customers are. if you could run that DS3 or OC48C into NYC from Maxbass, ND, it would have been done by now. unless the telco execs preferred golf, then maybe all the networks would be clustered around atlanta or pebble beach.

    now, if friend customers had been optical, there is 20 or so miles that their muxes could have been located further away, but political boundaries in organizing the telcos make that another horrid choice.
  • i work for a telco, wcg, and we had an OC12 ATM circuit affected that rode offnet on verizon fiber... it was going to take so long to restore that circuit that we disconnected it.

    I guess we have so many OC12's to throw around that we can just as easily disconnect them????
  • Back home in Arkansas this summer, we actually suffered a phone outage in our area. It was a total telecommunications black out. Not even cell phones could get service in the area (presumably due to the towers' connections to the land lines somewhere). Anyway, about a month later I found out what happened. Turned out that some yahoo had stolen a backhoe to dig a grave for their pet cat. Only, while digging the hole he or she hit a major telecommunications trunk, cutting off service to many square miles of telephone customers. Needless to say, I was kind of pissed to find out that I lost phone service because some redneck was digging a grave for their pet cat.

  • I live and work upstate at a manufacturing company. Although this is our primary presence, we do in fact have an office in NYC. After the ``WTC attack'' happened, the first thing I did, was ping a server in our NYC office.

    No problems expected, our office in manhatten is located at 1775 Broadway in the NEWSWEEK building.

    About a day or two passed, everything was still fine. All of a sudden our main factory T1 goes down, ouch, we'll have to fallback to ISDN, which of course was also down. It seems someplace upstream, a verizon T3 was out. All the data curcuits in the area where out. I called my local office to find the ISDN was out, because although the pop was local, the curcuit was of course routed through verizon's west street office.

    Deluged with helpdesk calls, noone at uunet or verizon could take our calls. We called the local cable company and got a backup uplink onsite nextday. Upstream here was a qwest fibre feed -- now thats reliable.

    I was mystified as to how the damage in NYC could have affected our curcuit here, 125 miles north of the city. The T1 was bouncing throughout the following week until power at west street was restored and equipment was again functioning. Note - all through this, our verizon->uunet link at 1775 broadway stayed up without a hitch.

    Im not sure what anyone else experienced, but all Ive learned is if you think you are redundent, check your last mile. Depending on verizon is like depending on a politician's promise.

    I would be interested to hear anyone with similar (or not) experiences.

  • Not widely known outside the telecom industry is the real estate aspect of electronic switching. Early 1ESS was big, but 5ESS was a lot smaller than #5 crossbar.

    From a regulated telco point of view, this was great, because central offices could be shrunk or consolidated, and the real estate sold off. This produced a huge one-time boost in profits, because the revenue from selling off "excess" real estate went directly to the bottom line. This yielded some huge profits in the 1980s.

    But the result was more centralization, with bigger and fewer central offices. This has made telephone systems more vulnerable. The transition from microwave to fiber hasn't helped either, because fibre tends to be concentrated along the obvious rights of way (railroads, pipelines, freeways, etc.)

  • It's only new because it's in NYT. There is a whole area of research devoted to the problem - designing survivable networks - with labs, a wealth of publications, university courses. A couple of almost obvious basic considerations:

    a) If you need a protection on a link between A and B you need another, disjoint link (to form a
    ring). That is expensive indeed. However, you can't get 100% protection against a link failure without paying twice.

    b) A node failure (such as Verizon) is much worse than a link failure, because it severes many links at once.

    Design of survivable networks is very complicated, and is as much an art as a science. Many networks are not designed with survival in mind. Someone raised the question of what happens when an ISP is taken out. Many ISPs have star-like networks, with a few central hubs. Take one hub out - you better have another access point, or, better, an account at a different ISP. Transocean links are also a problem. Remember about a year ago a big fat cable was damaged in the Pacific, leaving much of Australia without Internet?
  • Twenty-plus years ago, when it was still one Bell System, one of the occasional topics of discussion within the local switching systems engineering organization at Bell Labs concerned how many critical pieces of the phone network could be taken out by a few people with a few trucks full of high-nitrogen fertilizer and diesel fuel (like the bomb used in Oklahoma City). The switching centers in lower Manhatten were always high on the list of sites you would go after...

"Only the hypocrite is really rotten to the core." -- Hannah Arendt.

Working...