Catch up on stories from the past week (and beyond) at the Slashdot story archive

 


Forgot your password?
Close
typodupeerror
×
The Internet

Massive Fiber Cut Slows Net 204

Posted by Hemos from the i'm-moovvvinngg-vveerrryy-sloowwlllyyy dept.
netpuppy writes "East coast to west coast connectivity (or the other way around) feeling slow today? Here's why. It appears that the attack of the raging backhoes has hit Ohio today, where an unnamed public utility managed to cut through 4 OC-192 circuits while working on gas lines. 4 OC-192s are roughly equivalent to 40 Gbps traffic, and trunks this size usually carry both voice and data on them. AboveNet, GTE, and Metro Fiber (now part of Worldcom) seem to be the worst hit, according to this Inter@ctive Week article. " OK, I'm not just crazy. It has been slower then molasses today.
This discussion has been archived. No new comments can be posted.

Massive Fiber Cut Slows Net More

Massive Fiber Cut Slows Net

Comments Filter:
  • Most line have markers telling you to call before you dig. But most of the time you call and they say they will be out to mark the lines, but it such a low priotiry for them that usually it gets pushed to the bottom of their pile. So most projects can't wait for the line owners to get off thier asses and mark the line, and go ahead and dig any way to stay on schedule. (My wife works for a Gas Company and has had to call for one of their projects.)

    But whatever the situation they need to be carefull when they dig.
  • We have now put up a press release [mids.org]. Also, we have created a static copy [miq.net] of The MIDS Internet Average front page [miq.net] showing the period of the outage.
  • if fibre lines were marked the same way gas lines are marked in cities. hmmm....
  • I think if the day after a nuclear exchange you're worried about whether or not you're going to be able to read your email, then today's society's priorities are more than slightly skewed.
  • I doubt it... a few years back, Telstra controlled 90% of AU's incoming bandwidth, at approximately 127MBps (don't laugh, we only have a population of 19 million for an area the size of the US - the most sparsely populated country on earth outside Antarctica. Nowadays, with the springing up of Optus (C&W), AOL, and a few other big names, i think our capacity is approaching 400MBps.
  • I shouldn't mention the name (*sigh*, clue, huge multinational with common two letter initials, who make everything from calculators to computers and medical equipment. Oh, and it isn't TI), but I used to maintain large WANs for our clients - one had a network of approximately 250 ISDN lines around Australia, some very long length. Uptime was superb (99.985%+) and most outages were, as the above said, cable cuts (and powerfailures)
  • I used to work for a construction company laying new fibre optic cable. I did the calling of all of the under ground utility owners. (very tedious) But, my boss kept on telling me not to screw up, because if I messed up the cost for 1 (one) fiber cut is $100,000. WOW that's too much.
  • It already can be a major problem with regards to private WAN links. I've seen more than one occaision when a location on a WAN loses connectivity due to a fibre cut. It isn't pretty, and it makes my life interesting at times ;-)
  • No WONDER I could not get onto slashdot at work today. heh. this reminds me when someone made a ghetto fire under a bridge somewhere in Minneapolis(I think), cutting most connectivity off for the twin cities. I think it was through MR.NET. anyone remember this a couple years ago?
  • If you are interested in this stuff, look at The history of the Internet [isoc.org] for fuller details. But the basic story is that the idea of packet-based networks arose independently in two places. The first was Kleinrock et al at MIT, the second was Baran at RAND. The former group was interested in them as a way of efficiently sharing the same lines between many different computers. The latter was interested in them for creating communication systems that could survive nuclear war. The two groups did not know of each other.

    The Internet arose out of ARPANET which was based on the work at MIT. The goal was to allow computer resources at different research institutions with different types of computers to be shared. The fact that surviving nuclear war was not a goal can be seen in the fact that the machinery used to set it up had no protection against the electro-magnetic effects of a nuclear warhead. Furthermore the initial set-up heavily relied upon a single back-bone. With no redundancy in your physical network, what good is a redundant protocol?

    In fact the initial proof of concept and then proposal for ARPANET was made before the MIT people even heard of the work at RAND. Indeed the two groups found out about each other at a conference where the ARPANET was being proposed. Don't believe me? Check it out [isoc.org] for yourself!

    Cheers,
    Ben Tilly

  • Re:Same boat... (Score:1)

    by Anonymous Coward
    yah, i was upstream from y'all, your uti engineers are really persistent. but it didn't get them fixed any faster:)
  • Everybody seems to be complaining about the fact that things are slower today, but what about the fact that 40Gbps is now being rerouted through other means. That's pretty damn impressive. Sure the people who snipped the lines are idiots, but the whole net didn't come crashing down.
    Anyway just my $.02
  • >Wouldn't you love to have 40gig of bandwith piped into your home? ;)

    A friend of mine does - he runs an ISP that only resells
    bigish pipes (frame relay, T1 etc) out of his basement.

    He claims to have more incoming bandwidth to his house
    than Australia :-)

  • Re:IP vs Phone lines (Score:4)

    by Anonymous Coward on Wednesday September 29, 1999 @11:57AM (#1649739)
    Why is it that IP trunks seem to be backhoed at the drop of a hat, while phone lines are somehow immune? Why doesn't the phone line between Boston and San Francisco go down?

    answer: phone lines are not immune. they get cut all the time. however, the phone network, like the internet, has redundancy. whe na phone line gets cut calls are rerouted to other trunks, making the problem oblivious to the layman.

    so, why do fiber cuts turn out so much worse? because the data network is bursting at the seams. a voice line cut can be directed to ununsed lines. a data line cannot be done in the same manner, because there are no vacant lines, or at least not as many as were lost...

    why is this the case? one, because the data network is expanding faster than new fiber can be laid down. we're too big for a britches. also, the phone network, which has grown at a decidedly slower pace, has, at both the users' and phone/data companies' requests, deemed to be more important. let's face it, if a phone line went down to a company hq, they'd be lost for as long as that line was down. just now are people realizing that their data line, is now becoming more important than their voice line. :)
  • Well, datacom companies should team up with gas companies and put the fiber right beside the gas lines. That way everyone would be scared to dig near em and thus no busted fiber lines :)
  • In the day and age where you can sue your own parents for not making you clean your teeth properly when you were an infant, I'm sure you can sue them for something..
  • Caching proxies? Get 'em installed at your ISPs!! In situation like this...it can help lower the burden a lot - by caching the pages for the people who browse, you save bandwidth for people who have to telnet, shop, or post at /. I think I'll get mine.
  • I see alternate routes popping in - my 1.5 sec pings to /. are back to 100ms as abovenet kicks in different connections (DNS still going through europe though :-)

    Of course when nuclear war hits it will take at least 3 hours to fix ....

    1. Contribute to K-rad, a KDE frontend to the GNU/rad utility to calculate radiation exposure.
    2. Check /. to see if the major backbone providers are doing ok.
    3. Run traceroutes to guess which places weren't hit. (Whoa, the Maldives are doing well. Doh! DC! It was all in vain!.)
    4. Try to avoid the BSoD (Brown Smoke of Death).
    5. Assist local disaster-relief organizations by migrating their proprietary birth/taxes/death-certificate database to an open one running Linux and MySQL, with a Perl/Tk frontend on X clients.
    6. Don't miss the opportunity to plug FS/OSS: the cathedral is centralized/vaporized, but the bazaar keeps moving forward.
    7. Reload /. incessantly in your hope to get a First Roast.
    8. Help out any hurt penguins you find while on the beach.
    9. Never forget the importance of an off-site backup!
    10. Yay! Finally, there are plenty of IPs available!
    11. Engage in a productive vi/Emacs flamewar.

  • dern it (Score:3)

    by Suydam ( 881 ) on Wednesday September 29, 1999 @11:21AM (#1649748) Homepage
    Why is it that this happens so frequently in the midwest (i live in Michigan).

    My company's T-1 line has been cut at least 3 times in the last 5 years.

    This kind of thing could be a MAJOR problem as the internet becomes more important to big companies. I mean right now, sadly, it's still a novelty to many of the people in the world. But can you imagine the hell we'd be in for if everything travelled over the 'net and someone cut a big trunk like this? Yikes.

  • ...that would explain why /. has been slow all day long. Good to know it's not our pipe.

    On a related topic, aren't lines like this clearly marked? I mean, how tough is it to call the number on the sign and see where the line is?

    Or did I miss something?

  • Re:RANT:And they worry about hackers (Score:1)

    by Anonymous Coward
    The first comment was mildly funny, but yours is just wrong. Of IQ is 80%+ genetic in most populations (which is true -- the lowest estimate that anyone reputable, i.e., everyone except rabid Marxists, suggest it 65-70%), then how is it statistically invalid? Methinks that you and industrial psych have not yet met. You will find him interesting, albeit inflexible.

    Actually, for crude measures like IQ, you can breed people just like dogs. And because you are almost always dealing with a far wider and (in the US) more diverse pool of genetic material, inbreeding is pretty damned hard, unless you are from a very closed section of society (I was thinking of Boston and parts of Connetticut, not Arkansas, actually). You get people no weirder than before, but, after a few generations, regressing to a far higher mean IQ. One of the more interesting parts of looking at the effects of post-WWII college education and population mobility is that this seems to have spontaneously created a breeding class of people that has actually seperated out a bit from the American population as a whole. Add grad school, the effect becomes even more profound. Add money, and you wind up with similar self-segregation. If you look at population patterns in the US, for the most part you are seeing (outside of some large US cities) a desegregation of neighborhoods and later (about a generation later) marraige patterns and a realignment solely along income lines. The thing that sorts is money and the filter seems to be higher education, and within that the better and worse schools and the amount of work beyond a BA/BS. This is going to have some interesting effects -- black intermarraige exceeds 30% in most urban areas and the children generally show no racial/ethnic preference (yeah -- flame away -- the control is the white mean, so sue me) so within 40 years or so you will see black people essentially start to disappear as a seperate and identifiable ethnic group.

    Pop stats and industrial psych are fun. Unless they aren't doing what you want them to, in which case I am sure that it can be hellish when reality refuses to cooperate.
  • After all, the utilities want YOU to call before you dig, but did THEY call the 1-800-Miss-Utility ????

    And looking at the Inter@ctive week article, ETIC of **ONE HOUR** ??? To repair fiber ??? Or should I say, to repair that much fiber ???

  • Incantation... (Score:1)

    by Anonymous Coward
    Backhoe, Backhoe, Digging Deep
    Make the Backbone Go To Sleep!

    Muhahaha
  • Why is it that IP trunks seem to be backhoed at the drop of a hat, while phone lines are somehow immune? Why doesn't the phone line between Boston and San Francisco go down?

    After the big MCI outage, it's time the phone companies put big red signs on OC-* pipes saying "NEVER cut through this pipe". I take it the gas company isn't about to pay for the lost and delayed packets.
  • hahaha i didn't know that.
    i wonder if they have a sign saying
    "dont smash into this wall or you will break the internet"
    and if they dont, they should.

    tyler
  • Looks like I'll have to give up my online Q3 game tonight. I wonder if I've got any hardcopies... I mean books... around. :-)

    What's all that wet stuff outside?!?

  • You are correct, this would not fall under criminal
    law, but the next best thing in the civil arena: Torts

    Any act, intentional or otherwise, that does "harm" to another.

    Oh, and don't forget: if someone kills you, not only can you charge
    them with murder, but sue them for wrongful death and get a wad of cash.

    "If you kill me, I'm gonna bleed your bank account dry, buster!"

  • Texas (Score:3)

    by scumdamn ( 82357 ) on Wednesday September 29, 1999 @11:27AM (#1649760)
    I don't know about y'all, but our connection doesn't seem to be any slower here. That poor bastard with the backhoe is luck he didn't cut that line around these here parts. We'd have his hide! Hell, Billy Bob'd probably string him up! Imagine not being able to get on EBay to see if he won the auction for the entire video tape library of the Dukes of Hazard. The horror!

    All opinions expressed with tongue firmly in cheek between the skoal and the ceegar.
  • Wouldn't you love to have 40gig of bandwith piped into your home? ;)
  • Is there a law against this sort of thing? I mean, can those who did it be arrested? They should be...

    And what about lawsuits? I know that if I ran an e-commerce site who's traffic was affected by this outage, I'd be pretty pissed about the possible loss of sales. Is it possible to sue the utility company that did this for something like that? Or do you just have to chalk it up to bad luck and move on?

  • Sudden realizations about the cold war (Score:4)

    by Sloppy ( 14984 ) on Wednesday September 29, 1999 @12:02PM (#1649765) Homepage Journal

    Now we know that the "vehicle" in "multiple independently targeted reentry vehicle" (MIRV) is actually a backhoe, rather than a nuclear warhead. I bet the guys who designed the 'Net to be nuke-tolerant are feeling pretty damned embarrassed right about now.

    Some people think the cold war was won by USA outspending USSR. But the real truth is that someone finally leaked that we were building bombs rather than just backhoes. Ivan's pants must have gotten pretty soiled at that revelation. Just think: all along, we totally misinterpreted what "We will bury you!" meant.


    ---
    Have a Sloppy day!

  • WOW! (Score:1)

    by kyanite ( 73015 )
    Now that's what I call Slashdotting!
    _________________________
    Words of Wisdom:
  • Uhoh! forgot the subtle irony tags!

  • Re:Nuke safe huh? (Score:3)

    by Ripp ( 17047 ) on Wednesday September 29, 1999 @12:07PM (#1649770) Journal
    Correct, but as the other poster mentions, it just gets re-routed. But stop and think for a minute.

    Power grids and telephone circuits can be affected the same way, take out one of those big "power towers" that traverse large spans, and a couple of remote stations, a few satellite uplinks, some telco switching stations/relay towers, and havoc *will* ensue. It doesn't matter if it *can* be re-routed, the resultant chaos and downtimes would cost probably millions. Cyberwarfare isn't about information, it's about $$$$$ lost when the infrastructures disappear.

    Then we'll be falling back to all the guys with their ham sets. ... --- ...

  • Kudos to the inventers of the internet which allows communications between computer networks to be automagically rerouted when a path is severed by a nuclear warhead or a trusty backhoe! :-)

    And I aint talking about no Al Gore either.

    Hasdi

  • That explains a mystery (Score:4)

    by hedgehog_uk ( 66749 ) on Wednesday September 29, 1999 @12:08PM (#1649772) Homepage
    I work in London (England) and earlier today a colleague told me that he had tried pinging a site in the US. He claimed that the packets were making it across the pond OK and then were being routed via Australia. We didn't believe him. Guess that I'll have to apologise tomorrow.

  • Name that CircuitID! (Score:3)

    by RISCy Business ( 27981 ) on Wednesday September 29, 1999 @12:09PM (#1649774) Homepage
    HA! I was WONDERING why things were slow to my ISP today! Those OC192's are probably the ones that Columbia Gas has pipelines running alongside, on I80. (Ha! You thought I wouldn't name the gas company!) IIRC, ICG laid the first OC48, and MCI/MFS as well as Sprint laid more, and more, and more. They're doing a LOT of construction along I80 from what I hear, so it's NOT surprising that a cut occured. Those lines are on a gas-pipeline right-of-way, and there are 5ESS Demarcation points along I80 for 'em.

    There've been previous fiber cuts that resulted in me passing a group of 15 MCI/WorldCom vans on my way home. I betcha if it's the link I suspect, they've got the onramp *packed* with all their techs trying to explain to the people staffing the tollbooths that they're with (insert-company) and they're in on an emergency call and they'll be getting right back off! *LAUGH!* Gotta love the Ohio Turnpickle, eh? :)

    -RISCy Business | Rabid unix guy, networking guru
  • (nostalgia)

    ahh... the days when archie.au was fed by a single T1

    (/nostalgia)
  • out of curiosity, why is mae-east, such an important box, located in a parking garage? amd who maintains it?
  • The conservation goes a long way part is VERY important! moderate this one up! if more people just used their archived pr0n/warez/mp3s, we'd all be a bit better off over the next week(or however long?) that it takes to get it back together....


    Dan
  • Disclaimer: I work for MIDS - I am posting this here as I think it may be of general interest.

    The MIDS Internet Average [miq.net] shows the effect of the fibre cut in the context of the Internet as a whole.

  • Actually, although these cables may follow major access routes, Sprint initially created their fiber network along the right-of-way of the Southern Pacific Railroad, as it was originally formed by the SP. (Hence the SPR in SPRINT as a previous poster noted; INT is probably for internal or something). Considering the SP doesn't run to the east coast, they either use roads or some other railroads tracks. As for how do I get there, there are trucks with rail wheels. But, overlay a rail map on an interstate/US/State highway map some time. You're likely to find that many of the major intercity rail lines correspond quite closely to the major highways, especially those portions of the US highway system which was build, for the most part, alongside the rail lines. Where they separate, at least along Sprints network, I bet the fiber follows the rail and not the highway.
  • Least your 14.4K can still connect with this slowdown....it connects, but it goes SOOOOOO SLOW....even for a 14.4k to the point where i have to look at .\ in the library. -btw....anyone have a 56K external for sale? buono@erols.com
  • Yep. It was quite an unpleasant experience. And to think, MRNET is lucky if they have 1 OC-192 worth of bandwidth across the state.
  • It's unfortunate this guy didn't slice open some underground power lines and zap himself and his backhoe ... He wouldn't be around to cut it again when it's time to harvest the potatoes.

    -Rob
  • Looks like those guys finished digging that hole right about lunchtime, eh? By far the biggest change I've seen on the ITR. Kind of funny how a backhoe could slow down time (internet time that is). Anyone have links (or know offhand :) to a total M/G/T|bs per day index on total Internet Traffic?
  • Marge: I really think this is a bad idea.
    Homer: Marge, I agree with you -- in theory. In theory, communism works. In theory.

    Episode 1F15, "Bart Gets an Elephant"
  • dopplar effect.

    BOFH, ah, how I miss thee.

    www.pushove.com new's for jack asss'
  • see, why the hell were these cables running through what was some chicken feed corn field in Ohio? Who is the idiot that didn't mark that there was a HUGE FRIGGIN' CORD underground that probably cost a small fortune? Oh, and how much is the replacement construction and infrastructure going to cost? Geez oh man. See, that's why I like Qwest. All their cords are along train track lines where they're the only ones that are allowed to dig and even if something is cut, they can just pull another cord through the piping.

    So lets see, in the past month or so, we've had a problem with MCI WorldCom and UUNet (correct me if I'm wrong) and now some backwater public works moron who was probably driving the backhoe with the blade down on his way to a coffee break killing some serious piping. Hmmm... what if we have a cataclysmic earthquake that splits North America in two by 3 inches... what would happen? Would all of that fiber optic cable stretch or would it pop? hmmm...

  • Re:IP vs Phone lines (Score:3)

    by Xenu ( 21845 ) on Wednesday September 29, 1999 @12:14PM (#1649789)
    Ma Bell put a lot of effort into designing a system that could route calls around congestion and equipment failure. Their routing system was very flexible and they had a NOC (Network Operations Center) to stay on top of problems. I'm not sure how much of that system has survived the breakup and deregulation of the Bell System.

    The data networking companies don't seem to be as concerned about reliability and availability. There are too many single points of failure. I've heard stories about the lack of excess/spare capacity in some big IP networks. The recent MCI Worldcom frame relay network failure was unforgivable. Some people were cut off for a week.

    You have to assume that lines will go out, equipment will fail and that software may not work properly.

  • Looks like /. is on the other side of the break from my location. This packet travelled all the way around the globe to get here. Anyway, there are only three possible companies responsible for this disaster. Cincinatti Gas and Electricity, Columbia Gas or East Ohio Gas. Each is equally likely, as Ohio has some sort of 'consumer choice' law that allows you to choose your provider (who is in turn responsible for your particular gas feed) My bet is on CG&E, as they seem to have the largest consumer base and the mose infrastructure under their control. When I figure out which one it is (I'm calling each now)
    I'll let you all know.. Let them feel the weight of /. .

  • Re:dern it (Score:2)

    by PD ( 9577 )
    A T1 does what, 1.6 Mbits/sec? The article said that the cut fiber could carry 80 GB/sec.

    So that makes it the equivalent of 50,000 T1 lines.

    Or 2.38 million 33.6K modems!

  • Actually, I put my money on either Columbia or East Ohio. First, I think East Ohio has the larger consumer base, but second, most of the lines, to my understanding, go through urban northern Ohio, and not through Central or Southern Ohio (of course there are some pop's here in Columbus as a result of Compuserve and Oarnet, perhaps I am wrong about this.) Second, consumer choice doesn't change the company who owns the lines going to your house, but the company who supplied the gas itself. Finally, if you've got a map of the lines through Ohio...I'd love to see it...I bet very little of the lines go through rural areas.
  • I think he has a 4xOC3 ring pass thru his basement that would be ~600Mb/s - he doesn't use all of it it - shares it with the university down the road - but sounds like his claim that he has more bandwidth than .au through his basement is about right

    BTW I still remember when e-mail and news to .au started up ..... bandwidth was OK but latency was low .... (something to do with how long it took to air-mail those mag tapes full of UUCP spool files across the pacific :-)

  • My own curiosity, which I'm sure will be my downfall someday, directs me to ask how on earth one actually fixes such an expensive piece of cable. I always fix ethernet wires by just re-stripping and connecting the RX and TX pairs anew; what's it like for fat pipes? Duct tape?

    It's probable that someone reading this has fixed a broken line before. Raise up... you know who you are.

  • Work (Score:1)

    by rizzo ( 21697 )
    My father, brother, and all of my paternal uncles are construction workers. My father is a backhoe operater, and the rest are laborers, of which I was every summer until I graduated from college.

    I wouldn't insult them by comparing the job we did to typing on a keyboard. Try standing on new, white concrete in 107-degree heat.

    I was lucky enough to get an education and out of the family rut, and I never take that for granted. The minute I even think of complaining about writing code for xx hours a day, I just remember the days I worked the same hours shoveling sand in pouring rain. If I came home complaining of carpal-tunnel, I'd get smacked in the back of the head, and rightfully so.

    PLUS I WAS at the edge of a hole when a backhoe cut a gas line. Luckily it didn't ignite, but the burst and hissing scared the shit out of me. Realizing I could have been killed in an instant at 18 because of a shitty job made me want to get out more. These men do risk their lives and often have no way out.

    A slowdown or inaccessibility of the Internet is an inconvenience, no one gets killed. We all need to get our priorities straight.
  • Well, this probably would not really work on a national scale, but I always have trouble with my phone lines, whenever we have landscaping or any sort of digging done, my phone lines get hit! so we dug down and layed a row of bricks accross them for the duration of the yard.... No more cuts so far ;)
  • The concept of a network based on quantized data with a possibility of all kinds of ugly things happening in transit like out of order delivery and quanta loss is "nuke-tolerant". The implementation of the Internet is not such a network, and for cost reasons will never be.

    That is not to say that there are no such implementations; there are. I've a cousin who admins one such network... her CSU/DSUs are designed to failover the signal pulse from EMP with opto-isolation (for those links that aren't optical to begin with). Her network is redundantly routed and cross-connected. Her software is fault tolerant like you wouldn't beleive. Considering who she works for I suspect most of the important nodes are sheilded quite well from EMP. The only element of the whole mess Joe Consumer can't buy off a shelf is the EMP sheilding.

    The only people who should be embarrased are people who think that The Internet meets DARPA's original specs - or even comes close.
  • I think the poor life of the construction/utility worker is a bit overblown in your description, but they definitely need a specific compensation they probably don't enjoy now: Free Internet (and not the banner-ad/NetZero kind). Consider it a preventive measure, by increasing their personal understanding of, and appreciation for, the data infrastructure.

    (And yes, I've done a bit of construction work myself...)

  • Don't worry.

    Good practice dictates that a reliable backup copy be made before deleting the original.

  • Knocking out a national broadcast network (Score:3)

    by orac2 ( 88688 ) on Wednesday September 29, 1999 @12:20PM (#1649807)
    Slighty off topic but a good tale: My father has been working for a national European broadcaster for a long time. Decades ago when they were setting up a new television and radio campus they had to run some data lines out to the techies' new buildings. He and his colleagues indicated the line where they wanted the backhoe to dig and went to lunch in the canteen. About 20 minutes later all the lights went out. Somebody had neglected to put down any markers and the backhoe had cut through the main power cable *after* where it was joined to the back-up generator. As national braodcasters in Europe also double as the goverment's Emergency Broadcast System this caused a lot of people way up the chain to get alarmed. While the cable was being repaired they had to bundle a crew down to the old emergency studio and transmitter in the General Post Office (Think Krusty the Clowns broadcast in The Simpsons).

  • Actually, my corporate office is in Texas and this is directly effecting our national WAN traffic. My Texas based IT department is catching shit because the people upsatirs seem to think this is our fault. Can someone with some insite remind me how a gopher can bring down legs of national WAN by chewing on the backbone cable and it can still be my fault . Oh yes, you can also add the list a fiber line in southern Florida that got hit by a construction crew this morning. Those Damn construction crews need to be more careful with my data. Days like today are not worth getting out of bed for.

  • There is now a thunder storm in that same area that the lines are down. The company that is fixing the lines has stated that a storm is keeping them from fixing the lines. They have no estimation of when it will be fixed.

  • themes.org (Score:3)

    by adraken ( 8869 ) on Wednesday September 29, 1999 @12:24PM (#1649811)

    just for all you to know, themes.org is being hosted on abovenet (one of the severely affected isps) and i can't currently reach abovenet's dns servers.

    sum: themes.org probably will be down until this is fixed. please be patient.

  • For what? Last I knew, something done without malicious intent wasn't a crime. If it had been a bunch of kids with wire cutters, you could probably get them for vandalism, but it was just some moron with a backhoe.

    People fuck up. You can't arrest them all, or everyone on the planet would be in jail.

    - A.P.
    --


    "One World, one Web, one Program" - Microsoft promotional ad

  • I work and play here at an Ohio university (Case Western Reserve if any of you care). Net access here is terrible. The fastest transfer I've managed has been a meager 120 kbps to anywhere in the world (considering we have OC-3 out, things are usually quite peppy).

    I really wish it were possible to see an accurate weather report on how such things affect the rest of the 'net. It's difficult to interpolate results from very few data points. Of course, the feasibility of such a system is quite difficult and prone to error.

    In either case, I'm advising everyone here to be nice to the 'net. There's no reason folks need to be downloading their pr0n, mp3z and off-site ftp installs of Linux during something like this. A little conservation can go a long way.

  • Actually, tcp/ip over ax.25 (ham radio data) is great; the 44.0.0.0/24 range is reserved only for amateur radio networks!

    As soon as I have enough money, I'll be buying a second tnc (terminal mode controller, more or less a modem, with packet "stuff" added on), to connect to my handheld 2m/70cm radio, and my newton, so that I can do wireless internet from my hand (and much cheaper than a cellphone after a few months).

    There are several great places to begin seeing what packet radio is all about, if anyone's interested you might drop by http://www.frostnet.advicom.net/chris/bookmarks/Ha m_Radio/ and take a look at the listed sites.

  • a bit of humor (Score:1)

    by Anonymous Coward
    I believe the conversation during the cut went something like this. "What the hell, I just got ahold of a whole buncha fishin line, in tha ground. Who'da barried some good ole fishin line." "I dunno, but it sure is purty, I think we needta pull soma it out and take it with us, whaddaya think cleetus?" "Sure, lets pull it out... wait, here comes the boss." "What in the holy hell? What did you 2 dimwits do, do you know what the hell that is?" "Fishin line, we wuz gonna take sum and use it tommorrow." "Oh hell, it's not fishing line, see, these orange marks mean that you don't dig through here. Thats 4 OC-192's, you idiots" "heh, whats that? sounds kinda technical." etc, etc, etc.... Who trains these guys?
  • As the world relies more on the 'net, the communications infistructure will have more and more seperate connections... redundency... So those four cables are a noticable problem now, but as there is more data, there will be more, seperate connections (yes each connection will also get faster, but still...)


    Next time something like this happens, there will be more other ways for the data to go, so the % slow down will be less... (I hope :-)

    --Ben

  • Gee. I don't recall anyone posting this kind of message about sysadmins.. I mean, how many of us have been paged or called at 3 AM to run in and reboot a server, router or something else. (And it's certainly not WORK that my PHB is doing at 3AM... :P )

    -Chris
  • I am not sure if it is just me, but why aren't these things marked? I still remember a few months ago when slashdot was shut down for several hours because of the friendly construction worker deciding to have a little too much fun with the tractor. If it isn't power, it the phone line. One solution can be found in the summer issue of 2600. The article is on ground based networks and it does make for interesting reading.
  • That's 9 kW total power to run multiple amplifiers on a very long cable. They insert an amplifier in the cable every N kilometers.

  • Re:So... (Score:1)

    by Anonymous Coward

    You'd be very surprised (and dismayed) to know just how ill organized and uninformed these companies are. I worked for a while within a very large company. It couldn't keep stuff straight amongst the engineers, splicers, assistant engineers, and the cubicle-hags in charge of posting. The engineers would lay down the plans, the splicers would do something different, the engineer assistants were supposed to tell the posters what modifications were made, and the posters were supposed to update the plans and put those out and nothing was ever done right or on time. Up-to-date records were (and undoubtedly still are) nonexistent. The only people that actually ever made any sense were some of the head splicers at the various local offices, but they were too busy so you'd get some idiot who would tell you information that was completely wrong (i.e., all they had to do was go to a site and look at how everything was rigged and then tell that information to the engineers so they could make up the plans; but they'd have things coming off of poles that weren't even there). Basically, the phone companies, at some level, don't know where everything is, so you can't expect other utilities to have any idea.

    It's sad, but true.

  • Re:So... (Score:1)

    by wx ( 96994 )
    i know in the state of missouri, and perhaps other places that if you cut any line, phone, water, gas, whatever, you pay a rather healthy fine..
  • I think we need a moderation category for "Beowulf Reference."

    The only thing I can't decide is if it should be +1 or -1. :)
  • The severed end of that fiber gives a whole new definition of piping output to /dev/null.

  • To inject a bit of Humor if I may. (Score:4)

    by Allnighterking ( 74212 ) on Wednesday September 29, 1999 @03:45PM (#1649839) Homepage
    About 8 years ago while working in Asia in a country to be left unamed (to protect the guilty). We had, had about 20 line cuts in a two week period. All within the same five mile stretch of Highway. It was because local farmers whereout backhoeing their field dikes to prepare for the next seasons planting as well as crews from Highway construction. We kept repairing the cuts in the fiber, putting larger and larger signs and underground tapes all in the local language (none of which helped they dug where they wanted too.) One of our local co-workers came up with a solution to the problem. We place signs along the stretch in ENGLISH only. The locals, and highway crews became so sure that this was some special secret government project that over the course of the next 6 months we had only one cut in our area of responsibility. *sigh* Kinda like my physics prof who said the only thing fibre optics would be good for is "hippy" lamps and nothing more.
  • As someone who did construction for a couple of years, I can tell you that while you might think there are maps that can tell you where lines are, they are almost always wrong. I've seen clearly marked lines on a map that were more than 10 feet from where they were supposed to be when you get in the ground and start digging. If the original diggers used flags to mark the lines, that makes it a little easier but even then it's like looking for a needle in a haystack but the needle has a string around it. A little better but not much.

  • They never said... (Score:3)

    by neuroid ( 6952 ) on Wednesday September 29, 1999 @12:31PM (#1649844)
    it was IDIOT proof. Just nuke proof.

  • Re:So... (Score:4)

    by netpuppy ( 77874 ) on Wednesday September 29, 1999 @11:34AM (#1649851) Homepage
    ok, I worked at an isp for a while, and here's what I saw.

    Apparently, people digging tend to call blue stakes or whoever when they are digging in areas that might have natural gas lines, 'cuz they'll explode if they screw up. Fiber lines, on the other hand, pose no such problem unless you're standing in water when you hit one (there's a lot of power going through to feed all the repeaters on a long-haul circuit). So utilities and construction workers don't tend to worry about calling any utilities to find out if fiber is buried, because they really never see the effects. I believe they are legally liable, though. Anyone know if someone has ever been brought to court for cutting fiber??

  • Whew! (Score:5)

    by Cuthalion ( 65550 ) on Wednesday September 29, 1999 @11:35AM (#1649852) Homepage
    Good thing there wasn't anyone around there smoking a cigarette. What with all those loose bits sloshing around, the slightest spark could have set off that 'internet explosion' people keep telling me about.

  • Nuke safe huh? (Score:4)

    by Hobbex ( 41473 ) on Wednesday September 29, 1999 @11:39AM (#1649861)

    I would say that this goes to show the utter bullshit that is the whole cyber terrorism thing. Why spend billions of dollars trying to police imaginary squads of crackers set to destroy our information infrastructure, when a couple of idiots with shovels can create major mayhem like this?

    I wonder what an organized group of wire cutters who did a little bit of research on their targets could accomplish. I have a feeling it wouldn't be pretty.

    I can't say I noticed anything myself (the net has been dog slow for me as long as I can remember, so), but if a small event like this can cause major problems, then the Internet is definetly closing up on critical mass....


    -
    /. is like a steer's horns, a point here, a point there and a lot of bull in between.

  • Fiber Route Engineering, Cable Location, et al. (Score:5)

    by Anonymous Coward on Wednesday September 29, 1999 @12:35PM (#1649870)
    Well, the powering issue being talked about isn't exactly what would be called "common". Most fiber in the ground today has no current-carrying conductors. True, it has a metallic conductor or two in it, and in some cases, a metallic sheath or armour, but carries no exectricity. The metal conductor is used exclusively to locate the cable with inductive/RF location gear. When it comes to re-generation of fiber "signals", byt simple amplification, or a full teardown of the bitstream with drop/add functionality, it's usually done at a special location (in-ground vault, above ground fiber hut, whatever) with external power. That's power supplied by a power company, with backup power. Since fiber traffic routinely travels 50+km without intervention at all, it's a total waste of resources and cable to lay a power cable for equipment. Now, all said, there are exceptions where there is no power readily available, like deep-sea cables, deserts, and such where there may be a power cable present in places. But in general, not present. Now, back to a little more on topic :) As for location, depending on the area, this can be tricky business, big time. In dense metropolitan areas, there can quite literally be cables a foot or less apart "horizontally" as well as several layers deep "vertically". Location of cables isn't always as precise as it ought to be, especially when the phone companies records don't reflect actual construction oddities, like perhaps a four foot fiber "loop" at the base of a pedestal. Well, when the person locating cable gets within a couple feet of a pedestal, you'd just assume it goes right in, right? Wrong :) And another company plans to put a pedestal right next to the existing one, and, well, it's time to break out the Scotch-Lok or quick-fix fiber kit :) Granted, most cable and fiber cuts happen out of pure ignorance or stupidity. Farmer Joe is out in the field fixing a drainage tile with his tractor and doesn't give a single thought to what else may be in the ditch with his tile. Where did those kits go again? :) And as a final note on the buried cable thing, nowadays, it's becoming common, almost required, practice to bury a brightly colored plastic ribbon above the cable being installed. Usually yellow or orange, it's really easy to see against black dirt, and would hopefully be seen before the contractor hits the cable itself. Doesn't do much good in the case of boring or knifing cable in though. All of this is just general information. The cut out east could have been done in one of a dozen or more other common contractor cable goofs that I won't even try to speculate on. Either way, it boils down to this - contractor started digging without requesting a locate, or if the cable was properly located, and the contractor either mis-read the location markings, or ignored them, that contractor had better hope he has a good insurance company. If the cable was not located at all, and a locate request was made and recorded in a timely manner, usually 24 to 48 hours before digging, well it's the telco's tough luck. You know, it's a small wonder that things work as well as they do considering the unimaginable number of perils out there, backhoe's included :) Sorry this is so long! FBG the AC
  • The redundancy and fault tolerance was a design principle of Arpanet because it was intended to be suitable for military communications. If the system fails during a war then people die, so high reliability was needed at any cost.

    But now the Internet is a commercial enterprise, and failure is now an option. At worst, some large corporations lose their VPNs and have to prioritize and pick up the phone again. For most of the net it's just a matter of losing porn, IRC, and MP3.

    Now if you're a backbone maintainer, do you double your capital costs to achieve more than minimal redundancy just to give the public a warm fuzzy feeling? Or do you maintain the least expensive network you can without losing customers? Market forces will drive the QOS on the net to the lowest tolerable level, and for now people will tolerate a lot of net failure because their lives and livelihoods don't completely depend on it...yet.

  • Given, they have a tough job. So what, they get paid to work like dogs, they aren't a chain gang for crying out loud.

    Around here, they put commercials on tv all the time saying, "before you dig, call Miss Utility" and flash an 800 number. So you mean all us yokels have to check in with the utility folks but the utility folks don't have to check themselves?

    Sure accidents happen, but I can't see a public utility diggin around and not knowing where their own lines are. Sheesh. :-)

    The Divine Creatrix in a Mortal Shell that stays Crunchy in Milk
  • Let's suppose that a nuclear exchange destroys the US. (We can also suppose it destroys Russia or China, but they're insignificant because they don't have much bandwidth to begin with. :)) If you're going to nitpick the nuclear war example, let's just suppose that for some generic reason, the available bandwidth decreased significantly and rapidly.

    Now, we can't count on the users cutting down on their bandwidth use conscientiously. How, then, can we keep the critical services running? For a start, we need to define "critical services". I'll say that the greater the ratio of content to bytes, the higher the priority of a service. The only practical way to filter packets by service is to filter by port. You can run a filtered service on any old port you want, but the goal is not to prohibit services so much as drastically reduce the bandwidth used so that the network remains usable.

    DNS, for example, would have a very high priority, and be one of the last services to drop. Without DNS, the network becomes significantly less usable. The services designed for text communication also would have a high priority: smtp and the assorted email services (no attachments), nntp (again, no attachments), finger, time services, gopher, and the like. Even http might be allowed through, but filtered by mime type (text/plain, text/html, x-www-form-unencoded, etc.).

    Still, there would be a significant drop in the usefulness of the network. We need more bandwidth than we need to ensure reliability. Make bandwidth, not war! :)

  • Re:another reason to favor Qwest (Score:3)

    by Allnighterking ( 74212 ) on Wednesday September 29, 1999 @04:07PM (#1649878) Homepage
    Actually these cables usually follow major access roads whenever possible. This allows for access to the lines regardless of weather conditions. (the mud in an Ohio cornfield can get 2 feet deep after a really good rain) Next 4 to 6 feet above the cable they bury a bright orange tape that warns about cable below. Hopefully this can be seen and Hopefully it doesn't settle out of alignment with the cable below. Third most cable cuts I've seen were not the result of direct contanct with the cable by the backhoe but rather the result of mud and rock slides/shifts due to digging near them. Cables aren't and can't be that strong. More so pulling a cable isn't as easy as you make it sound. This isn't a piece of thread It can weigh several tons (tonnes) and requires heavy equipment to pull it through a pipe. Finally one of the problems with using train tracks is how in the hell do I get a truck back where the cable is? Finally glass doesn't stretch.
  • ...what explanation should we use for /.'s slowness every other day?

    - A.P.
    --


    "One World, one Web, one Program" - Microsoft promotional ad

  • "internet explosion" is not a thing, its a magic phrase. It works like this; you're giving a presentation to stupid people with lots of money who keep hearing about this internet thing. Prefix our magic phrase with the word "tapping" and you'll invoke the grand money gods. You will suddenly find yourself with ridiculous amounts of cash. I suggest the next available flight to somewhere with no diplomatic ties to the US.

  • Bah, the new technology... (Score:3)

    by Rombuu ( 22914 ) on Wednesday September 29, 1999 @11:40AM (#1649888)
    We never had this problem back in the days of good ol' POTS service. Good old copper wire never cut cut or went dolkjsd;flkh#&&(

    NO CARRIER

  • Re:Doh (Score:2)

    by Serk ( 17156 )
    Hmmm, they could always pack the fiber bundles inside of high explosives. Yeah, it's still get cut another time or two, but word would spread among the back-hoe operators REALLY quick to NOT mess with them fiber cables.

    (Yes, I'm joking...)

  • Packets are getting rerouted .... everyone suffers (Score:3)

    by taniwha ( 70410 ) on Wednesday September 29, 1999 @11:48AM (#1649897) Homepage Journal
    Well - that would explain this:

    1 198.182.167.17 (198.182.167.17) 0.628 ms 0.564 ms 0.532 ms
    2 adsl-63-194-218-254.dsl.snfc21.pacbell.net (63.194.218.254) 26.688 ms 29.433 ms 15.868 ms
    3 core1-fe4-1-0.snfc21.pbi.net (206.171.134.209) 13.394 ms 13.701 ms 15.957 ms
    4 gsr1-g1-0.snfc21.pbi.net (209.232.130.20) 16.098 ms 13.846 ms 15.215 ms
    5 sfra1sr3-so-1-1-1-0.ca.us.prserv.net (165.87.161.74) 16.676 ms 16.208 ms 15.202 ms
    6 sfra1sr2-11-0-0.ca.us.prserv.net (165.87.13.17) 16.684 ms 16.286 ms 15.226 ms
    7 above-advantis-ds3.sjc.above.net (216.200.0.81) 21.833 ms 24.418 ms 29.271 ms
    8 core1-core4-oc3.sjc.above.net (216.200.0.85) 25.864 ms 23.882 ms 17.173 ms
    9 core2-core1.sjc.above.net (209.133.31.110) 47.734 ms 39.032 ms 39.107 ms
    10 main2-core2.sjc.above.net (207.126.96.138) 44.072 ms 40.088 ms 40.069 ms
    11 core2-main2.sjc.above.net (207.126.96.137) 42.369 ms 44.488 ms 46.094 ms
    12 * * core3-core2-oc3.iad.above.net (209.249.203.65) 590.011 ms
    13 abov-core1-mae-east.netaxs.com (209.249.119.234) 695.855 ms 721.634 ms 848.819 ms
    14 dn-netaxs-gw.dc-core.h5-0-45M.netaxs.net (207.106.127.94) 1125.874 ms 1282.923 ms 1408.877 ms
    15 h900.ca2.wdc.dn.net (209.207.190.5) 1263.551 ms 1250.176 ms 1252.700 ms
    16 209.207.174.23 (209.207.174.23) 1234.975 ms 1256.689 ms 1208.096 ms

    and worse yet DNS lookups from here going thru Vienna :-(

    But remember folks there's redundancy in the backbone routing but when something big goes down
    everyone else gets to suffer as the traffic
    gets piled on top of their usual connections.

    If there's a lot of traffic going thru Europe I
    bet they're getting really steamed over there
  • When you are aware of the econonmic backlash from cutting through a fat pipe, adrenaline is the first thing that hits :-) Then you start worrying about your new career "would you like fries with that?". Then you leap up and go tearing around the building looking for the laser splicer kit.

    Laser splicers are expensive, so they are kept in a locked cabinet. Fire axes are your friend, and handles come right off with the first few blows. Then a quick rush back to the site of the break. It was in the ingress vault, carrying a few dozen fibre cables to the head end equipment, so there isn't a lot of room to manoeuvre, and the closest electrical outlet is 2 extension cords away. Breaks never happen anyplace nice, like in a well lighted place with a table nearby. Breaks are always at the bottom of a sewage ditch, or in a crawlspace or under the ocean. Murphy has a law about this.

    Time lost, about 7 minutes until repairs started. Time to repair is about 12 minutes, if you are good. 20 if your hands are shaking and the sweat is pouring off your brow.

    The fat pipes, an OC-12 in this case, are actually very small mono-mode fibre optic threads, less than a millimetre in diameter. They are inside a thin plastic sheath, wrapped in some other protective materials, but those protective materials are stripped back inside of the vaults, so eejits can drop some heavy equipment on them, and slice them right through without any resistance at all.

    To re-splice a fibre requires that the protective sheath be stripped back a few inches on either side of the break, then the fibre has to be cleaned with alcohol and other contaminant free cleaners so there are no impurities sitting on the outside of the fibre.

    Then you have to put the ends into a cleaver, which looks like an old film splicer or a paper shear. The fibre has to have a nice clean break on the end, so the ends can be butted against each other before fusing with almost no loss of signal. Most backhoe induced breaks shear the fibre at a long angle, so you lose an inch or so. That is why there are always loops of extra fibre every so often, for slack.

    Then you put the two ends into the fusion unit, which hold the ends together. Then you hit the button, and a powerful laser melts the ends slightly so they flow together, then cool into a new, not quite perfect optical path.

    Then you have to re-cover the exposed fibre carefully with a new sheath, then wrap the splice with some more protective tape, and THEN you can wrap the whole area in duct (gaffers) tape :-)

    Then comes the paperwork to document the splice, the new losses introduced, the higher BER, etc.

    And if you are lucky, nobody noticed the break since it was late on a saturday evening and only AOLers were affected for about 20 minutes. I love routers and backup routes.

    the AC

    [the names, places, dates have all been written in the third person so as to not identify the guilty party or service provider affected. No packets were hurt during the writing of this post. Stunt doubles were used for the dangerous cabinet opening scene :-]
  • Yeah, my traceroute to slashdot last night went through australia. But it was late, and I didn't bother saving the trace. Just another artifact of the internet :-)

    the AC
  • Unless the lines were in some serious sheathing, most backhoes wouldn't even twitch going through it. If the operator wasn't watching the hole close (most tend to be more concerned about where the bucket's going) he'd probably never notice it.

    As an aside, I once worked with a gentleman who could knock your hat off with the bucket of his hoe (I held still, most he tried it on wouldn't). I saw him pick up a boiled egg in the bucket without breaking the shell. That kind of trick requires a level of skill not equalled by most airplane pilots. Those machines are *not* that easy to operate, and idiots don't tend be allowed on them.

  • Re:So... (Score:5)

    by netpuppy ( 77874 ) on Wednesday September 29, 1999 @11:48AM (#1649918) Homepage
    From Wired 4.12 ... article called Mother Earth Mother Board ... url at http://www.wired.com/wired/archive/4.12/ffglass.ht ml

    The signal coming down the FLAG cable passes through the doped fiber and causes it to lase, i.e., the excited electrons drop back down to a lower energy level, emitting light that is coherent with the incoming signal - which is to say that it is an exact copy of the incoming signal, except more powerful.

    The amplifiers need power - up to 10,000 volts DC, at 0.9 amperes. Since public 10,000-volt outlets are few and far between on the bottom of the ocean, this power must be delivered down the same cable that carries the fibers. The cable, therefore, consists of an inner core of four optical fibers, coated with plastic jackets of different colors so that the people at opposite ends can tell which is which, plus a thin copper wire that is used for test purposes. The total thickness of these elements taken together is comparable to a pencil lead; they are contained within a transparent plastic tube. Surrounding this tube is a sheath consisting of three steel segments designed so that they interlock and form a circular jacket. Around that is a layer of about 20 steel "strength wires" - each perhaps 2 mm in diameter - that wrap around the core in a steep helix. Around the strength wires goes a copper tube that serves as the conductor for the 10,000-volt power feed. Only one conductor is needed because the ocean serves as the ground wire. This tube also is watertight and so performs the additional function of protecting the cable's innards. It then is surrounded by polyethylene insulation to a total thickness of about an inch. To protect it from the rigors of shipment and laying, the entire cable is clothed in good old-fashioned tarred jute, although jute nowadays is made from plastic, not hemp.

  • Check out the Internet traffic report (Score:5)

    by Oates ( 18921 ) on Wednesday September 29, 1999 @11:48AM (#1649919) Homepage
    I checked out the Internet traffic report to see what kind of effect this harmless mistake had. North American traffic statistics [internettr...report.com]

    Keen! Can you spot the time the big bad backhoe operator cut the cord?

  • Hey, cut them some slack.. (Score:4)

    by Pyr ( 18277 ) on Wednesday September 29, 1999 @11:49AM (#1649920) Homepage
    Construction/Utility repair guys don't have an easy life. They're the ones out there in the middle of the night when your power line has gone down, or the ones fixing that broken sewer pipe, or the ones who make sure you have water. Near here they're doing major construction over on Grand Avenue, and to avoid traffic they have to do it in the middle of the night. They accidentally cut off a hunk of phone lines and about 1000 people had no phone service, but that happens pretty rarely.. I'd like to see YOU guys doing hard physical labor all night or all day trying to avoid speeding cars and a maze of pipes and cables and never screw up occasionally.
  • I wonder if this has anything to do with the slow connection rate that I get when I try and telnet into my webserver that sits in the next town over from where I work. After all, when I do a traceroute, I get shipped through machines in Chicago, SanDiego, and other exotic places. And I'm in South Carolina. A month or two ago, there was a slashjot about some guys trying to map the Internet. I think that it would be easier to map a world covered in spaghettti noodles, because that's what the Internet looks like to me!
    Brad Johnson
    Advisory Editor

Slashdot Top Deals

Work without a vision is slavery, Vision without work is a pipe dream, But vision with work is the hope of the world.

Close