Slashdot is powered by your submissions, so send in your scoop


Forgot your password?
It's funny.  Laugh.

The Most Dangerous Server Rooms 463

Ymerej writes "The Register has an article on dangerous server rooms. Have you seen worse?" Perhaps The Register would like a picture of my desk if they really want to be scared.
This discussion has been archived. No new comments can be posted.

The Most Dangerous Server Rooms

Comments Filter:
  • by bplipschitz ( 265300 ) on Thursday October 24, 2002 @05:16PM (#4525164)
    . . .where the city rats are bigger than the IT guys.

    And they carry card keys.

  • Now that's how you ensure job security!
  • c'mon (Score:3, Funny)

    by Archfeld ( 6757 ) <> on Thursday October 24, 2002 @05:19PM (#4525186) Journal
    calling application programmers rats is insulting to rats, Pls refer to them by their proper designation...(L)Users :)
    • then accountants with key cards could not be rats, just simply bean-counters, (another demographic group to toy with MWAHAHAHA! )

      What's your (l)username?...ok...**clickety click** Why thank you for increasing our IT budget by 1 billion$ i'm afraid you'll have to go. Do me a favor and check if the pover is on, just stick a piece of metal in the mains... You won't feel a thing :)
      • LOL I miss the BOFH :) Things just aren't the same these days...What with ergonomics and windows and things. It is almost as if the company is trying to hint that people have rights and shouldn't be treated as mere playthings for my amusement. Imagine that :)
  • Server room (Score:4, Funny)

    by Anonymous Coward on Thursday October 24, 2002 @05:19PM (#4525187)
    My server room is so bad not even the rats will go in there
  • by Anonymous Coward on Thursday October 24, 2002 @05:19PM (#4525190)
    For example, statistics show that people who work in server room almost never catch any STDs. I wonder why that is...
    • Statistics also show that "server room affairs" by non technical personal, are responsible for more than half of office place STD transmission.
      The "photo copy room trysts" follow in a close second place, while the "boss's office boinkings" are becoming more uncommon as more CEOs are of the female gender.
    • by 4of12 ( 97621 ) on Thursday October 24, 2002 @05:44PM (#4525351) Homepage Journal

      For example, statistics show that people who work in server room almost never catch any STDs. I wonder why that is...]

      Because the condoms actually work as advertised?

  • There are more (Score:5, Informative)

    by cheezycrust ( 138235 ) on Thursday October 24, 2002 @05:19PM (#4525192)

    Actually, this is the third article about the subject.

  • by elbarsal ( 232181 ) on Thursday October 24, 2002 @05:22PM (#4525210)
    The pictures they show are pretty entertaining, but it's not real danger until you toss in some higher voltages (480vAC anyone?)

    One of my favourites was actually sitting inside a motor control center, with all sorts of high voltage DC motor starters right behind the main computer terminal. Don't lean back.


    • You want dangerous, well the only fire prevention we have for our 23 servers, is one halon extinguisher. So if the room goes up, we can save it, as long as we don't want any oxygen...
      • by Telastyn ( 206146 ) on Thursday October 24, 2002 @05:44PM (#4525355)
        Some of the more complex and heafty NOC's I've heard about actually have halon, or similarly based fire prevention systems for the entire room. Auto sealing doors and the works. The old "you have 20 seconds to exit the room" plot device brought to life.
        • These are actually pretty common. I remember the university's data center was like this, there were 3 large red buttons that would temporarily stop the halon release so that you could get out or do something to the servers before the room was flooded. My current employers datacenter sadly had only high temp overhead sprinklers, by the time they go off its more about saving the building as the servers are already slag.
        • by Pfhor ( 40220 ) on Thursday October 24, 2002 @08:17PM (#4526303) Homepage
          Actually my college still has them, and they work.

          My boss was telling me about how a guy working in the AC system kicked up some dust and it triggered the Halon system. A voice came on to announce they had 15 seconds to get out of the room before it would be deployed. My boss of course hits the button and stops the countdown. But he lets ago, apperantly you have to hold the button until someone can come by and turn off the system. So he and the network admin go diving out of the room just as the halon is released.

          There are other labs on campus that have Halon warning labels on them also, and I wouldn't dare try to check if its true.
      • by Cramer ( 69040 ) on Thursday October 24, 2002 @05:44PM (#4525358) Homepage
        Well, that's sorta the whole point of halon (and thus, why isn't used anymore.)
        • by Yobgod Ababua ( 68687 ) on Thursday October 24, 2002 @06:24PM (#4525574)
          Actually, it isn't used any more because it was found to be ozone-depleting and banned by international agreement.

          http :// /wn15-208.html

          Several of the ozone-friendly replacements (I forget their unfriendly alphanumeric designations) still work by sucking all of the oxygen out ofthe room.
        • by Anonymous Coward on Thursday October 24, 2002 @06:59PM (#4525838)
          Well, that's sorta the whole point of halon (and thus, why isn't used anymore.)

          Having thoroughly studied the subject, umm, no, you're wrong.

          Halon is no longer used because:
          a)it was poisonous
          b)it was ozone depleting

          The number one cause of death in server room fires is toxic smoke inhalation from burning plastic/paint/wire insulation/circuit board components/etc. Why? People go in to rescue backup tapes, and are rather quickly incapacitated when they inhale the smallest amount of toxic fumes. Folks- NEVER ENTER A BURNING ROOM WITH A FIRE. PERIOD. Backup tapes are NEVER more important than your life. People think "oh, I'll hold my breath." They don't think about how far it may be, or they may get disoriented in the smoke..and they think "oh, just a little breath, i need to breathe." That gets you coughing. Then you die. Smoke inhalation is actually(I believe) the number one cause of death in ALL fires.

          Let's not even start about what opening a door into a hot, mostly sealed room letting in a ton of oxygen does.

          My "Blue Book" (computer security, mostly focuses on physical) says to NEVER store backup tapes in the same room as server equipment, PERIOD, because the temptation to rescue them is too great.

          CO2 is the agent that suffocates that case, yes, the point is to remove as much oxygen from the room, and yes, you will suffocate very quickly.

          FM200 is the most common substitute and is fairly harmless, but works on pushing oxygen from the room, and because it is much heavier than air, requires very good sealing of the room in order to work properly. It is next to useless on fires high up in the room unless the system is very well designed.

          An alternative is INTERGEN, which is 100% harmless and is mixed specifically to make your body naturally breathe faster, which is a good thing if smoke isn't really a problem and you just need to keep from suffocating. It also works mostly on rapid temperature change and chemically interfering with combustion(not by removing oxygen), and its weight is much closer to natural air, so it doesn't sink.

          Thus, the server room doesn't have to be sealed(Halon and FM200 both require it), but the amount of extinguishing gas is greater and piping is slightly different.

          In server rooms of sufficient size that use Halon, CO2, etc... SCBA units are usually recommended if not required. Interrupt switches and delays are very common as well, although both are designed mostly with reducing false discharges, which can cost $5-10k per 1,000 square feet.

          More and more commonly, dry-pipe water systems are being used, particularly in 24x7x365 managed buildings. The idea is that if the fire is small, an alarm goes off and a countdown starts. On site staff can take appropriate action(cut power, run in with a fire extinguisher, that sort of thing) or, if it's bad, let the system come on. They're usually zoned triggered and fogging nozzles are used to minimize equipment damage.

          Unfortunately, for your average 1,000 sq ft room, you're looking at about $30k to $50k, not including modifications to the room to get it sealed. Dry pipe water is less expensive, but one false alarm could cost you 10 times the system's cost in equipment alone.
      • I've been in two datacenters where there were regularly spaced 'emergency air supply' masks, right next to the regularly spaced fire extinguishers.

        Not sure how well they would work in a room full of smoke and halon, but I suppose you could run from station to station, and then somehow pry open the massive firedoor...
    • by dattaway ( 3088 ) on Thursday October 24, 2002 @05:43PM (#4525348) Homepage Journal
      I work in industrial maintenance and the most interesting electrical problems happen when the weather changes.

      120VAC isn't too bad. Connections soaked in water might survive for a while until the corrosion finally breaks it down, melts the wirenuts, etc. Getting shocked by it isn't enough to blow your fingertips off. 240 volts is usually just 110 volts split into two phases, so it doesn't present any worse of a problem. 480 volts is another story...

      480 volts gets interesting when the humidity rises and gets absorbed by the dust surrounding breakers and other switching components. Often, it will flash across the phases, vaporizing the debris, and mysteriously tripping the breaker. No one will figure this out until they happen to take a close look at the wiring, and the humidy from their breath will illuminate the brightest flash they have ever seen in their lives.
      • by GigsVT ( 208848 ) on Thursday October 24, 2002 @07:00PM (#4525854) Journal
        Heh, I've got a 480v story for you. I work in IT for a manufacturing company. We had an apparently faulty air conditioner take out most of the plant, which is no small feat.

        First, a little background info. It's an old building. The electrical is a mess of old and new circuits, some three phase 480 delta, some 240/120 single phase, and one major branch circuit even with a high leg. The former maintenence manager was of the mind to "get things working", rather than "get things right". After he was fired, I started helping out a little with the maintenence staff when emergency things came up, since I know a little about electronics, mostly to help them with computerized and digital control systems.

        Anyway, from the street, there are 5000 amp fuses, huge suckers, then 1200 amp fuses on a few main branches. From there to a 1200 amp panel breaker for a major section of the plant (the one that the server room is on), along with most of the manufacturing. In that 1200 amp panel there is a 250 amp 3 phase breaker for the air conditioner. This is all 480v delta 3 phase.

        Somehow, that air conditioner breaker failed. It vaporized part of the busbar, tripped the 1200 amp panel breaker, and blew the 1200 amp fuse for one of the phases, leave us down a phase. For the benefit of those who do not know, 3 phase motors running with one dropped phase tend to burn up... quickly! Ideally there is an thermal overload circuit to shut them down before that happens, but that doesn't always work. So bang.. the lights are off, and motors start to burn up in various places around the plant.

        Once the maintenence staff figures out what is going on, that we are down a phase, they throw the mains on the service entrance panel for that 1200 amp branch. All seems to be good. We just need to replace that 1200 amp fuse and the faulty breaker right?

        Heh. Well, it happened that we didn't have any spare 1200 amp fuses. A 1200 amp fuse isn't something you can run down to your hardware store and get. We send an employee to the next town where a store has exactly three of them in stock. $400 each. We tell him to buy all three. He comes back with one.

        We replace the fuse, and the maintenence staff replaces the breaker. Upon reenergizing the circuit everything seems fine... until they go to put the protective metal cover back on.

        The panel literally explodes. I wasn't in the room at the time thankfully, but the guys that were there say is was bright, loud, and scary. Apparently what happened was pieces of molten busbar had dripped near the bottom breaker in the panel and were close to shorting out the phases. The slight movement caused by putting the cover back on jarred the chunks of metal and shorted out the phases.

        So... the 1200 amp panel breaker trips.... but not fast enough to save the upstream 1200 amp fuses near the service entrance. The ones we didn't have spares for. Again. And now it blew all three of them. And the store only had two in stock.

        So we get back on the phone. We find another store that has two in stock, so we send an employee out to get all five, from both stores. He gets it right this time.

        We finally replace the three fuses, triple check EVERYTHING, and throw all the breakers back on. We had sent all the employees home hours before... they couldn't do anything without power. But we are finally up... nearly 6 hours later.

        Needless to say, some things have changed as a result of this, and it really underscored why the former maintenence manager was fired. We called the electrical engineering firm that had most recently surveyed our power systems, and had them run some more short circuit computer simulations, things like that.

        Upon reading their report, I learn that our service panel has a ground fault interrupter, but it was turned all the way up to 1500 amps to prevent nusience trips, after it tripped several times due to our really bad "normal" phase imbalances.

        Things are definitely improving, and we are much safer now then we ever were. It goes to show how one bad maintenence manager with a reign of terror, and a long tenure, can really screw things up though. I compare it to a programmer that never comments their code, and uses lots of goto statements, only the stakes are much higher.
        • by dattaway ( 3088 ) on Thursday October 24, 2002 @07:36PM (#4526077) Homepage Journal
          Ouch. Every place I worked at had 14.4KV branched out to several substations fused at 90A, which was good for at least 1.6MW, which branched out into even smaller substations. A failure at one point was rarely noticed elsewhere. Except for the occasional exploding capacitor on the pole outside. Worked great for years with few surprises...

          One day I would find out why the 14.4KV fiberglass-epoxy reinforced fuses had mufflers installed on them. Remember that electrical current resists change. If the circuit breaks, the magnetic field surrounding the current collapses and increases the voltage until it goes *somewhere.* Each substation transformer was the magnetic equivalent to a ten foot tall capacitor. Well, if the fuse blows, the the remaining energy in the transformer's magnetic field immediately collapses (the magnetic equivalent of a ten foot tall capacitor) and detonates the fuse filament. This muffler vents this energy harmlessly into the substation as heat without blowing the panels off.

          One day, when turning back on the power from vacation, we would find one of these fuses didn't have its muffler installed... And we would learn how things would *seem* to work on two phases.

          We would try to install more fuses without the muffler on that phase. The magnetic field was strong enough to pop the fuse out of its holder when the switch was thrown. A wire tie solved the problem while parts were being ordered.

          Moral of the story: if you work on high voltage equipment, always leave it as you found it.
  • Too funny. (Score:5, Funny)

    by unicron ( 20286 ) < minus cat> on Thursday October 24, 2002 @05:22PM (#4525211) Homepage
    It's all about removing floor tiles and then forgetting to set up warning cones. The clearance between our tiles and the concrete floor underneath is a good 4 or 5 feet; I would not want to fall stiff-legged into that.

    Interesting side note: apparently finding high-priced Cisco gear not connected to anything is not that uncommon. I've also heard horror stories of guy that traced a cable my hand(toner was on the fritz) that looped 4 times around the data center but wasn't hooked up to shit, on either end. Took him an entire afternoon.
    • by SquadBoy ( 167263 ) on Thursday October 24, 2002 @05:36PM (#4525299) Homepage Journal
      Oh yea. We have several Nokia IPSO boxes that are hooked to *nothing* and more than a few Motorola routers hooked to nada. Now on the subject of floor tiles. It appears that before I started to work here they got a "good" price on raised flooring. At least two people including myself have fallen when the stand bent while we were standing on it. Also when they first moved into the building it seems that a sewer backed up and flooded under the raised floor, to this day I will still find how shall I say debris under there. So I don't touch my face and wash my hands with *hot* water and soap after everytime I have to go under the floor.
    • by IIRCAFAIKIANAL ( 572786 ) on Thursday October 24, 2002 @05:38PM (#4525312) Journal
      This reminds me of a humorous story:

      The Magic Switch []

      I can't remember the author (GLS) but if you google you'll probably find a more original version.
  • by Longinus ( 601448 ) on Thursday October 24, 2002 @05:22PM (#4525216) Homepage
    Check it out here [].

    This is by far to custom case I've ever seen.
    Look a bit dangerous though ;-)
  • I'm a slob; I admit it. Even if I didn't, it would be obvious from the way I dress and the state of my desk. But at my job, I am cleaning up the spaghetti mess in the ceiling and trying to lay the wire cleanly from patch panels - switches.

    Documenting connections has a real payoff in troubleshooting. But doing stuff on aesthetic grounds is a harder sell. I have a gut sense that a clean layout is important even if you know the destination both ends of a wire whose middle runs through a snarl. Here's what I came up with:
    my version of the community policing broken windows theory.

    It's psychologically harder to do slipshod, shoddy work if everything around you has been done well. And it's hard to do a proper job if everything else is slipshod. As a matter of housemate politics, it's easier to leave the nth dirty dish in the sink than the first. You are only adding an increment, not changing state.

    Doing the Right Thing is contagious. At least, it is among folks I care to work with. Doing the Wrong Thing is catching, too. Morale is higher and people challenge themselves more at a shop that is run well.

    That's how I pitched it, and my boss bought it.
    • by _ph1ux_ ( 216706 ) on Thursday October 24, 2002 @06:16PM (#4525516)
      "...But doing stuff on aesthetic grounds is a harder sell..."

      This is not always true!

      If you have a company that you are trying to get new customers and investors to come on board - and you need to show them your technology, often times they like to come over and take tours of your datacenter - HQ - IT Dept or office in general.

      Believe me - the asthetics of the cables that make your network have a very large impact on how people can view your company.

      For example, this picture [] shows that this particular setup went up in flames at some point (look at the burn marks on the door and melted cables). now you think that if this closet was setup properly (which would include asthetics) that it would have gone up in flames so easily. And do you think that any client/partner/investor would think you had your priorities straight if they saw a closet like this?
  • Look out! (Score:4, Funny)

    by Anonymous Coward on Thursday October 24, 2002 @05:27PM (#4525245)
    The Dawson's Creek Trapper Keeper is merging
  • by scott1853 ( 194884 ) on Thursday October 24, 2002 @05:27PM (#4525248)
    You want scary:

    Our server room has wobbly floors and tower cases stacked 2 high sitting in the middle of a 16'x16' room, with the more important servers sitting on the top.

    We also got a new air conditioner that has an electronic switch and we have problems with brown outs. So in the middle of the summer, the power goes out and the A/C doesn't come back on, usually on a weekend too.

    One more thing: brace yourself:

    We use Windows servers with IIS!!!!!!!!
  • by graphicartist82 ( 462767 ) on Thursday October 24, 2002 @05:29PM (#4525258)
    about 3 years ago, I was pulling cable from 1 room to another.. while standing on a ladder pulling the cable out of the cieling, the cieling collapsed on me! About half the tiles from the room fell on to the floor, and a florescent light hit me in the back on its way down.. that's pretty dangerous!
  • by hopbine ( 618442 ) on Thursday October 24, 2002 @05:31PM (#4525271)
    As a long time Customer Engineeer for a major manufacture, I've had my share of basement computer rooms and all the subsequent flooding thereof. That being said, my favourite customer had a mainframe in the basement of his 100 year old private house. He was running a time share (remember those) system over 300 baud modems. To enter the place one had to walk beside his wifes pottery kiln. However once inside a really excellent air conditioned room there was the beast, a brand new HP3000 series 2 with 500Mb of disk. He was my favourite because on my first after hours preventive maintenance call - in fact it was the first time I had ever seen the place - his wife called out after a couple of hours.." Phil, dinner is ready !"
  • Suffocation (Score:5, Interesting)

    by eyeball ( 17206 ) on Thursday October 24, 2002 @05:32PM (#4525276) Journal
    I was in a large shared server facility that still used a Halon system, which when released, fills the area suffocating the fire (and any living creatures in the area as well).

    Anyway, one day we were working in our cage, when we heard a warning alarm, and saw all the employees running for their lives. Not knowing that the alarm meant the Halon system was about to be activiate, we joined in anyway and ran for the emergency exits.

    It turned out the fire alarms were set off by accident by someone drilling and creating dust, and luckily the people on-site disabled the fire supression system before it went off.

    • by uptownguy ( 215934 ) <> on Thursday October 24, 2002 @06:48PM (#4525759)
      I was in a large shared server facility that still used a Halon system, which when released, fills the area suffocating the fire (and any living creatures in the area as well).

      Look -- geeks sue for the wrong kind of keyboard -- ergonomics = safety after all -- do you really think there is just this dangerous gas ready to be released where you work and you have exactly 20 seconds to escape before you die?

      Taken from the first site that came up on Google... []

      "Three things must come together at the same time to start a fire. The first ingredient is fuel (anything that can burn), the second is oxygen (normal breathing air is ample) and the last is an ignition source (high heat can cause a fire even without a spark or open flame). Traditionally, to stop a fire you need to remove one side of the triangle - the ignition, the fuel or the oxygen. Halon adds a fourth dimension to fire fighting - breaking the chain reaction. It stops the fuel, the ignition and the oxygen from dancing together by chemically reacting with them. Many people believe that Halon displaces the air out of the area it is dispensed in. Wrong! Even for the toughest hazards, less than an 8% concentration by volume is required. There is still plenty of air to use in the evacuation process. (Emphasis added by me)

      I'm not an expert, but urban myths bug me...
  • by Silas ( 35023 ) on Thursday October 24, 2002 @05:32PM (#4525278) Homepage
    In rural Indiana, you don't always have space to have a whole room devoted to servers and network equipment, ya know?! But I was still surprised when I visited my former ISPs local point of presence - in one of their employee's one and only bathroom at his house. Photo here []. Do some laundry, take a dump, watch some network traffic go by. Uh-huh.
  • by weird mehgny ( 549321 ) on Thursday October 24, 2002 @05:33PM (#4525283)
    ...looks like some new cybernetic monster I imagine will make an appearance in Doom 3.
  • by peterdaly ( 123554 ) < t c> on Thursday October 24, 2002 @05:33PM (#4525289)
    At my last job we had a network of about 600 local users. Our server room had two racks of equiment on the building's ups, so the racks plugged right into the wall.

    One day we had a broom leaning up against the wall next to two of our cabinet's. Someone bumped the broom, which fell in the long arc that brooms do when they fall along a wall when leaning. One the way, it happened to unplug our two cabinets from the wall. So much for uptime. The place when quiet and we all just stared at each other for a few seconds. This is in an envirnment where downtime isn't really tolerated at all.

    Our task the next weekend?

    We took a whole package of zip ties, loosened up the plug wall plate, zip tied the plugs around the back of the outlet wall plate with an ungodly amount of zip ties, and screwed the wall plate tight again.

    Our version of 120volt twist locks. :-)

    Was interesting to hear what people would ask after seeing it for the first time.

    Not quite the server room from hell, but the story's on topic.

    • by afidel ( 530433 ) on Thursday October 24, 2002 @06:09PM (#4525477)
      I have a kind of similar story. Our UPS and generator are attached to three sets of strobe lights and alarms, one on each entry door and one inside the datacenter. Well one day I go into the datacenter because the UPS alarm is going off making all sorts of racket. I try to clear the alarm and figure out what is going on, well while hitting the button to clear the alarm my finger slips across the surface mount switch to the one under it. Problem is the switch under it is the OFF button! Who makes a UPS with a surface mount off button let alone one not protected by a flip guard? Well I have to say that I now HATE the sound of silence, at least with respect to datacenters. Worst part about the experience, the alarm was actually just a notification to perform routine maintenance, clean the air filter.
    • by Dun Malg ( 230075 ) on Thursday October 24, 2002 @06:44PM (#4525727) Homepage
      I have a good broom story much like that. I was installing a whole stack of routers in the server room of a very large travel agency. The room also contained the brains of their entire phone system. The whole thing (4 huge compaq servers, dozens of routers/switches, the phone KSU) was hard wired into a 3'x2'x4'high UPS unit to keep it all running no matter what happened to the power. Janitorial services folks always seem to think that the server room is the best place for the brooms, mops, dusters, etc. (perhaps because there are no chairs, they think no one ever goes in there?) and I had to move a half dozen brooms and dustmops to get to the back side of the equipment rack. So I leaned them against the wall by the door and was happily working away when a lowly "intern" type came in and knocked over one of the brooms. It fell, in that same well known broom-arc, bounced off the doorknob and landed right on the front panel of the UPS and flipped the little rocker switch that cut off all power to everything hooked up to the UPS. The room suddenly became deathly quiet, but the silence was soon broken by the yells of 200+ travel agents on three floors whose phones and network connections had suddenly gone dead. Normally, the UPS is supposed to have a plastic cover to prevent such things, but some dolt had removed it for reasons unknown. I flipped the switch back on and everything was back online within five minutes, but they were still quite upset.
  • The fact is (Score:3, Funny)

    by dan dan the dna man ( 461768 ) on Thursday October 24, 2002 @05:37PM (#4525303) Homepage Journal
    keeping your wiring like that is the only way to stay employed.. I mean.. would you like anyone else to try and decipher your wiring schema?

    I'm messy by nature, but I know that as long as I'm the only person who understands the mess, I'm indispensable ;)

    And I got a new job yesterday... I pity my successor :P

  • by Trinity-Infinity ( 91335 ) on Thursday October 24, 2002 @05:37PM (#4525307) Homepage
    ... when my foot broke through one of those floor tiles in the server room. Funny, someone told me they were high resistence, that must not be exactly the case ;)
  • by Havokmon ( 89874 ) <> on Thursday October 24, 2002 @05:38PM (#4525314) Homepage Journal
    Apparently, this is what they say is their scariest server room. []

    That doesn't surprise me. I have a strange feeling many of you have been caught doing this [] in front of your pc.

    So much for my Delta Force keyboard layout ;)

  • by guidemaker ( 570195 ) on Thursday October 24, 2002 @05:38PM (#4525323)
    We launched a website with publicity on a live primetime TV show about the internet (in the UK), while the server (singular) was still running under my desk. It was a little while before we moved it out of there and, amazingly, I never accidentally shut down the site with my knee.

    Of course, when we did, eventually, move it into a server room, the aircon subsequently broke down and, being an underfunded dotcom, nobody wanted to spring for repairs. We lost at least one server that way (thankfully not a live-facing one).
  • Problem is... (Score:4, Insightful)

    by Chicane-UK ( 455253 ) <> on Thursday October 24, 2002 @05:38PM (#4525324) Homepage
    Our main wiring closet looks a little like the 'not so bad examples' of wiring closets in those pictures - though nothing like the more extreme ones.

    The problem is, once the thing gets into that kind of mess, you rarely have the chance to bring down the entire network to repatch all the cables and cable tie them into some kind of order.

    Not only that, but if you have loads of trunks and VLAN's configured, putting it all back in the right order can be a total ballache!
  • by Monkeyman334 ( 205694 ) on Thursday October 24, 2002 @05:40PM (#4525331)
    to ensure it could withstand the island's regular earthquakes

    Has anyone else been in a non-earthquake-prone place and then had an earthquake? Here at work our server room was completely unprepared for an earthquake. Some of the machines came off the racks, some of the whole racks fell, our T1s got damaged or disconnected somewhere in the process. The whole disaster showed us how stupid some users can be. First, the T1 provider calls from Boston or something, "Duh, we show that your T1 lines are down, blah blah blah..", "Uh ... yeah ... we just had a 6.2 earthquake and we're rebuilding the server room", then he gave a kinda "is he serious?/boy am I embarassed" pause and gave me his professional opinion: "Oh, okay, that might be the problem." And then the fucking dial-up users. They're on the TV saying We know there has been an earthquake, please do not call the police to tell them there has been an earthquake, try to use the phone for emergencies only!. And our users are trying to get on freaking AOL instant messenger via dial-up. I was half expecting them to say "Oh yeah, I tried to dial in and it doesn't work. The phone works fine because I just called the police to tell them there was an earthquake."

    Anyway, we bolted down all the racks, shame on us for not doing it in the first place.
  • by synq ( 55040 ) on Thursday October 24, 2002 @05:43PM (#4525343) Homepage
    Think you're safe when you've got your computer room all tidy and clean?

    Forget it. We had a problem with the airconditioning at a medium size company in Delft which had it's heat-outlets on the top of the roof.

    These outlets were not so well protected to cold as was shown two years ago when, after a freezingly cold weekend we came into the server room and it was really boiling hot. Problem was the huge ventilators on the roof were stuck frozen.

    It was the only time we had all windows and doors open in the middle of winter. But this cold could well have started a fire.
  • Reminds me of... (Score:4, Interesting)

    by Nos. ( 179609 ) <> on Thursday October 24, 2002 @05:43PM (#4525347) Homepage
    I work in a 6 story building, and our telephony room happens to be on the 5th floor. The space immediately above is leased by another group.

    I get a call from the folks on the 5th floor saying there's water coming out from under the door (they don't have access to the room). Being the onsite telephony person, I rush up wondering what the hell is going on. Sure enough, water has soaked the carpet around the door. Opening the door I see as much as 5cm (2in) of water sitting around the base of the switch and various servers connected to the switch. All in all, probably around $300K worth of equipment, and I don't dare go in, because there are power cords lying in the water. Finally get the power turned off, the water out, clean everything up. All in all, costed us a couple hundred dollars for some new cables, one monitor, and various odds and ends.

    Apparently what happened was someone on the next floor up was in a bathroom, turned on a faucet and forgot about it. The water managed to move about 5m (15ft) down the hall before deciding to pour out into our telephony room!

  • My house. (Score:4, Informative)

    by FyRE666 ( 263011 ) on Thursday October 24, 2002 @05:44PM (#4525353) Homepage
    I'll have to drag my camera up into my attic to take a picture of the mess up there one day. I've 3 machines that run 24/7 up there, covered in dust, various other crap and spiders (and boy do those spiders grow big up there!)

    I put them up there as I got sick of the noise, then ran ethernet cabling down to sockets around the place. My main web server (see sig) has been happily running up there for about 1 1/2 years now - barring the odd upgrade. It's not pleasant to open up a case full of dust and cobwebs, BTW!
  • BAH! (Score:5, Funny)

    by Telecommando ( 513768 ) on Thursday October 24, 2002 @05:44PM (#4525359)
    Those are nothing. NOTHING, I tell you!

    I've got a set of routers located in a crawlspace where the only way to get to them is to walk across boards spanning small metal beams that were put in to hold a suspended false ceiling. One missed step and you'll drop right through the ceiling, AND IT'S A 2 STORY DROP! Once I dropped a power pack while replacing it and nearly killed a gal working below. Power pack exploded like a bomb when it hit.

    We recently had a "security audit" where they recommended we should mount those routers in a locked cabinet for increased security. Not a mention about the boards, lack of handrail, safety net, etc. Heck, who needs a locked cabinet? Just remove one of the boards and NO ONE can get to those routers, not even the people who are supposed to maintain them!

    Back when we used thinnet one of the managers didn't like stringing new coax through the building whenever we remodeled or moved people, so he had us cut all the coaxes to length PLUS 25 FEET! He figured if someone moved we could pull back the excess and save time. The cables all terminated in what came to be called the spaghetti room, from the coils of coax all over the floor. We had to step over all the coaxes to get to the routers and hubs. Eventually, the coaxes got damaged from all the abuse and had to be cut off to length anyway, but for several years it was a serious tripping hazard for anyone who entered that room.
    • Re:BAH! (Score:5, Funny)

      by vsprintf ( 579676 ) on Thursday October 24, 2002 @08:05PM (#4526237)
      Ha. In my day, we didn't have money for your fancy boards across the beams. We had to jump from beam to beam, uphill, both ways, and hang by the legs while making connections. But we were HAPPY to have those beams. Not like today when youngsters complain about luxuries . . .
  • by haus ( 129916 ) on Thursday October 24, 2002 @05:46PM (#4525368) Homepage Journal
    My personal favorite was building a small network out in a field. We set up our four machines [286's] in the dirt, got our power from a generator being towed by a five-ton and wired together on a 10Base2 network. For the first day or so the only shelter we had for the machines was a tarp that we pulled over them when it started to rain.

    Lying on the ground, underneath the leaky tarp, hoping that I did not get electrocuted, or if I did that I would not be held accountable for the damaged equipment [trust me, this was not my idea], I decided that re-enlistment was not a great idea.

    [former] USMC geek

  • by Real World Stuff ( 561780 ) <> on Thursday October 24, 2002 @05:48PM (#4525382) Journal
    If our IT depatment let out infrastructure degrade to anything close to that I would terminate the lot. These are representations of unprofessional half-assed setups. There is no justifiable reason for these examples. A small investment of time and an operational plan prevents this. These managers should be hung.
  • by Anonymous Coward on Thursday October 24, 2002 @05:50PM (#4525391)
    That's right you read that correctly.

    I once visited a client who had his server racks in an old lockerroom shower. This would not have been so bad except when one of my co-workers leaned against the wall and hit the valve we discovered that the pipes hadn't been capped by just had the shower heads removed....that's right three full racks of equipment in a live shower. =)

  • by twoslice ( 457793 ) on Thursday October 24, 2002 @05:53PM (#4525408)
    I was told this story from a reliable source...

    An HP technician (yup they have at least one) was restoring the data to a customer's fileserver but the backup software was asking for tape#2. The customer only ever had one backup tape that they recalled, so they were quite perplexed until the security guard entered the server room...

    Apparently every morning around 3am when he made his rounds he found the backup server screen blinking "insert next tape" -- The security guard proudly said that he was pushing in the tape for at least six months now...

  • by liamk ( 411747 ) <liamk&liamkeegan,com> on Thursday October 24, 2002 @05:55PM (#4525421) Homepage
    I work for a network consulting company, so I've seen some pretty funny stuff in the last few years. Here are some right off the top of my head:

    One company didn't order a rack mount kit for their KVM switch (some Belkin model), so they duct taped it to the main monitor. No subtle tape loops under the KVM..... they wrapped the tape three or four times around the KVM and the monitor.

    Another company was remodeling their server room but neglected to move the servers somewhere else. There was an inch and a half of drywall and sawdust on top of all the network equipment and servers. The circuit boards looked like it had snowed on them.

    I'm doing an audit on some systems. I see a motherboard sitting in a cardboard tray (the kind you get when you purchase a 24-pack of Coke from Costco), along with a hard drive, floppy, power supply and network card. No case. No cooling. Turns out it was their PDC and print server. That's quality craftsmanship.

    This isn't about server rooms, per-se, but I did some work for a national pizza chain. They had modems at a central site that were supposed to make a phone call to the stores to print out order tickets. We were sent to figure out why they weren't printing. At one site, the printer was on the floor next to the prep counter where they add the toppings. Someone had spilled a good quart of marinara sauce into the printer. They gave the outside of the printer a good once over, but the inside was just nasty.

    We were sent out to troubleshoot a voice-over-IP problem at a garden nursery. We arrive on site and lo and behold, there was a dead rat on top of the router. It didn't have anything to do with the problem, but it sure was unexpected.

    I love when people don't properly plan their electrical power consumption in their server rooms. I walked into some company's server room, plugged in my laptop to the rack mounted power strip, turned it on, and blew the breaker for two racks of servers.

    I watched a wireless network installer gob Liquid Nails onto the back of an Aironet access point and stick it to the ceiling. I hope they never want to upgrade that particular access point.

    Any other good stories?
    • by tswinzig ( 210999 ) on Thursday October 24, 2002 @06:24PM (#4525573) Journal
      I love when people don't properly plan their electrical power consumption in their server rooms. I walked into some company's server room, plugged in my laptop to the rack mounted power strip, turned it on, and blew the breaker for two racks of servers.

      I dunno, it sounds like they planned their power consumption PERFECTLY.
    • by cirby ( 2599 ) on Thursday October 24, 2002 @06:42PM (#4525721)
      I got a call from a small publishing company to do some work on their machines (they just bought some new Macs). So I looked around, and found the network cables, and the printer, and...

      "Um, where's your print server?"

      "We don't have one."

      "Yeah, you do, all of your machines are talking to it, it's here somewhere."

      "I've been here seven years, and we don't have a server."

      I traced the cables into a closet. That's blocked off by a workstation/desk. After some convincing, I managed to get them to let me move the desk, and I got into the closet. Where I found a 1987-vintage Mac II, happily munching along as a print server. Hooked into an old phone company-style UPS. Covered in a solid inch of dust and debris. And running without anyone noticing it for at least seven years...

  • Hmm. (Score:3, Funny)

    by mindstrm ( 20013 ) on Thursday October 24, 2002 @05:57PM (#4525429)
    This brings back memories.
    I remember working at an isp way back when and the server room was so bad that you could basically lean on the rats nest of wiring like it was a makeshift hammock.

    Walking around behind the racks meant being completely aware of which line you put tension on, lest it knock some connector not fastened down out of place.

  • by forged ( 206127 ) on Thursday October 24, 2002 @05:57PM (#4525430) Homepage Journal
    ...can be pretty bad, too. Check this [] out in Lebanon.

    A couple of weeks ago I came accross a couple of photograps of actual telephone exchanges in the streets of Beirut. You just wouldn't believe it, and it took me a few seconds to understand the picture. there were so many wires that you can hardly see the box behind -- kinda like Johnny Mnemonic, except with 10x more wires, and 2 or 3 handsets plugged in seemingly random (or probably not) outlets. I'll post again if I can find it back.

  • Radar (Score:5, Interesting)

    by cduffy ( 652 ) <> on Thursday October 24, 2002 @06:00PM (#4525442)
    I've got one, heard from an old-timer friend of mine who used to be a field rep (and, at other times, an AIX kernel coder) for some of IBM's big iron.

    The situation: The client's systems are crashing, on a regular basis, for no understandable reason. No remote diagnostics work, so they send out my friend.

    He gets to the server room, and keeps thinking he's seeing things out the corner of his eye. He tells everyone to leave the room, and turns out the lights. The room glows.

    The server room at this place was sitting under a huge radar system. (He had some additional explanation -- used to be a physics major -- but I didn't entirely follow it). They moved the equipment (a substantial undertaking!) and the problem went away.
    • Re:Radar (Score:4, Insightful)

      by Anonymous Coward on Thursday October 24, 2002 @06:34PM (#4525656)
      Sounds like something was lost in the translation.

      Most likely, when he flipped off the lights... they stayed on; the high-powered, high-frequency RF inducted into the wiring could be enough to keep a fluorescent lit, just like at a Tesla coil demonstration.

      CRTs and neon bulbs in equipment would no doubt show the same effect... and of course, if you have enough current in the air to light a fluorescent tube, think of what that does to your system and communications buses!
    • by Anonymous Coward on Thursday October 24, 2002 @07:30PM (#4526037)
      Not strictly rats-nest, but it follows from the previous.

      This goes back to the sixties, when computers had twitching reels of mag tape and paper tape was king.

      The company had a regular overnight run. A control tape was put into the high speed tape reader, all the relevant mag tapes mounted, and the computer got on with its six hour job (about 20 second job by todays standards). Originally there was an operator on duty, but he blatantly had nothing to do, so they decided they could do without him.

      But as soon as the operator disappeared, the job started failing at dead of night.

      OK, bring back the operator - he can fix the problem and restart the phase which went wrong.

      But, as soon as the operator came back, the problem went away. And this was the pattern - if they watched the system, it worked perfectly. But left alone, it invariably failed.

      So an engineer decided to sit there and not touch anything. He told the operator to go away, as if he was't there. Which he did, turning the lights out and leaving our hero in the dark - except for the glow of the high-speed tape reader, which shines a strong light through the holes in the punched tape onto photocells. And as he watches, a moth appears and flies through the pool of light, confusing the tape reader and aborting the job.
      • by Pig Hogger ( 10379 ) <> on Friday October 25, 2002 @12:28AM (#4527464) Journal
        So an engineer decided to sit there and not touch anything. He told the operator to go away, as if he was't there. Which he did, turning the lights out and leaving our hero in the dark - except for the glow of the high-speed tape reader, which shines a strong light through the holes in the punched tape onto photocells. And as he watches, a moth appears and flies through the pool of light, confusing the tape reader and aborting the job.
        And then he taped the moth into the computer log book, and scribbled: " Second actual case of computer bug being found ".
  • Sad (Score:5, Insightful)

    by _ph1ux_ ( 216706 ) on Thursday October 24, 2002 @06:00PM (#4525443)
    I have always thought that how organized a companies wiring is - is a direct reflection of the staff that works there.

    I know that when i moved into one company in Redwood City - the network wsa a nightmare. We had rooms that looke like that - but over the next 2 years we replaced almost every wire on that network - and demanded budget for proper closet setups - and got it.

    We eliminated all those closets that looked like that, and learned one hell of a lot in the process.

    I think that if your closets look like that - you are asking for fire - and it shows just how lazy you really are. No arguments of "I'm too busy" allowed - it just means your a lazy slob period.
    • Re:Sad (Score:4, Insightful)

      by Eol1 ( 208982 ) on Thursday October 24, 2002 @08:00PM (#4526211) Homepage Journal
      I disagree. Sometimes you inherit it (not always laziness). Recently ( as in 6 months ago) took over a 3 year contract from another company and they left wiring situations from hell. (a good example is the 100 pair multimode fiber buried about 5 feet down in the mud and ran for mile to our satellite shot with about 30 splices in it). Our wiring closets are even worse. Contract specifies operating and maintenance ONLY, not installation or engineering. You can be damned if you think I am cleaning up 3 years worth of installation hell from another contracting firm. Not in the contract and unless it goes down, not covered under maintenance. Nor am I going to subject my techs to months of misery fixing somebody elses mess especially when our contract will also be up in 3 years with no chance of renewal.

      Just because somebody has wiring from hell and jury rigged systems doesn't always mean they are lazy SOB's ... sometimes you just don't feel the need to fix other peoples problems. If your company didn't care before, they sure as hell don't care now, and if they do, they can write it in the contract.
  • Sooo... (Score:3, Funny)

    by bobdotorg ( 598873 ) on Thursday October 24, 2002 @06:15PM (#4525509)
    Taiwanese typhoon? pfffft. Come on - lets get some webcams in these dangerous server rooms and then /. em for some good old fashioned geek fireworks.

    Yeah - you know you want to.
  • SW too. (Score:3, Insightful)

    by bored ( 40072 ) on Thursday October 24, 2002 @06:41PM (#4525707)
    Lots of software is the equivilant of those pictures. The only problem is that you can't see that from the pretty pictures on the box cover. Instead you see it in crashes and strange behavior. Two things cause it, first unskilled careless workers, second stressed workers in unrealistic situations. The second seems to be more previlant. They guy who wired that first mess probably knew that its better to label everything and run it down the side of the rack to the gutter in the floor (notice the one rack is accually nicely wired under the mess on top). Instead you have the manager who gives you 1 day to wire up 100 computers, or 1 week to add some big feature to the code base. The result is scrambling like mad and a "just plug it in and make it work don't make it pretty" attitude.
  • by VitrosChemistryAnaly ( 616952 ) on Thursday October 24, 2002 @06:44PM (#4525728) Journal
    IT nerd in bar: Hey, baby, you like dangerous guys?
    Hot Chick: Yeah, I guess so.
    IT nerd in bar: Sometimes I take my pocket protector out of my pocket!
    Hot Chick: Uh-huh...
    IT nerd in bar: Sometimes I take my mint condition Megatron action figure to LAN parties!
    Hot Chick: Uh-huh...
    IT nerd in bar: And sometimes I let my server room get really messy so that it's a hazard to my life!
    Hot Chick: Wow, that is dangerous. I'm really turned on...hop on me right now...
  • by EccentricAnomaly ( 451326 ) on Thursday October 24, 2002 @06:44PM (#4525734) Homepage
    I'm sure that some of you have worked in large server rooms with a big red emergency power shutoff button on the wall...

    At my old university, one of these server rooms was emptied as new, smaller hardware came available and the room was no longer needed. They turned this room into an office for a student organization... leaving the large red button, but taking the "Emergency Shutoff" sign.

    This unlabeled button sat neglected on the wall of this little office for about 7 or 8 years until one day a curious student just had to find out what the button did...

    The network for all of the engineering schools at this university of 36,000 students went down for most of a day..

    The best part is that the button is still in the little office with the students, and it is still unlabeled yet fully functional... They did hide it behind a file cabinet, though :)

  • by t0qer ( 230538 ) on Thursday October 24, 2002 @06:50PM (#4525778) Homepage Journal
    I've worked tons of places pullin wire...

    I worked at mothers cookies in oakland CA for a spell. You think malaysia is dangerous?

    I would pull off the 880 by the colliseum,mc arthur blvd? It takes you through a very derilict section of oakland factories.

    The entire complex is surrounded by bobwire. If you drive down the street, less than a block, you are in one of the worst neighborhoods in oakland. I had a friend who lived in the neighborhood, he wasn't shittin me either about the danger. I went to his house twice to hang out and that was enough for me.

    The server room was cool, about 40degrees all the time so you wore a jacket when you went in there. But pullin wire....Ohhh my god!

    I had to run a fiber line from the main building to some office in the back of the bakerery. Now before you get the picture of little mothers running around with cookie sheets and kenmoore ovens you have to understand.... That is not what a huge production cookie plant is.

    Imaging a HUGE fricken warehouse with conveyer belts of cookies going everywhere, machinery whiiring and cookies going into boxes and filled with creme and those animal cookies with the dots, all in this HUGE room about the size of a football field.

    One end was the mixing end, where they had these mixing machines the size of my garage. Into those would go 50 gallon oil drums of butter, lard. Huge bags of flour being loaded by forklift, ect.

    Now at the time, wireless hadn't really made it mainstream. So my mangers convienced mothers fiber would be good since it would provide the best ROI. They were sold and I was sent out to work.

    Now the factory was built from steel girders covered with that tin roofing, the stuff that looks like a ruffles potato chip. I got up there to where the top girders are and before my eyes was the most treachorous wire run I ever saw.

    Remember what I said about 50 gallon oil drums of lard? Well, when the cookies baked, the lard would vaporize and rise to the ceiling then settle on the steel girders. Over the years a 1/4" layer of lard had deposited 70 feet up in the rafters.

    I put my finger in the goop to see how slippery it was.

    No friction.

    I called the office and told my boss. Later he called one of his friends to subcontract the work out too.

    *Disclaimer* Despite the lard, mothers cookies makes a great product, and was an awesome place to work. If you ever get the chance to work there, jump on it, you won't regret it (or the 50cents a bag price for employees :)
  • Wiring closets (Score:4, Insightful)

    by crumbz ( 41803 ) <<remove_spam>jus ... o spam>> on Thursday October 24, 2002 @07:02PM (#4525868) Homepage
    As a telecom contractor, I know that wiring closets have the lowest priority in terms of cleanup or the "make-it-look-pretty-the-boss-is-coming" effect. You should see some of the ones from the old Ma Bell days buried deep in the hearts of old office buildings. Yikes!
  • by r_j_prahad ( 309298 ) <`r_j_prahad' `at' `'> on Thursday October 24, 2002 @07:03PM (#4525872)
    I used to consult for a client whose server room was in the women's restroom. It had the largest unallocated space of any room in the building, so they stuffed in two low-boy cabinets full of DEC gear right next to the ladies' crapper.

    I had to remember to knock before rebooting.
  • by frank_adrian314159 ( 469671 ) on Thursday October 24, 2002 @07:05PM (#4525880) Homepage
    ... we didn't even have office buildings. We had to keep our S/360's out in the barn, and when the hogs crapped on 'em, we'd have to dig 'em out from under the... well, you get the idea. And we used barbed wire to wire 'em to the power grid, too.

    You kids these days just have it too fscking soft, I tell you...

  • by Anonymous Coward on Thursday October 24, 2002 @07:10PM (#4525912) ler.jpg [] That'll make any machine room the most dangerous.
  • by the_other_one ( 178565 ) on Thursday October 24, 2002 @07:30PM (#4526039) Homepage

    At the old company that I used to work for. They had no server room yet so the servers just sat in the corner. One day I was working late and the cleaning staff came in. The first thing that they did was to plug the vaccuum cleaner into the UPS that the main server was hooked up to.

    It seems they had always been doing that!

  • by Parsec ( 1702 ) on Thursday October 24, 2002 @07:49PM (#4526143) Homepage Journal

    When our old NEC mainframe came out we had a few tiles with 18"x10" holes formerly used for cabling in our raised floor. We still don't have exactly enough tiles, but furniture is arranged better now, a year later, so that these holes are strategically covered by desks, shelves or other equipment.

    Better than accidently wheeling your chair over that duct tape patch! I kid you not.

  • by jmulvey ( 233344 ) on Thursday October 24, 2002 @09:35PM (#4526677)
    OK, not exactly a data center story, but funny and true story nonetheless:

    Back in '86 I had a top-of-the-line Corona IBM-PC clone (cost me nearly $5,000 then). It had those big full-height floppy drives (two!) and was a very well-built, sturdy unit.

    I was working as a computer hardware technician at the time, and I had recently bought a bunch of 256k memory chips. I brought my computer to work to show it off to the guys, and also install the memory where I had a nice anti-static station.

    So there I was with all my buddies, showing off my toy. I open the case of my computer, ready to wow them, and at least a pound of dog kibbles spills out of the case. Dog kibbles are strewn all over the computer motherboard. We all kind of stood there for a moment, dumbfounded.

    Eventually, I discovered the cause. My house was infested with mice, this I had known. But what I didn't know was that, in the middle of the night, mice will steal dog kibbles from the dog dish, and hide them in little places they can get to later. Apparently, they had been climbing in through the full-height floppy drives and storing the kibbles.

    Interestingly, it never seemed to affect the computer!

  • by yack0 ( 2832 ) <keimel AT gmail DOT com> on Thursday October 24, 2002 @09:42PM (#4526708) Homepage
    If I see ANYONE near the server room with a camera this week, I'll personally remove your jimmies and make them part of megapod 3.

    You have been warned.

    (we're in the middle of a rebuild, so it's major chaos before restoration to order)

  • by Skjellifetti ( 561341 ) on Thursday October 24, 2002 @10:10PM (#4526845) Journal
    It is OK to smoke cigarettes in the server room at Philip Morris. They keep ashtrays there for the sysadmins.
    • by Pig Hogger ( 10379 ) <> on Friday October 25, 2002 @12:46AM (#4527536) Journal
      It is OK to smoke cigarettes in the server room at Philip Morris. They keep ashtrays there for the sysadmins.
      20 years ago, I was working at a Phillip Morris subsidiary, and the HP-3000 server not only had a big "THANK-YOU FOR SMOKING" sign on the wall, but one of those big ashtrays right on top of the CPU.

      HP was so happy to have the account that they didn't bitch at all each time they had to replace a disk drive every 3 weeks or so...

Never buy from a rich salesman. -- Goldenstern