Datacenter Robbed for the Fourth Time in Two Years

mariushm writes "According to the Register, the Chicago-based colocation datacenter C I Host was attacked by armed intruders recently, making it the the fourth time in two years that armed thugs have made off with data. According to a letter C I Host officials sent customers, 'At least two masked intruders entered the suite after cutting into the reinforced walls with a power saw ... During the robbery, C I Host's night manager was repeatedly tazered and struck with a blunt instrument. After violently attacking the manager, the intruders stole equipment belonging to C I Host and its customers.' Aggravating the situation, C I Host representatives took several days to admit the most recent breach, according to several customers who said they lost equipment, all the while reporting the problems as 'router failures'."

The evil thing here

is that it was reported as "Router Failures" instead of the real cause.

And if they have been robbed before - why not increase the security? Four times? - That's some kind of record. Maybe it's time to check if the localization of the whole thing is incorrect and move it to a better location where it's less likely to suffer from this kind of incident?

Advertising for a guard - "bring your own gun"

I couldn't make this stuff up. They're advertising for a security guard [careerbuilder.com],

"We are seeking motivated individuals" ... translation: work cheap.

"Prior security experience preferred." ... translation: not really a requirement, but if we can get it at no extra cost ...

"Some College is also preferred." ... if you managed to drop out of college instead of high school, you're more "presentable" to our insurers, who are now royally pissed at us ...

... and if you scroll down to the bottom of the page:

"Armed Hand-gun license/permit and ability to supply own weapon a Huge Plus! : translation: "we're cheap! You're desperate AND stupid! Let's talk!"

Re:Advertising for a guard - "bring your own gun"

1. The police think it was an inside job
2. The employee wasn't on the premises when the door lock was jimmied open (no, they didn't "cut through a reinforced wall" - they just forced a door open according to the police report and people who went to visit the site the next day to check on their equipment)
3. Fat chance getting a concealed carry permit in Chicago.

   Illinois is one of the few states that has no provision for the concealed carry of firearms by citizens. Open carry is also illegal, except when hunting. When a firearm is being transported, it must be unloaded and enclosed in a case.

   Only cops and military get CCW permits.

CI Host doesn't want to spend the money on secure facilities, instead replacing that with a rent-a-dumb-warm-body. Dumb, because taking this job w/o a permit is just begging to be thrown in jail, and if you have the permit, you can sure as heck do better than CI Host.

Re:Bringing own gun is a good idea

You can't GET a carry permit in Chicago, unless you're a cop or military.

CI Host didn't invest in proper facilities. Contrary to the article summary, the robbery was made by people forcing open the door lock to the office, when nobody was there, and an employee "just happened to show up later in response to the alarm". And the crooks "just happened to have tasers" instead of guns. And the crooks "just happened to steal all the non-existent video surveillance cameras".

And the walls are not "reinforced" - they're plain ordinary office walls. Unless you want to count a new coat of paint as "reinforcement".

Re:The evil thing here

They didn't bust through a wall this time - they forced the lock on the front door of the office "suite", according to a customer who went there the next day to check on his equipment.

The "reinforced walls" exist in the same universe as the "router outage".

There were no employees on duty at the time of the break-in. One employee showed up and got himself tazered, AFTER the door had been forced, in response to an alarm.

This was the 4th break-in in 3 years. That alone is suspicious. Taser? Sure, can't kill a co-worker, right?

Security cameras? There's now a question as to whether they (security cameras owned by CI Host) existed in the first place. The only cameras anyone has seen are a few owned by other businesses in the building ...

All very suspicious.

Re:The evil thing here

I agree, the routers [wikipedia.org] _didn't_ fail, that's how the thieves got in in the first place.

Re:The evil thing here - continuation.

"at least two masked intruders entered the suite after cutting into the reinforced walls with a power saw,"

In what way was that wall reinforced? Dual layer of sheetrock? If it was sufficiently reinforced it would have delayed the intruders long enough for the police to get there (unless the police chose to not respond). If I was insuring that company I would drop the insurance dead by now due to lack of sufficient protective measures. If the measures were approved by the insurance company I would recommend all other clients to change insurance company.

Anyway - maybe it's time to weave in copper mesh into the T-shirt of all datacenter employees to protect against tazers.

And notice from a comment to the article that any so called man trap doesn't exist - and the security seems to have been far too relaxed. Just a fine example of how not to do things. A good datacenter is located where almost nobody knows where it is - preferably underground in a nondescript location in the countryside. A set of optical fibers will take care of all the traffic. And very few persons shall have physical access to the hardware. Think about how the military handles their datacenters.

Re:The evil thing here - continuation.

Stronger walls, and maybe armed security guards. Heck, we have them up here in Canada and we don't have a tenth of the violent crime problems Chicago has.

Re:The evil thing here - continuation.

I actually had a server hosted in that very Chicago facility. (I actually got referred to it by clicking a "$75 a month colocation" advertisement link on slashdot)

The datacenter in question is in a terrible neighborhood, and I can't see anyone bothering a truck there in the dead of night.

There was no man trap, and no security of any sort, just a tech guy who let me in and opened the glass datacenter door for me.

I doubt they have a panic button of any sort either.

You disable the one guy on call and there would be no police coming, period.

Re:The evil thing here - continuation.

if you don't know what a datacenter is, you're probably nowhere near interested in this entire field.

The fact that he is asking at all proves that he is interested in the field.

You must have very poor memory, nuzak, if you can't recall what it was like learning your trade.

When I was in high school, I knew I wanted to work in the computer field, yet I didn't know what a datacenter was. If I had met a pretentious jerk like you, I might not be managing a datacenter, today. Luckily, I met people who answered my questions instead of people like you. I also asked questions on slashdot, and learned quite a lot from this place.

And to answer the original question: A datacenter is a room full of computers with experts monitoring and maintaining them 24/7. It has special air conditioning, wiring, and security. Sometimes people rent servers or space there. Sometimes they contain only the computers of the company that owns the datacenter.

Re:The evil thing here

In that forum, it was posted that the hosting provider had posted a job application for somebody willing or able to carry a gun. They were hiring at minimum wage or just above minimum wage if you had experience with guns.

So, they're looking to hire people that carry guns that are willing to accept a job at minimum wage. That should tell you something right there.

Re:The evil thing here

somebody willing or able to carry a gun.

Illinois is one of the most anti-gun states, and Chicago has even more strict rules on top of that. It's almost as bad as Britain. About the only way you're going to have an armed security guard in Chicago is if he's actually a sworn law enforcement officer or you have *really* tight political connections to those in power.

In a more, uh... "free" state, yes, armed security is a realistic proposition. However such states usually have less violent crime too, so you don't need them as much.

Re:The evil thing here

Illinois is one of the most anti-gun states

Sounds like a great reason to locate a data center in Texas.

-jcr

Re:The evil thing here

It couldn't have happened to a better bunch of scumbags. [slashdot.org] In fact I wouldn't be surprised at all if they robbed the datacenter themselves to destroy evidence or just for insurance fraud.

Pointsec

Hopefully they were smart enough to encrypt the data this time. If not, the company should truly be shut down...

Obligatory...

Don't tase me, bro!

Seriously, though, this sounds like something out of a really bad Hollywood B-Movie.

I didn't know you could do stuff like this in real life.

Re:Obligatory...

No, it sounds like something out of a ridiculously popular A-movie that makes 100s of millions of dollars. You underestimate the american public's willingness to watch total crap.

inside job

smells like an inside job / insurance scam 1st would be an anomaly , 4th time i would be looking very hard at the companies and its staffs finances

Re:inside job

It most likely was an inside job. A little while back, I was working for a company that was installing some VoIP phones for CI Host and the list of employee phone numbers kept changing from visit to visit - "Oh that guy? No, he doesn't work here any more."

A friend of mine that used to work there said that "being in jail was a fairly common excuse for missing work there". The employees seemed to hate working there, to put it mildly.

And the cokehead that owned the company loved to fire employees at a moment's notice, left and right. I highly doubt there's any employee loyalty there.

So in short, you've got highly unhappy employees that get fired at an amazing rate, with some seriously negative employee loyalty and they're surprised when stuff gets stolen?

To be fair, there was a router failure

It turns out the router was unable to route wherever the thieves had taken it.

Not using them anymore

Hmm, I'm not using them anymore. They had regular power failures in Dallas - claiming 'UPS maintenance'. My home DSL setup is more reliable than their data centre.

Re:Not using them anymore

Ditto. I switched off them a year and a half or two ago. I don't remember what the final rub was, I think it was reliability (website or email going up or down seemingly randomly?). I've been on Dreamhost since, and been pretty happy (note: referral link at the bottom of my website).

I agree with the other posters. They lied. They obviously have no security (or they are facing an inside job). Four robberies in two years?

I'd switch off 'em real fast if I heard this news. I like Dreamhost but if I heard this about them I'd probably switch off them fast too. How can I trust a hosting company that can't even secure their own premises?

Still in business?

After the first robbery, I'd seriously consider moving my data. If my data is still there after the second robbery, I feel stupid. If my data is still there after the third robbery, I should lose my job. If my data is still there after the fourth robbery, I need to promoted to executive management.

The entire purpose of off-site storage is disaster recovery, and prevention of major disasters like this. Why are these guys still in business?

Re:Still in business?

Ok, sure it kinda sucks that the place you store your data was robbed 4 times... but what are the odds it could happen a 5th time?

where are the rent a cops at?

In the brake room watching tv / playing games?

Doing the same thing in the big room with all of the tv screens?

Re:where are the rent a cops at?

In the brake room watching tv / playing games?

No, one door to the left in the clutch room.

rj

Honesty.

I'm a Dreamhost customer, and the past couple years they've had a few issues, and some people have taken it as an opportunity to bash the hell out of them. Having used many hosts over the years myself and for customers, I've found them to be on par with, if not a little above many hosts. The biggest difference is DH is HONEST about their issues, on their status blog. When they fuck up, they say so. To me, that's more valuable than a host that makes it self look like it's more stable by lying, such as the parent article's subject. It means when DH tells me something, I can at least trust it's the truth.

location, location, location

Suddenly, buying an old army bunker complex to house your datacenter doesn't seem that excessive.

Fool me once....

shame on you. Fool me four or more times shame on me!

router failure

Well..if they had said "route failure" that would of been accurate since technically the route HAS changed for those servers......

There has to be more to it than this.

That was a truly "professional" operation. It's happened four times. They're specifically targeting this datacenter, and management's reaction is anomalous. If I were a police investigator, I'd start probing. Maybe there's nothing to find, but I'd bet money that there is.

Kind of reminds me of that pizza bomber a couple of years ago.

A few possibilities....

Well, if they actually care enough to try to prevent these attacks, I can see three solutions, any of which should be highly effective:

Deadly force. If you are being robbed at gunpoint on a regular basis, your employees can legitimately say that they fear for their lives, and thus, purchasing of firearms is legally and morally justifiable. Perhaps a couple of guards posted at the entrance with semiautomatic rifles, plus three or four in appropriately concealed locations within the facility (or more if the facility is large enough). Criminals (armed or not) will think twice before attacking.

Electrical interference. Hook a 230 kV transmission line directly to the rebar in the walls. Anyone who tries to cut their way in will likely spontaneously combust, or at the very least, be knocked several meters. Such an attack won't happen twice.

Oxygen deprivation. You probably already have halon fire extinguishers. Assign everyone emergency oxygen masks and a red button remote. In the event of an attack, press the red button and put on your oxygen mask. Assuming you dump enough halon, it will bond with all the free oxygen in the room, incapacitating or killing the intruders in seconds. Assuming they survive, they should still be unconscious when the police arrive to arrest them.

Re:A few possibilities....

Armed guards are probably legal, and using the fire suppression system you could probably get away with if you didn't specifically give orders to use it as a weapon, but the electrical booby trap is almost certainly illegal. If nothing else it almost certainly contravenes local health and safety laws.

Re:A few possibilities....

Deadly force. If you are being robbed at gunpoint on a regular basis, your employees can legitimately say that they fear for their lives, and thus, purchasing of firearms is legally and morally justifiable. Perhaps a couple of guards posted at the entrance with semiautomatic rifles, plus three or four in appropriately concealed locations within the facility (or more if the facility is large enough). Criminals (armed or not) will think twice before attacking.

If the criminals REALLY want to get in it's not a good idea to arm the employees anyway. Specially assigned guards maybe, but the employees of a data center - no... Just imagine if someone has a bad day... Shooting out at the boss, servers and everyone else in sight. Or if the criminals know about it they will shoot first and check later.

Electrical interference. Hook a 230 kV transmission line directly to the rebar in the walls. Anyone who tries to cut their way in will likely spontaneously combust, or at the very least, be knocked several meters. Such an attack won't happen twice.

Assuming that it's concrete walls... But it's a good idea until the maintenance guy comes in to drill a new hole for a cable.

Oxygen deprivation. You probably already have halon fire extinguishers. Assign everyone emergency oxygen masks and a red button remote. In the event of an attack, press the red button and put on your oxygen mask. Assuming you dump enough halon, it will bond with all the free oxygen in the room, incapacitating or killing the intruders in seconds. Assuming they survive, they should still be unconscious when the police arrive to arrest them.

Halon use is outlawed, at least in some countries since it has a bad effect on the ozone layer. Carbon Dioxide is almost as good, and has the same effect. Of course - you may use any non-oxidizing gas like pure nitrogen or helium instead. As long as it lowers the oxygen level in the compartment. A much more evil way is to use carbon monoxide instead of carbon dioxide. In this case the survivability is even lower, but if it's released by accident it will be much more nasty.

But it seems that the datacenter hasn't taken action as it should and moved the servers to a different more covert location. The daily operation can remain at the same location, but since the servers aren't there anymore the criminals will have to leave empty-handed. This requires that the persons running the night-shift doesn't know about the real location of the servers unless they also are relocated.