


How To Prevent Being Hacked Via Backups? 214
Popsikle writes "A few days ago one of the Web's largest hosting discussion forums was supposedly hacked via their backup servers. From the story: 'We've since learned that this very deliberate, sophisticated and calculated hack against Web Hosting Talk was carried out by gaining access to our offsite backup servers. From our backup servers, the hacker gained access to the WHT db server. The malicious attacker deleted all backups from the backup servers within the infrastructure before deleting tables from our db server. We were alerted of the db exploitation and quickly shut down the site to prevent further damage.' What sort of security do you put on your backup infrastructure? Looking at your backup solution could you be completely taken down by either someone obtaining a backup or accessing your backup servers? What sort of recommendations does everyone have for this not to happen?"
Easy fix (Score:5, Insightful)
Offline and offsite storage (i.e. iron mountain) is a simple (though sometimes costly) way of doing things.
it'll solve this problem quite easily.
Re:Easy fix (Score:5, Interesting)
Even easier.. put a external HD with Truecrypt under your bed. Maybe take a second one into the office, Lock it in a drawer, just in case your house burns down..
Every week or so, use 1 to backup. Alternate which one. Add more drives, or more often backups until you get to the point you sleep easy at night.
Re: (Score:2)
iptables -A INPUT -s $SITERANGE -j ACCEPT
iptables -A INPUT -s $OFFICEGW -j ACCEPT
iptables -A INPUT -j DROP
I don't see why my offsite backup solution should be such an issue - it is just a bunch of windows servers sitting behind an OpenBSD firewall with pretty much exactly those kinds of rules loaded into it (more restrictions on ports etc.). Also, if I screwed up those iptables, see sig ;)
Re: (Score:2)
Re: (Score:2)
And if you're a bigger business you use a WORM solution. Technology - it's flexible.
Re: (Score:2)
What part of "Add more drives, or more often backups until you get to the point you sleep easy at night." confused you?
Re: (Score:2)
Just don't use Maxtor drives if you actually want to be able to retrieve your data at any point.
Where I used to work we had 6 dead Maxtor drives in as many months. Luckily this was spread across 3 servers which had 4 backup drives each but it certainly taught a lesson that some hard drive manufacturers are less than reliable.
It's probably also worth pointing out we had some Seagate drives bought at a different time, stored along with all the Maxtor drives and used in the same way without a problem, so it di
Re: (Score:2)
We must have different ideas of what "easy" means. To me, "set up a cron job which stores a backup online every hour" is easy, while "buy two hard drives and drive them between buildings each week" sounds like a lot of work.
Re: (Score:2)
If you have to go to the other building anyways, it's not so much work.
Offsite backups are needed in case your building burns down. (A nuke would take out your other building too, but at that point you may not care.)
The problem with a cronjob (only) backup system is that it needs online storage (doesn't matter if it's tape or disk), and if the backup system can alter the backup, then so can a bad guy who gets in (and certain failure types could too, depending on the specifics.) Sure, there's some w
Re: (Score:2)
Also, one set of backups isn't enough. What if things broke before the last backup, and you need to go back further?
The phrase "set of backups" is rather ambiguous here -- but if you're using an intelligent backup system (like tarsnap) you'll have multiple snapshots stored.
Re: (Score:2)
Even easier.. put a external HD with Truecrypt under your bed. Maybe take a second one into the office, Lock it in a drawer, just in case your house burns down..
Every week or so, use 1 to backup. Alternate which one. Add more drives, or more often backups until you get to the point you sleep easy at night.
Or hire a midget to follow you around with a backpack full of backups. Then hire an ogre to protect the midget. Carry a gun in case the ogre and the midget mutiny against you, and keep a cyanide capsule in your tooth filling, in case either one of them gets the gun.
Re:Easy fix (Score:5, Insightful)
HDs are NOT backup media.
Please provide a citation that hard disks are noticeably worse than tape, which you appear to otherwise recommend.
Re: (Score:2)
Hard drives are cheaper per G, and have been since 2002. Hard drives do fail, but usually due to impacts, or while in use.
It's like all those people who insist that putting objects in RAM will be slower than a "real" database...
Re: (Score:2)
Tapes Are Rubbish (Score:3, Insightful)
yes, because people regularly break into my NOC, pull the drives out of my backup servers and practice juggling. Tapes are rubbish. Tape is expensive and unreliable. Anyone that tells you otherwise is selling the stuff.
Hard drives have replaced tapes around here. For decades we juggled tapes in a vain attempt to maintain useful backups only to find that EVERY SINGLE TIME we needed critical data from backup, the data was unrecoverable.
For the past few years now, we have been backing up all of our critica
Re: (Score:3, Informative)
I hope nobody confuses your inability to think beyond your world with fact, because this is the hugest crock of shit.
I have about 12 TB of data that I am required to be able to restore to any day within the last five weeks. One week of backups = 20 TB. In addition, I must:
1) always have at least four weeks' worth of data in different building on the same site
2) always have at most one week's
Re: (Score:2)
Re:Easy fix (Score:5, Funny)
Re: (Score:2)
Don't sell yourself short! Because it's stored on a hard drive, which can be continuously online without suffering wear or stretching, you were already automatically verifying the backup's integrity every week. So you already knew that the hard drive failed - the day that it failed. You were able to use the
Re:Easy fix (Score:5, Insightful)
No, they are not, you are correct. In my post, I was assuming that this was a small website or business, not a mission critical company product. I didn't mention software, or tape libraries, or hot backups. I think sometimes its better to have some backups, then none at all. An external drive can be bought for next to nothing.. really, I can get a 500GB HD for about the cost of a couple of tapes, but then I have to buy 2 tape drives (in case one has hardware failure).
Believe me, I could go on about backup windows, media, and techniques, but was hoping by keeping it simple, they would not be overwhelmed. By not being overwhelmed, they might start the process.
Also, by not using a computer based backup, they would not have the same problem as the site mentioned in the linked article, where someone first cracked their backup servers, and deleted their only backups.
Re: (Score:2)
This is definitely true in that they should not be relied on as a primary or sole backup, but they can be used as a second backup that is easier and quicker to restore.
The IT dept that I run does cross server backups to DLT tapes, as well as backups to a NAS.
The DLTs are taken offsite and stored in a fireproof safe and would be used for something catastrophic, but the NAS backups are used if we need to restore a file here or there because somebody does something stupid.
This way somebody doesn't have to brin
Re: (Score:2)
I do a very similar thing.
My backup machine does nightly archives to tape, but it's also got an array of disks [blogspot.com] for nightly backups. If the backup server dies, those hard drives are instantly mountable on any of the machines by just plugging them in.
The details in the link above aren't right, especially the gordian bash-knot, since I've been moving to AMANDA, but the premise is the same. Nightly backups to multiple media, full weekly backups to long term storage media.
Re: (Score:3, Informative)
Ok, there were several replies to my post and, since I'm not going to reply to all of them, I decided to pick one that is clear and definitively worth discussing with.
I do agree with your backup strategy. It is sound. I use that one some sites, having backup made to tapes, and a secondary storage area (or a pre-backup staging area) on a NAS, for fast recovery of trivial files. Restoring files from the NAS is usually fasted, as you stated, than from a tape.
Using BOTH HDDs and tapes to supplement each others,
Re: (Score:2)
HDs are NOT backup media.
Why not?
Obviously you can't plug in a single external HDD, leave it in the building all the time, do nightly dumps to it, and call it a backup. That's wrong for at least a half-dozen reasons.
But what's wrong with buying a pile of external drives and using them just like you would a tape? Plug one in every day, run the backup, and then store it someplace safe offsite. Seems to me that it would work just the same as a tape.
Re: (Score:2)
Seems to me that it would work just the same as a tape.
Yes, it would. I think the guy you're responding to is referring only to drives left accessible (especially mounted) in a computer 24x7.
Of course, a tape drive has the same problem -- if a bad guy gets into your backup server, he can erase any tape in the drive, and if there's an automatic robot, he can tell the system to erase every tape it has access to. (Might take a while, however.)
So, take his rant and replace `HDs are NOT backup media' with `Backups need to be kept offline when not in use for
Re: (Score:2)
HDs are NOT backup media
Since when? Disaster recovery should, in theory, be spread out over HDD, Tape and Optical. Long term storage should be mirrored between optical and HDD, with regular refreshes.
Misdirected frothing. (Score:3, Insightful)
HDs are NOT backup media.
It appears you've taken the oft-repeated mantra that RAID is not a backup solution and gone a step further by suggesting that hard drives themselves aren't backup media.
Which, by the way, is ludicrous. Hard drives use tried-and-true technology, they're cheap as hell, and transfer speeds are outrageously faster than any other media in contention. Suggesting that they don't make good backup media is well, I'll give you the benefit of the doubt and say misguided.
Re: (Score:2)
Re: (Score:3, Informative)
At my last job we had 3 layers of backups; the first was to a large harddrive (actually a RAID away, but still an array of harddrives.) Then fro
Re: (Score:2)
The downside is transportation costs along with bulk and no robotic mass handling of the units.
On the upside, if a solitary hard drive fails, you're only out the data on that drive. If a tape r
Tachikoma (Score:5, Insightful)
Take a lesson from Ghost in the Shell, hire digital Tachikoma to protect you :) Problem solved!
Re: (Score:3, Funny)
Those Tachikoma will do what ever it takes, especially once they gain individuality.
Re: (Score:2, Informative)
See also: The classic answer to computer problems (Score:5, Funny)
>>What sort of security do you put on your backup infrastructure?
It depends.
I guess it depends how valuable the data is, how current it needs to be. Does it need to be kept secret or simply kept uncorrupted? How fast do you need to access these backups. The harder for you, potentially the more secure.
For starters let's have a copy off-line in a location where the servers are not. Heck, choose another location for more security. I am thinking: A tape and a DVD in a safe with a lock on it. And another somewhere else.
If you really need to keep it safe, commit it all to memory and then shoot yourself in the temple.
Re:See also: The classic answer to computer proble (Score:5, Funny)
If you really need to keep it safe, commit it all to memory and then shoot yourself in the temple.
hey, the guy might NOT be jewish.
did you consider that?
Re:See also: The classic answer to computer proble (Score:5, Funny)
Well done sir. Either way, it'd be a horrible way to parish.
Re:See also: The classic answer to computer proble (Score:5, Funny)
Re:See also: The classic answer to computer proble (Score:5, Funny)
Christ, you guys are merciless. Next you'll be wanting to nail him to a tree or something.
Jesus, I come here for the intellectual discussion. I didn't expect this kind of Spanish Inquisition.
Re: (Score:3, Funny)
Keep a 3 day backup and cross your fingers.
Re: (Score:2)
Excuse me being lame, but can you please explain the reference?
Re:See also: The classic answer to computer proble (Score:5, Funny)
Too soon?
Re: (Score:2)
>>What sort of security do you put on your backup infrastructure?
It depends.
I guess it depends how valuable the data is, how current it needs to be. Does it need to be kept secret or simply kept uncorrupted? How fast do you need to access these backups. The harder for you, potentially the more secure.
For starters let's have a copy off-line in a location where the servers are not. Heck, choose another location for more security. I am thinking: A tape and a DVD in a safe with a lock on it. And another somewhere else.
If you really need to keep it safe, commit it all to memory and then shoot yourself in the temple.
I think you are missing an even bigger problem. How many of your servers does your backup infrastructure have admin access to - or if not admin elevated access? Are your backups a push or pull? If they are a pull you now have INBOUND firewall rules from your backup segment into your other network segments allowing the backup server to talk and start the pull. Or if you have a backup segment, there is not firewall protection, except host based firewalls, which have rules to allow the backup server to talk.
Re:See also: The classic answer to computer proble (Score:5, Interesting)
First off, lets focus on one thing, if your company is already at a size, complexity, or business need of having backup data electronically replicated offsite (whether this is a true hotsite you're replicating to, or just a method for not rotating tapes through your front door every day), then we're not talking about USB drive data protection schemes here...
OK now, the offsite system should NOT be an administrator access node of your existing backup solution. It should receive replicated data sets and have the ability to operate as a DR server to recover them, but it should be controlled remotely from YOUR site, and should not be able to initiate backups, restores, or make operations changes to another site's DR servers. This is handled by wither firewall rules or user structures.
Second, Physical access at the DR site needs to be as or more strict than at your primary site. If you don;t own the building (or host services through someone like SunGuard) they should have your units in locked racks. Insist that anyone with access has passed Government C2 or higher clearance, and DO NOT give ANYONE outside your organization passwords to those systems.
Next, If the system allows, the user accounts at the hot site should not be the same at any of your other sites.
Next, You STILL NEED TO ARCHIVE DATA, preferably monthly at worst, weekly may be needed depending on your data set size. Replicated systems can corrupt data, and replicated date sets typically only contain your current data set (no rollback copies). Generally, if your replication system is handling moving data automatically, you can leave your local DR copies in your building in a secure area. Daily backups should be archived to local disks so if you have a DR server failue you still have something to recover from (recovery over the WAN is usually not an option). If you have a fire or disasterous event, your offsite systems is your #1 backup, last month/week's archive is your #2.
I've spent a lot of time working with multiple DR companies, and I've supported medium and enterprise class businesses alike (my current employer has 12 mainframes, a couple hundred AIX servers, and over 2000 x86/64 boxes. DR is a BIG thing around here!) Once you enter enterprise class, we're not typically talking "backups" as you know them, more SAN replication and disk write journaling. Recovering 400TB from backups simply isn't an option in a 24-48 hour recovery model! However, if you've got less than 50TB, you can VERY REASONABLY build a great solution out of DPUs (Data Protection Units) made by Unitrends (unitrends.com). They're totally tapeless systems, can share jobs between multiple units at multiple sites, have secure packet level replication technology, and can use 256bit real time encryption as 8GB/minute (per chassis!) with up to 7 concurrent jobs running (per chassis!). a 50TB environment would use 4-8 DPUs (usually departmentalized) conencted to a SAN, and another 1-2 units offsite to handle replication and DR recovery. They also support P2V and V2P recovery and dissimilar recovery. Fully integrated BareMetal backup (they actually invented and patented that one!), and support for 40 operating systems in one multi-site seamless DR solution. You should REALLY look into them. For small sites (less than 1TB data) you can get into one of these units for around $15K with all the bells and whistles (another 10K for replication). There are NO PER CLIENT OR PER AGENT LICENSING FEES, it's ALL inclusive, the only thing you license is storage. Oh, their Data Vault unit can handle up to 40 sites, and each site can be run by a different company with different accounts, and their data can be logically seperated on the vault making it impossible for one customer to interfere with another's data, you can even lock the vault operator out of the solution. This has been validated by the DOD as a secure solution...
Re: (Score:2)
I suspect this problem was the result of outsourcing of some kind.
Companies figure that there is no way they could possibly get backups right, and don't want to pay to have somebody run them or set them up with an automated solution.
So, they outsource. Of course, they pick the CHEAPEST vendor out there. That vendor probably helps them punch some holes in their firewall so that they can remotely connect, log into servers using administrative passwords, and copy off data. That cheap vendor probably doesn't
Re: (Score:2)
Re: (Score:2)
Yeah, jails are totally insecure; they're full of criminals.
Encrypt it (Score:5, Insightful)
Encrypt your backups.
Don't let your backup system have access to your main system.
Allow your main system write-only access to your backup system, for the sole purpose of delivering new backups.
Re: (Score:3, Insightful)
Ammendum:
Anything else needing to be done would require a admin to do it, and that's the point.
Re: (Score:2)
And if you can't figure out how to do the above, use tarsnap, which is designed around the principle of an untrusted (and potentially conspiring with the NSA) backup system.
Re: (Score:3, Insightful)
Don't let your backup system have access to your main system.
Ah, but this is a large company, and presumably they use 'enterprise' products. Some (most?) of those tend to be secured with rexec equivalent security (nsrexecd and .nsrhosts, coincidence eh?) for jobs executed from the backup server, and any better security is an afterthought. But they have good salesmen, and they're 'enterprise', and they cost a lot of money so they must be safe to use right...
The sad state of affairs is that at many large corp
eggs in multiple baskets (Score:5, Interesting)
1) divide your eggs in at least two baskets, thoughtfully designed to protect their integrity
2) keep your baskets in physically isolated locations
3) take steps to protect your eggs from theft
4) after retrieving your eggs, inspect them for tampering before using them in your souffle
5) purchase insurance for the off chance you get yolk on your face
Re: (Score:3, Funny)
Re: (Score:2)
The "5:erocS" problem (Score:2)
(As for why Slashdot doesn't support Unicode in 2009, see if you can think up a decent explanation, I can't...)
Slashdot is written in English, whose native words do not use characters outside US-ASCII. Characters outside US-ASCII have been used to attack the site in the past [slashdot.org]. Therefore, Slashdot staff put no effort into making sure that characters outside US-ASCII can be used in comments.
Re: (Score:2)
Bulshitt. Slashdot is a technical forum, meaning quite a few non-ASCII characters are potentially needed. Some examples are Â,Â,±,Î",â and even £ and â.
Oh wait, that's "degree symbol","micro symbol", "plus-or-minus","delta","almost-equal", and "pound currency" and "Euro currency" symbols.
Re: (Score:2)
Oh wait, that's "degree symbol","micro symbol", "plus-or-minus","delta","almost-equal", and "pound currency" and "Euro currency" symbols.
Hmm...
° ==
µ ==
± == ±
Δ ==
£ == £
€ == €
Looks like slashcode is fucking up HTML entities as well, which (if memory serves) don't provide much in the way of "attack vectors."
NICE JOB DUDES!
Why were your backup servers (Score:5, Insightful)
Re:Why were your backup servers (Score:5, Interesting)
There are a couple ways to 'automate' your backups being "offline" but still automated; not perfect, but better than what they apparently had.
Server on the Internet, with a private subnet behind said server where your backup server(s) are. Internal interface on server must have absolutely zero services running, and the connection must not be routable.
However, backup servers also need to be not kept "online" when not backing up. You can:
* Keep them powered off and only power them on (automatically) when you're doing backups (which occur on a schedule from the main server).
* Keep them on, but with the network interface down.
* If the backup servers use external storage but are kept on, have those on a separate power unit that powers up only on schedule as well.
Personally, this sounds like an "inside job": it's malicious and performed with a great deal of knowledge of the operation. That speaks of a disgruntled employee to me, not some kiddie at it for kicks. There are a lot of steps which can be taken, but realistically, you're not going to be able to deal with something like this unless you've got your data backups completely 'unattached'.
Re: (Score:2, Interesting)
"Their" is not singular, you cretin. Use "his": it's a perfectly fucking good word.
Oh, come off it. [bartleby.com] Quit wasting the world's time.
Isolate the Backup Servers (Score:2, Insightful)
Why were your backup servers accessible from the outside network? Your backup servers are arguably even more valuable than your production servers. They should be behind yet another firewall. You can even have the production servers connect to them through a separate network interface. (Network interfaces and Cat5 cables are cheap.) If you are really paranoid, you can create folders where the server can upload data, but can't erase or overwrite what it has uploaded.
Re: (Score:2)
offsite backup much?
Re: (Score:2)
Have onsite backup behind it's own firewall, accessible through a separate network interface AND offsite backup. Recovery time using the onsite backup will be much faster, and you'll have additional redundancy.
Prevention for exploit via backups (Score:5, Funny)
Re: (Score:2)
That's the total paradox - do backups and be screwed or don't and be screwed.
Well, for what it's worth, if you really must, use a different OS to run the backup. This other OS is always not networked, always booted from a read-only drive, or whatever is required to keep it from being corrupted. You have to tolerate some downtime though, so maybe copy your working files to another machine if possible. Security and reliability are a bitch, but combating entropy has its price.
Obvious Oversight by WHT (Score:5, Informative)
There was a very blatant oversight and an unfortunate assumption on the part of WHT and iNET Interactive.
They quite obviously overlooked the fact that the WHT servers (and ONLY the WHT servers) would ever need routine access to the backup servers. Therefore it was an obvious security hole that could have been plugged by restricting traffic through iptables to only iNET-affiliated IPs. Any teleworkers who needed access should simply use a VPN to iNet's offices if they really need access to the backup systems. If under some extreme circumstance (such as the loss of a database) an outside party needs access to the backup servers, the system admin can then add an exception under iptables.
And on that note, the other incredibly thoughtless assumption was that any traffic coming from the backup servers would be approved traffic. So once the attacker gained access to the backup servers, the database servers were one insecure hop away.
I think this proves the following very important points to the entire IT industry:
1) Internal infrastructure should remain just that - internal! Restrictions should always be put in place as to who can (or can't) access a system.
2) No traffic can be guaranteed authorized or authentic. It's one thing to add an SSH keyfile to your home servers, but in an enterprise environment everything must be highly scrutinized. It's no longer a matter of protecting systems from users - it's now a matter of protecting systems from other systems as well.
I was personally affected by the loss of information at WHT and while it's annoying, it's a fact of life and can't be undone. All that's left now is to pick up the pieces, secure the site as best they can and move on with lessons learned.
Simple solution.... offline backups (Score:2)
Re: (Score:2, Interesting)
If your system is compromised for an extended period of time, and you don't know about it, your backups can still get tainted.
i.e. the hacker might modify your backup process in a subtle way, like getting random bits corrupted on the tape, sufficient to make it useless for restoring.
Hardly anyone actually completely tests their backups, to make sure they'll work.
After 6-12 months, the hacker hoses the real DB, not expecting their trap, you try to load from backup from physical media, only to make th
Re: (Score:2)
Re: (Score:2)
Anything man can create, man can fsck up. Someone motivated enough could not only set up the system to silently corrupt the backups, but modify the testing system to report "OK" anyway. Set up the system, wait one full backup cycle, nuke the original, and the target is hosed.
Most backup solutions are designed to protect against accid
Lock that shit down. (Score:2)
It is also an reasonable initial assumption when setting up shop, that backups of any kind should be situated in a more bomb/fire/quake/flood/theft proof place than your main operation. That goes for external intrusions to your hot failover systems too.
A: The same way you protect any other resource (Score:2)
Why were all the backups online? (Score:5, Interesting)
I understand why the most recent backup would be online, but I assumed it was standard procedure to have the remaining archive offline.
I use duplicity for offisite and nfs for onsite (Score:2)
I couldn't care less if any of those machines got compromised. Ok, the onsite nfs backup servers would be kind of weird, since they are behind my firewall. So if they compromise my backup servers they could as well compromise any other machine on that lan. Offsite backups are stored via ftp using Duplicity. Even if they compromise the backup servers they would still have to break my n-digit random passprase in order to look into it. And even if they took a look inside my backups it still wouldn't compromise
Why was this possible? (Score:4, Insightful)
I don't understand. The attackers gained access to the database...through the backup servers? That leaves me with two questions:
1. Why were the backup servers accessible to the attacker?
2. Why was the database accessible from the backup servers?
It seems to me that the only way you need to access the backup servers is through some mechanism that allows you to transfer data to and from them. A single open port, which you need a password (or key) to use seems all that should be exposed. That shouldn't be too hard to secure.
It also seems to me that the backup server has no business accessing the database, and therefore shouldn't be able to access the database.
Unless, of course, the way the system works is that the backup server accesses the production server to retrieve the data from it. That doesn't seem the most obvious design to me, but it would at least explain why the backup server could access the database. Maybe that is a good reason not to design the system that way (on the other hand, it saves complexity on the production server, which is good). At any rate, it doesn't answer the first question, which is why the attackers were able to access the backup server.
My sympathy goes out to the WHT administrators. Good luck on recovering from this and figuring out what went wrong. I hope you will keep us posted, so that we can all learn from this incident.
Re: (Score:2, Insightful)
Unless, of course, the way the system works is that the backup server accesses the production server to retrieve the data from it. That doesn't seem the most obvious design to me, but it would at least explain why the backup server could access the database. Maybe that is a good reason not to design the system that way
That is a rather common design in backup systems, and there are good reasons for it: mainly, it allows securing the backup server better, especially in the (common) situation where one backup server handles several production machines.
If the backup servers offer only read-only access to anyone over the net (or not even that if recovery from backup is rare enough to be initiated from backup server console), it can be secured much easier than when if offers write access to clients backing themselves up.
[...] it doesn't answer the first question, which is why the attackers were able to access the backup server.
You ar
Several layers (Score:2)
We have a back up server the hosting company provides, and that is on a seperate subnet, no external facing ip. In addtion, we have an FTP server at another hosting co, and finally the emergancy back up in the office, which gets backed up to an external hdd and DVD.
RTFA, they don't know much about security (Score:2)
"Passwords are hashed with salt. It would be an unprecedented event to reverse engineer our passwords."
Yea, sure, like most users will choose good passwords when registering for such forums. Having the salt will slow down an attack against all passwords at the same time, but that's all. A normal brute force attack even against a "light" dictionary will probably reveal enough passwords for any evil purpose (like spam).
And really for a forum that's called "WebHostingTalk" they should be better at, well, WebHo
Insider (Score:2, Funny)
Redundancy? (Score:2)
You don't do backups, you do redundancy. If your IT model depends on one server doing its thing right, you have already gone down the path the doom.
Re: (Score:2)
You don't do backups, you do redundancy. If your IT model depends on one server doing its thing right, you have already gone down the path the doom.
You need point in time backups of all code and user data as well. Even the brightest people delete or alter the wrong thing every once in a while.
Can I hire him? (Score:3, Funny)
Dunno, but our backups are so secure that even the DBAs are usually unable to restore them. We might need this evil hacker to teach them a trick or two.
It's obvious (Score:2)
It is goes off site encrypt it. That's what I do.
You have to compress the data first as the tape drive can't do its own compression on encrypted data.
The only exception is content that's meant to be public in the first place.
Re: (Score:3, Interesting)
The one thing about encryption -- key management is just as important as the algorithm used. A business has to figure out how they are going to manage keys. Are they going to use a passphrase that only the backup admins will know, or are they going to use some type of RSA key functionality? If private keys are used, how are they backed up?
There are a lot of factors involved, and one important thing is not depending on one single person as one doesn't depend on one single server. There should be some typ
Re: (Score:2)
If private keys are used, how are they backed up?
This is less of a problem than it sounds. Your private keys don't change very often (in many cases, never); you don't need to access your backup-reading keys very often (ideally, never); and your private keys are small. It's a PITA to store your daily backups by printing them out and storing the paper in a safe deposit box; but that's an entirely reasonable thing to do with your private keys.
Backups (Score:3, Insightful)
Tape drive (or even other external device) with encrypted data. There, problem solved. Do you see why such devices/features are standard on anything that has the word "backup" on it?
However, if you insist on having servers running all day long that you want to backup to, then at least make sure they are a million times more secure than your production servers - as the name suggests, they are your BACKUP if anything goes wrong. This means - encrypted data, locked-down networking (i.e. absolute minimum necessary - one port, one user, one ultra-secure key), proper permissioning (it might not be a bad idea to set the permissions so that you can append to a file but can't read it, thus forcing you to have physical access in order to restore any data from backup, or certainly that you can't overwrite existing files) and physical security (i.e. properly hosted in a decent hosting outfit and/or securely placed in an offsite location where they can't be got at - even if this is in a secure cage in a branch office).
And letting the backup server have ANY permissions on your local network is just stupid. Push, don't pull. Tell the backup server what to backup, don't let it pick and choose and require access back to your network - it's a backup device not "just another machine". A simple "nothing outbound" firewall rule on that machine would have stopped them coming back to you, providing the backup was encrypted (I'm assuming they actually piggy-backed onto your network than stole the db passwords from your backups). And the backup should ALWAYS be encrypted because it might well contain information such as your passwords in it, especially so if you are storing other people's data.
Yeah, it costs money to do this properly, but that's the price you pay for not losing thousands of people's data. I imagine the kick-back from that data loss will run to more than the price of a tape drive or two in the long run. What they had was NOT a backup. It was a rapid-restore machine. That's fine to have *as well as* a backup, but no better than hanging a 250Gb USB drive off the database server (in fact, worse, because that machine was able to be remotely-compromised).
They're not backups unless they're TESTED (Score:2)
it might not be a bad idea to set the permissions so that you can append to a file but can't read it, thus forcing you to have physical access in order to restore any data from backup
Even for a restore to a sandbox system for testing the backup media?
My Solution (Score:2)
My little layout. (Score:2)
For backups where I work, we've got a few different systems in place. We've done away with tape backups long ago, and our primary backup system is a disk-based system. Twelve drives in a RAID 6+0 for maximum availability and ability to recover from disk issues. This system keeps a daily snap shot from the past week, a weekly snap shot for the past month, and a monthly snap shot for the past six months. We then place the most recent daily snap shot on a USB external disk drive, which we carry off-site ea
Clarification (Score:2)
Okay, let's sort something out here. They didn't get hacked by "backups," which would imply that somehow the backups themselves got trojaned or modified to cause compromise. They got hacked normally; the backup servers were just the means by which this took place. This is nothing new or exotic, since backup software is software like any other, and prone to vulnerability. In fact, there have been a lot of issues found with software from many vendors. Unfortunately, companies rarely patch their backup sy
What do encrypted backups solve? (Score:2, Insightful)
A lot of people are suggestion that backups be encrypted and I assume they mean the actual files/volumes, but I *really* don't see the usefulness of this case. Encryption backups may protect the data from being stolen, but it isn't going to protect it from being *wiped* in most cases. If you have root access then assumedly there's also raw access to device the backups reside on, in which case you can still nuke it with something as simple as "dd if=/dev/zero of=/dev/backup"
Now maybe if the backup server fur
Open Air Policy for Backups (Score:2)
Borrowing from the military mindset this is my suggestion:
"Any online backup locations shall have a physical break in the networking infrastructure that must be phsyically connected as needed. Once operations are complete the physical connection is again broke until it is needed."
(IMPLEMENTATION A)
From an implementation standpoint what you have at the core switching rack is a RED cable in the switch that is the connection to the backup server (onsite, i.e. disk to disk) that is unplugged at all times except
I love it... (Score:2)
I lost all respect for WHT years ago. It went down the tubes after Chicken left. Note they're well in with Rackspace who capitulated to SCO. Couldn't happen to nicer people.
Any problems with my current solution? (Score:2)
My offsite backup server is currently running Ubuntu Server with full disk encryption. The only access it has is with key based SSH on a non-standard port, restricted to a specific subnet. There are no ports forwarded to this machine; instead, it initiates a VPN connection from it's location directly to my local network. It gets a static IP on the VPN subnet, and that IP is the only subnet SSH will listen to.
I use a combination of Rsync over SSH and rsnapshot server side.
I'm not storing any hyper-critica
how to categorize your backups (Score:2)
Treat your backups, and your backup servers, with every bit of seriousness that you treat your main systems. Use the same level of monitoring, management, and security. Investing good practice in your system while slacking on your backup only narrows the attack window, it doesn't tighten the security - your security is only as strong as your weakest link.
And someone needs to add a filter to slashdot to ban IPs that submit >10kbyte posts with "obama" in them in every single new article on slashdot. I'm
Backup Servers == Gmail storage (Score:2)
I am pretty sure these people using Gmail's huge storage as their backup server.
I don't see any other reason why their backup server was accessible to the outside world.
And if you are thinking about they got access to the production boxes by hacking the backup server(gmail)
- it's pretty simple. They have emailed their production server passwords to their GMail ID, just so that
they don't forget it.
Re: (Score:2)
Re: (Score:2)
Unfortunately, backups normally inherently contain sensitive files like /etc/shadow which could be used by an attacker to gain access to the live server (Brute force attack)
This is why backups should be encrypted.
Re: (Score:2)
Re: (Score:2)
Don't forgot user passwords stored in cleartext by Subversion and unencrypted SSH passwords.
Err... as I was saying, the fact that people have sensitive information on their systems is why backups should be encrypted...
Re: (Score:3, Insightful)
i dont want thresholds. i want to be able to see valid comments even if some fanboi mods someone's valid, insightful comment -1 due to fanboi spirit.