NetBSD's Real-Time Network Backup 166
jschauma writes "One of NetBSD's developers, der Mouse, was interviewed by DaemonNews about his real-time network backup system (originally presented at BSDCan 2005), where changes to your local filesystem are automatically propagated to a backup server. In his interview der Mouse tells about his idea, how it works, and of course, how cool it is."
Correct me if I'm wrong (Score:5, Interesting)
Re:Correct me if I'm wrong (Score:4, Interesting)
but Multics had this in 1970 or so.... (Score:2, Insightful)
Re:Correct me if I'm wrong (Score:4, Funny)
Yes, but do you want to sell your children and a kidney for a Solaris server?
Re:Correct me if I'm wrong (Score:2)
Re:Correct me if I'm wrong (Score:2)
Network Appliance (Score:2, Informative)
Linux too (Score:2)
B.S. D? (Score:2, Interesting)
This is indeed very neat, but isn't it sorta how transactional databases have been working?
I also don't see how this solution is effectively any better than RAID... If anything, a backup server is more expensive than a second hard drive for a RAID system (though it may pay off eventually). I'd think the backup server would need to be maintained as well... and if your backup ever fails, it seems like it would require
Re:B.S. D? (Score:5, Insightful)
Re:B.S. D? (Score:5, Funny)
Termites, on the other hand...
Re:B.S. D? (Score:3, Informative)
(I read this in a linux software RAID tutorial once)
Re:B.S. D? (Score:2)
RAID is redundancy, not backup.
Re:B.S. D? (Score:2)
accidental deletion? (Score:3, Insightful)
Still, you'd want to be careful with this, it would suck to back up all the temp files generated by random processes.
Re:accidental deletion? (Score:2)
In UNIX systems, all temp files usually reside in /tmp, which need not be on a RAID partition (unless you want those processes to stay up when you lose one of your drives).
Re:accidental deletion? (Score:2, Interesting)
Failing that, something like OpenAFS with mirrored globally addressable volumes that can work at the system level rather than user level. Sure you can use IP security for OpenAFS, and a few brave folks have even gotten ne
Re:B.S. D? (Score:4, Insightful)
take 10 small servers that do the front end grunt work with 2-3 backup servers that keep complete working images of the servers and have access to their data..
a front end server dies service can roll over to a backend until the front is replaced and is quickly made jsut like the orginal a backend dies and you have a second and if all the backups die then you still have the front end to recreate the backups..
you don't normaly consider the bandwith costs as they are typicaly on a highspeed network between them and it offers you the option of replication over diffrent connections and areas..
all redundent disks help with is if a disk dies not if ram or cpu fails
some people have gotten too attached to their physical backups and tapes - personaly a backup is worthless if i can't have live access to it in a few min even if i am not physicaly at the point of failure..
this isn't particulary useful for small setups but is great for mid to large scale setups and offers plenty of room to grow.
Re:B.S. D? (Score:2)
This is a pretty shakey solution at best. You could change the default bootloader config and tell the machine to reboot using the chosen image but if there were the slightest problem that required a tweak on the system it may never come up and give network access again. If you replicate using a hybrid system where some portion of the configuration is already setup on the backup ser
Re:B.S. D? (Score:2, Interesting)
In a RAID cabinet, you have a bunch of identical drives, most likely purchased together, too. Then you submit them to an essentially identical environment and operating history. Barring a defect, and assuming wearout-type phenomena, something bad may well happen.
The weakest drive fails first. Power down the RAID box to replace the bad drive, so you can bring it back up and restore the data. The stress o
Re:B.S. D? (Score:3, Insightful)
well, no. enterprise level raid has spinning spares and hotswappable everything. you can lose two drives and still be running as long as you get those replacements in there before number 3 goes. been there, and yes, it happened when we shut down for maintenance. In the real world catastrophic failure happens. Raid is not used as a backup usually, it is used to keep data availabl
Re:B.S. D? (Score:5, Insightful)
Hotswappable RAID has saved my servers on more than one occasion. Likewise, the servers have also been saved by tape backups. RAID5, tape backups, and data replication all have different pros and cons.
I think it is incorrect to say RAID5 is not acceptable in any backup strategy. The more chances you get at data redundancy, recovery, and failover, the better off your organization.
Re:B.S. D? (Score:5, Insightful)
I'll comment on this later...
The weakest drive fails first. Power down the RAID box to replace the bad drive...
OK, this is where I start getting dizzy. If their data is valuable enough to have RAID, why were they such cheap bastards that they didn't get hot-swap drives? I've worked in a LOT of places that have RAID systems, and three of my own servers have RAID, yet to date, none of them were anything but hot-swap. Additionally, with a small amount of intelligence and a few extra dollars, the administrator always puts in a hot-standby drive that will automatically take over if a drive fails, allowing for the failed drive to be replaced at a more convenient time than 1:30am without sacrificing the redundancy. Sysadmins running really critical systems will often have multiple hot-standby drives.
The stress of the power-down and restart is enough to kill the second-weakest drive.
Now, see, here's the funny part. When you spend the bucks for SCA hot-swap drives, you actually get drives of decent enough quality that this is very rarely a problem. Even if you did have to shut the array down, which you won't because you bought proper hardware.
enough so that they've quit using RAID as "backup"
Further evidence of idiocy. RAID is not a backup. RAID allows you to keep running in the event of a specific type of hardware failure. But that is all it protects you from. Backups are still just as critical as they were before you had RAID. Anyone who uses a RAID array instead of proper backups deserves to have their data sacrificed to the gods of entropy, shortly followed by their own careers.
As for my delayed comment on the first sentence... Well, I suggest you get smarter friends.
Re:B.S. D? (Score:1)
From what I gathered in the article, it isn't similar to a database's logs because it just mirrors the writes, without saving the old disk image. Although it sounds like it would be fairly easy to just save the packet stream and have the ability to replay your disk image to any particular moment in the past.
That would be much better than RAID. After all, back
Re:B.S. D? (Score:2)
It's kind of like the SunRay system. Yeah, the idea is neat, but the architecture is a network-clogging, CPU-leeching nightmar
Re:B.S. D? (Score:2)
Re:B.S. D? (Score:2)
Re:B.S. D? (Score:2)
2. Off site backups. A second server located at another office just in case of Earthquake, Fire, Flood, Hurricane, Tornado, or some other disaster.
Raid doesn't replace backups. With encryption you could keep your backup server at a co-location facility, branch office, or home. Handy if the worst does happen.
Another option would be for a local consulting firm offer this as part of there ser
Re:B.S. D? (Score:2, Interesting)
I only skimmed TFA and it's not clear to me how like or unlike Windows' Distributed File System it is, but I'll give you a quick picture of what DFS does fo
Re:B.S. D? (Score:2)
Unless you have hotswap ability, if a hard drive fails, you still have to power it down to remove it. If you have a second server up and running, you won't have any downtime other than changing an IP address.
Sure it might be only 20 minutes to swap hard drives tops, but a server down during business hours is still a pain.
Neat. (Score:4, Interesting)
Here is the idea behind the setup I am currently using: Easy Automated Snapshot-Style Backups with Linux and Rsync [mikerubel.org].
Re:Neat. (Score:5, Interesting)
Re:Neat. (Score:2)
I should look into something like dirvish though to replace my current homemade 'backupd' which basically does the same thing with less flexibility.
Re:Neat. (Score:2)
Re:Neat. (Score:1)
Re:Neat. (Score:2)
Re:Neat. (Score:2)
How is this different from Windows VSS? (Score:2, Interesting)
Re:How is this different from Windows VSS? (Score:1)
Re:How is this different from Windows VSS? (Score:4, Informative)
I'm a little ashamed that I know that, but it's true.
Re:How is this different from Windows VSS? (Score:2)
Good idea, but there has to be a better way (Score:4, Interesting)
Personally I'd like to see something like the MS filesystem in development that allows SQL calls to be run against it (not sure if there's any other filesystems that are similar). Query every 5 minutes for changed data that fits the backup parameters (within the system dir, the user's home dir, certain filetypes) and then transfer the data as the network isn't being used.
That would achieve the same thing, but more flexibly and without affecting normal use.
Re:Good idea, but there has to be a better way (Score:3, Interesting)
Query every 5 minutes for changed data that fits the backup parameters (within the system dir, the user's home dir, certain filetypes) and then transfer the data as the network isn't being used.
Then you loose the realtimeness.
Re:Good idea, but there has to be a better way (Score:3, Funny)
That it runs away and hides where nobody can find it. So he also does lose it, but only because he loosed it.
Re:Good idea, but there has to be a better way (Score:2)
Re:Good idea, but there has to be a better way (Score:5, Interesting)
Also, there's "GEOM gate" on FreeBSD: http://garage.freebsd.pl/GEOM_Gate.pdf [freebsd.pl]
For other cool stuff with GEOM see here [events.ccc.de] and here [freebsd.org]. See also this discussion thread [freebsd.org] about ggate's limits.
Re:Good idea, but there has to be a better way (Score:3, Informative)
Re:Good idea, but there has to be a better way (Score:1)
Unless I'm reading you wrong here, with a 5 minute delay you can already do this with rsync, a shell script and a cron job. According to the article this guy is doing it in near real time across the network (from what I can tell) by intercepting the write calls to the file system driver(s).
Not sure how else yo
Re:Good idea, but there has to be a better way (Score:3, Interesting)
I want dual transactions. 1 for onsite and 1 for offsite. I'm not even interested in encrypting the data. I need to be able to kill my onsite immediately a
DragonFlyBSD has the better way... (Score:2, Informative)
Re:Good idea, but there has to be a better way (Score:2, Insightful)
No, but when it isn't frustrating, it's hysterical watching them try to corner the market.
Re:Good idea, but there has to be a better way (Score:2)
Re:Good idea, but there has to be a better way (Score:2)
As for
Re:Good idea, but there has to be a better way (Score:2)
If you want to get fancy, you could use ggate to create two network disks/partitions, and graid3 to create a RAID3 array. But the performance probably would be all that great.
Re:Good idea, but there has to be a better way (Score:2)
DoubleTake (Score:2, Insightful)
Cool, but not new (Score:2, Informative)
Re:Cool, but not new (Score:2)
Der Mouse? (Score:2)
Re:Der Mouse? (Score:3, Informative)
No, that would be "der Maus"
You crazy Americans -- Hier ist der Maus! [wikipedia.org]
Re:Der Mouse? (Score:2)
Not to be confused with Die Fledermaus!
Spoooon!
Re: (Score:1)
Re:Der Mouse? (Score:2)
He barely came up to my waist.
Seriously though, it's been 15 years or so since I studied German, so thanks for pointing it out, it looked wrong to me, but I couldn't be bothered to look up the gender.
Re:Der Mouse? (Score:2, Informative)
According to the article, he's canadian.
Re:Its his childish attempt to mock Theo. (Score:2)
Passionate hatred can give meaning and purpose to an empty life. -- Eric Hoffer
Cute coincidence.
--Ryv
Silly OBSD Troll (Re:... mock Theo.) (Score:3, Informative)
NBD? (Score:4, Informative)
There are pretty mature commercial tools for this stuff, as well - Veritas' VVR replication comes to mind.
Re:NBD? (Score:4, Insightful)
It doesn't compare at all.
From my (quick) scan of the article - think of NBD as a replacement for NFS (well, sorta) & this as a sort of network RAID (kinda, not realtime).
They're not really alike - for linux drbd [drbd.org] is probably closer.
Re:NBD? (Score:2)
Understandable; the people interested in remote replication mostly aren't interested in doing it with Alpha software.
Re:NBD? (Score:2)
Re:NBD? (Score:2)
RAID is not a backup solution!
To answer the OP's question, it doesn't compare at all. NBD lets you export drives over the network so that they show up as block devices on remote systems (meaning you can do raw operations on them, use LVM, etc.); this, on the other hand, replicates changes to another filesystem.
At first glance, this might not seem very effective for backup if deletes are replicated as well. That said, the benefit (as with replication in MySQL) seems (to me) to be that you can
Re:NBD? (Score:2, Informative)
Point In Time Recovery (Score:1)
Reinventing the wheel? (Score:2)
Re:Reinventing the wheel? (Score:1)
Re:Reinventing the wheel? (Score:2)
I'm sure having expensive HBA's can give you a lot better performance(as well as the ability to boot from an iSCSI drive), but j
It All Makes Sense Except... (Score:1)
RTFA (Score:2)
Thus the first time you ran this thing it would copy the whole disk image to the backup server. After that subsequent writes would be the only output.
web-based (Score:1)
Re:web-based (Score:2)
Re:web-based (Score:2)
Re:web-based (Score:2)
Re:web-based (Score:2)
I've done a few test runs which was enough to let me know I had to look into getting a multi-TB server farm before I could open it to the public. I've been trying to get investors for that (it costs around $250/mo per TB o
DRBD (Score:1, Informative)
From the website:
DRBD is a block device which is designed to build high availability clusters. This is done by mirroring a whole block device via (a dedicated) network. You could see it as a network raid-1.
Each device (DRBD provides more than one of these devices) has a state, which can be 'primary' or 'secondary'. On the node with the primary device the application is supposed to run and to access the device (/dev/drbdX; used to be
delayed backups are still useful (Score:2)
This isn't to say that instant backups wouldn't be nice for failover architectures, though. I just don't deal with systems that large, yet.
Re:delayed backups are still useful (Score:1)
protection (Score:2, Insightful)
Real-time accidental deletion, too. (Score:4, Insightful)
As always, RAID (and now this) is not a backup solution.
Re:Real-time accidental deletion, too. (Score:2)
Just today I had to recover the Inbox of a user who deleted a message but didn't know who sent it, when it arrived, or what the subject of it was. He also wasn't sure when he deleted it, so I had to do the restore twice.
I keep a lot of data backed up on disk, rsynched once a day. Some data I even back up once an hour. It doesn't cost anyt
oops! (Score:5, Funny)
Journaling? Use that... (Score:2)
Full rescan (Score:2)
UGH!
That kills it for me.
Linux posts have missed FUSE mirroring filesystems (Score:2)
And, yes, RAID is a very good solution if you've got the money and are smart enough to recognize when a disk fails...
Automatically replicate screw-ups too? (Score:2)
Re:Automatically replicate screw-ups too? (Score:2)
Fools, BSD is dea . . . (Score:2)
Isn't this the same as... (Score:2)
in crontab?
Being able to rollback local filesystems (Score:2)
Re:Invented by Larry Robertson in the mid 1980s (Score:2)
Yes vms volume shadowing works really well, compared with the different techniques in use with *nix. I was involved with running the traffic management systems here in Victoria off vms clusters with various types of shadowing, both through decnet and scsi (dssi).
Somebody told me a story about a vms system hit during the Oklahoma City bombing. The other half of the cluster was on the other side of the city.
Re:Dealing with Loss (Score:2)
Funny