software raid5 under Linux (Score 2, Informative)

I converted nearly our whole house network to software raid at the start of this year: Big RAID5s for our fileserver, backupserver, netserver and videoserver, and smaller RAID2s for our firewall and workstation boxes.

Overall, I'm very happy with it -- no more rebuilding from scratch every time a boot disk blows!! :) I'd started converting to SCSI boot disks everywhere, but a pair of software RAID2 IDE drives gives me a much better sense of security. My workstation did in fact blow a boot drive a month ago, and rather than being an emergency, I just ambled into Fry's after a week or two and bought a replacement and rebuilt the raid. No muss, no fuss. Feels like living in the third millenium!

I did learn various things the hard way that the HOWTOs don't warn of.

Note that you can't boot off RAID5, only RAID2. The hack they mention of putting /boot on RAID2 and everything else on RAID5 is not worth it with today's drive sizes. Give yourself a 2-16GB RAID2 with a complete bootable system on it, and save yourself mucho grief at very little proportionate cost in disk space.

As of kernel 2.4x, at least, the linux software RAID5 autorecovery is workable but less robust than one might like in the face of serious problems: I had one RAID5 setup totally destroyed because the hardware was flaky leading to constant reboots while RAID5 reconstruction was just underway. After awhile the kernel got confused about the order of the disks (which shouldn't matter, but apparently did) and the whole thing went into a Death Spiral. Lesson: If you're sure the problem is just one flaky disk, feel free to just swap in another and reboot. But if you are in any doubt, play it safe: Switch off RAID autodetect first thing (by fdisk'ing the partition type from FD back to 82. Get the hardware stable, rebuild the RAID by hand, then switch everything back to FD.

RAID5 is a comparative pain in the ass to work with vs RAID2, because under RAID2 any of the partitions can be mounted normally as a non-RAID drive in an emergency, getting you back on the air fast, but not so with RAID5. (You'll want a live Linux CD with a RAID-supporting kernel, likely. Knoppix &tc don't yet ship this way.) So only use RAID5 if the extra space really matters -- the big servers.

BTW: One of the reasons I like software RAID over hardware: If you have hardware RAID and the controller blows and you can't find a matching model, you may be stuck reverse-engineering their RAID scheme to recover your data. No worry about that under software RAID.)

I tested automatic failover to hot spare disks under the kernel, and it worked perfectly for me in a handful of tests. For whatever that's worth.

Do keep an eye on /proc/mdstats readout of your RAID system health. If you're asleep at the wheel and don't notice anything until enough disks fail to bring the whole system down, you haven't gained much. I have a crontab-driven set of Perl scripts which check all sorts of things weekly to minutely and email me if they look wrong: Checking for failed RAID drives is one of the things they do. If you don't have a comprehensive solution like that, the raidtools2 package has an ad-hoc solution specifically to email you on drive failure. USE IT.

FWIW, here's the system I've evolved for partitioning disks in such systems:

• First partition: One cylinder (the innermost one): Ext FS containing a THIS_DISK file in which I record when and why I bought the drive and any interesting history it has had. In an emergency when you're suddenly shuffling eight hot drives plus a couple spares plus a dead and replacement motherboard &tc, you WILL lose track of which disk was doing what. This little partition will save you a lot of grief.

• Second partition: Swap, in the outermost (lowest numbered) cylinders -- because these give the fastest transfer rates, up to about a 50% advantage. Putting swap on every disk lets the kernel stripe swaps across all available drives for additional parallellism and speed, and also ensures you've got adequate swap no matter how you configure in an emergency.

• Third partition: RAID2, a complete bootable Linux system. With this mirrored on every drive in the system, you're guaranteed able to get up again fast no matter what goes wrong. Thus, FD partition type.

• Fourth partition: RAID5, your serious filestore space, occupying the vast majority of the disk.

Oh, I presume it goes without saying that if you're going to build a honker fileserver like this, you want gigabit ethernet connecting it to its clients. There's no reason to run 10 or 100Mbit ethernet in a serious LAN at today's prices.

Good luck!

CrT