First, I have to say I'm truly awed that you have that much data. You must really love collecting pr0n -- er, have a lot of sound and video files. I recently had to set up two new servers. One is for business, and one is for personal data. For both, I used RAID 5. They run NFS and Samba, with different directories shared as needed to other systems. RAID 5 is EXTREMELY simple to set up (it's a one line command, once you install mdadm, which, on Debian, installed like a dream), and I'd just suggest Googli
You can also add more spares and more devices with mdadm, or replace faulty devices (not hot swappable, unless you have special hardware, and I don't even know if Linux supports that).
Here's another tip: If you're using Linux software RAID, carve your drives into multiple partitions, build RAID arrays over those, then use LVM to weld them into a larger pool of storage. It may seem silly to break the drives up into paritions, just to put them back together again, but it buys you a great deal of flexibility down the road.
Suppose, for example, that you had three 500GB drives in a RAID-5 configuration, no hot spare. That gives you 1TB of usable storage. Now suppose you're just about out of space, and you want to add another drive. How do you do it? In order to construct a new, four-disk array, you have to destroy the current array. That means you need to back up your data so that you can restore it to the new array. If there were a cheap and convenient backup solution for storing nearly a terabyte, this topic wouldn't even come up.
If, instead, you had cut each 500GB drive into ten 50GB partitions, created ten RAID-5 arrays (each of three 50GB partitions) and then used LVM to place them all into a single volume group, when it comes time to upgrade, you will have another option. As long as you have free space at least equal in size to on of the individual RAID arrays, you can use 'pvmove' to instruct LVM to migrate all of the data off of one array, then take that array down, rebuild it with a fourth partition from the new disk, then add it back into the volume group. Do that for each array in turn and at the end of the process you'll have 1.5TB, and not only will all of your data be safely intact, your storage will have been fully available for reading and writing the whole time!
Note that this process isn't particularly fast. I did it when I added a fifth 200GB disk to my file server, and it took nearly a week to complete. A backup and restore would have been faster (assuing I had something to back up to!). But it only took about 30 minutes of my time to write the script that performed the process and then I just let it run, checking on it occasionally. And my kids could watch movies the whole time.
For anyone who's interested in trying it, the basic steps to reconstruct an array are as follows. This example will assume we're rebuilding/dev/md3, which is composed of/dev/hda3,/dev/hdc3 and/dev/hde3 and will be augmented with/dev/hdg3
pvmove/dev/md3 # Move all data off of/dev/md3
vgreduce vg/dev/md3 # Remove/dev/md3 from the volume group
pvremove/dev/md3 # Remove the LVM signature from/dev/md3
mdadm --stop/dev/md3 # Stop the array
mdadm --zero-superblock/dev/md3 # Remove the md signature from the disk
mdadm --create/dev/md3 --level=5 --raid-devices=4/dev/hda3/dev/hdc3/dev/hde3/dev/hdg3 # Create the new array
pvcreate/dev/md3 # Prepare/dev/md3 for LVM use
vgextend vg/dev/md3 # Add/dev/md3 into the array
In order to make this easy, you want to make sure that you have at least one array's worth of space not only unused, but unassigned to any logical volumes. I find it's a good idea to keep about about 1.5 times that much unallocated. Then, when I run out of room in some volume, I just add the 0.5 to the logical volume, and then set about getting more storage to add in.
I'm thinking about rebuilding my home server which mainly is a mythtv backend. Currently using LVM to get bigger partitions than I have drives (my tv partition is at 480GB right now) and was thinking about RAIDing them for added security, but was put off by the fact you can't easily extend a raid.
I'll follow your tip, but will probably add boot and swap partitions to every drive. Not because it's needed on every drive, just to keep the setup consistent over all drives.
I'll follow your tip, but will probably add boot and swap partitions to every drive. Not because it's needed on every drive, just to keep the setup consistent over all drives.
I did that too. I also had a couple of drives which were actually slightly bigger than 200GB, so I used the extra space for/root partitions (mirrored).
This will work with those newfangled extended partitions, right? Didn't use those since the days we dual booted DOS and OS/2.
Bear with me - I'm still recollecting parts of my just exploded brain:-)
I can see you wouldn't want to extend a degraded raid though. OTOH, if one knows that one can reconfigure it later, no trouble just replacing the disk and then rework one partition after the other, at one's leisure.
What I'm trying to work out right now is this: I read the total size of a raid is (number of drives -1)*size of smallest drive. Right now there's 4 drives in that computer - 120,
I hope you excuse me for intruding in the thread, but I find this to be a quite interesting optimization problem. You ultimately want to optimize, T, the total size of the LVM of arrays, where:
T = Sum_{j=1}^{m}[(N_{j}-1) * S_{j}]
where Sum_{j=1}^{m} is the sum over m arrays, N_{j} is the number of partitions in the jth array and S_{j} is the size of a partition in the jth array. One thing that this equation shows us, is that each array in the LVM does not need to be the same size!
The other constraint in
Please ignore item 4; It was scratch work that I forgot to delete!
In any rate, with the 3 array LVM I outline above you get a total of 180+180+80 = 440 GB array with 40 and 90 GB left-over on two of your disks.
Thanks for the lead. I started with 40GB as 'ideal' size because it'll fit best into my disc sizes. I realized that even a 2 drive raid is ok, because it can still work - the more partitions, the better, though.
I put 3 4-drive partitions at 120GB each, 1 3-drive partition at 80, and a 2-drive partition at 40, with 50 left. Gets me 480GB usable out of 730.
I could even scale down to 20GB partitions - not to optimize usage, but to allow for smaller partitions which will help in extending/moving to another dr
Seems a rather roundabout way of doing it. The easy answer is to simply have a number of small partitions on each disk and use multiple partitions to split thing evenly. Say you've got a 80GB drive, 2 120GB drives, and a 200GB drive. Let's say we want to make 4 RAID volumes and then group them all into a single LVM volume. Dividing by 4 tells us each partition needs 20GB on the first drive, 30GB on the second and third drive, and 50GB on the fourth drive. 10 multiplies evenly into all of those drives so we
Can't do that. The idea of a raid 5 is that one partition is allowed to fail without data loss. Soon as you put 2 partitions onto the same drive, losing that drive will make your raid fail.
If you don't care about that, easiest is to forget about the raid and go with LVM directly on the drives.
Right now there's 4 drives in that computer - 120, 160, 200 and 250GB. By slicing them up to partitions and taking one partition of each drive into a raid, I'd get the same size than doing a single raid over all drives - a maximum of 3*120=360GB. Losing over half my current diskspace for redundancy.
Not really. You'd have 360GB in the RAID array(s), but you'd still have the other 40+80+130 = 250GB available to use in other ways. If you wanted to maximize your space, but have redundancy on as much of it
If you do something like this, be *VERY* careful how you cut partitions up on disk. Essentially, you are doing RAID-0 (striping, no redundancy) of RAID-5 blocks. This means that if a single Raid-5 block goes out (not partity rebuild, but failed), so does all the rest of your data.
What I do is simple do software Raid5, and when I need to expand, bring my computer and a new disk in to work, dump my data onto a big frickin disk array we have there, reformat the Raid5 with an extra disk, restore, bring home, vio
Essentially, you are doing RAID-0 (striping, no redundancy) of RAID-5 blocks. This means that if a single Raid-5 block goes out (not partity rebuild, but failed), so does all the rest of your data.
Maybe I'm missing something, but what you're saying doesn't make any sense. How would one of the partition-based RAID-5 arrays fail? The exact same way a RAID-5 array that uses whole disks would fail: two failed hard disk drives. In either case, if you're doing RAID-5 of complete disks or multiple parallel
I'm using something like that... (Score:5, Informative)
I recently had to set up two new servers. One is for business, and one is for personal data. For both, I used RAID 5. They run NFS and Samba, with different directories shared as needed to other systems. RAID 5 is EXTREMELY simple to set up (it's a one line command, once you install mdadm, which, on Debian, installed like a dream), and I'd just suggest Googli
Re:I'm using something like that... (Score:5, Informative)
You can also add more spares and more devices with mdadm, or replace faulty devices (not hot swappable, unless you have special hardware, and I don't even know if Linux supports that).
Here's another tip: If you're using Linux software RAID, carve your drives into multiple partitions, build RAID arrays over those, then use LVM to weld them into a larger pool of storage. It may seem silly to break the drives up into paritions, just to put them back together again, but it buys you a great deal of flexibility down the road.
Suppose, for example, that you had three 500GB drives in a RAID-5 configuration, no hot spare. That gives you 1TB of usable storage. Now suppose you're just about out of space, and you want to add another drive. How do you do it? In order to construct a new, four-disk array, you have to destroy the current array. That means you need to back up your data so that you can restore it to the new array. If there were a cheap and convenient backup solution for storing nearly a terabyte, this topic wouldn't even come up.
If, instead, you had cut each 500GB drive into ten 50GB partitions, created ten RAID-5 arrays (each of three 50GB partitions) and then used LVM to place them all into a single volume group, when it comes time to upgrade, you will have another option. As long as you have free space at least equal in size to on of the individual RAID arrays, you can use 'pvmove' to instruct LVM to migrate all of the data off of one array, then take that array down, rebuild it with a fourth partition from the new disk, then add it back into the volume group. Do that for each array in turn and at the end of the process you'll have 1.5TB, and not only will all of your data be safely intact, your storage will have been fully available for reading and writing the whole time!
Note that this process isn't particularly fast. I did it when I added a fifth 200GB disk to my file server, and it took nearly a week to complete. A backup and restore would have been faster (assuing I had something to back up to!). But it only took about 30 minutes of my time to write the script that performed the process and then I just let it run, checking on it occasionally. And my kids could watch movies the whole time.
For anyone who's interested in trying it, the basic steps to reconstruct an array are as follows. This example will assume we're rebuilding /dev/md3, which is composed of /dev/hda3, /dev/hdc3 and /dev/hde3 and will be augmented with /dev/hdg3
In order to make this easy, you want to make sure that you have at least one array's worth of space not only unused, but unassigned to any logical volumes. I find it's a good idea to keep about about 1.5 times that much unallocated. Then, when I run out of room in some volume, I just add the 0.5 to the logical volume, and then set about getting more storage to add in.
Re:I'm using something like that... (Score:2)
This is a great idea.
I'm thinking about rebuilding my home server which mainly is a mythtv backend. Currently using LVM to get bigger partitions than I have drives (my tv partition is at 480GB right now) and was thinking about RAIDing them for added security, but was put off by the fact you can't easily extend a raid.
I'll follow your tip, but will probably add boot and swap partitions to every drive. Not because it's needed on every drive, just to keep the setup consistent over all drives.
Couple quest
Re:I'm using something like that... (Score:3, Informative)
I'll follow your tip, but will probably add boot and swap partitions to every drive. Not because it's needed on every drive, just to keep the setup consistent over all drives.
I did that too. I also had a couple of drives which were actually slightly bigger than 200GB, so I used the extra space for /root partitions (mirrored).
This will work with those newfangled extended partitions, right? Didn't use those since the days we dual booted DOS and OS/2.
Yep. In fact, to keep the numbering clean, it's a
Re:I'm using something like that... (Score:2)
Your ideas and suggestions are welcome.
Bear with me - I'm still recollecting parts of my just exploded brain :-)
I can see you wouldn't want to extend a degraded raid though. OTOH, if one knows that one can reconfigure it later, no trouble just replacing the disk and then rework one partition after the other, at one's leisure.
What I'm trying to work out right now is this: I read the total size of a raid is (number of drives -1)*size of smallest drive. Right now there's 4 drives in that computer - 120,
Re:I'm using something like that... (Score:3, Informative)
where Sum_{j=1}^{m} is the sum over m arrays, N_{j} is the number of partitions in the jth array and S_{j} is the size of a partition in the jth array. One thing that this equation shows us, is that each array in the LVM does not need to be the same size! The other constraint in
Re:I'm using something like that... (Score:2)
Re:I'm using something like that... (Score:2)
Thanks for the lead. I started with 40GB as 'ideal' size because it'll fit best into my disc sizes. I realized that even a 2 drive raid is ok, because it can still work - the more partitions, the better, though.
I put 3 4-drive partitions at 120GB each, 1 3-drive partition at 80, and a 2-drive partition at 40, with 50 left. Gets me 480GB usable out of 730.
I could even scale down to 20GB partitions - not to optimize usage, but to allow for smaller partitions which will help in extending/moving to another dr
Re:I'm using something like that... (Score:1)
Re:I'm using something like that... (Score:2)
Can't do that. The idea of a raid 5 is that one partition is allowed to fail without data loss. Soon as you put 2 partitions onto the same drive, losing that drive will make your raid fail.
If you don't care about that, easiest is to forget about the raid and go with LVM directly on the drives.
Re:I'm using something like that... (Score:2)
Right now there's 4 drives in that computer - 120, 160, 200 and 250GB. By slicing them up to partitions and taking one partition of each drive into a raid, I'd get the same size than doing a single raid over all drives - a maximum of 3*120=360GB. Losing over half my current diskspace for redundancy.
Not really. You'd have 360GB in the RAID array(s), but you'd still have the other 40+80+130 = 250GB available to use in other ways. If you wanted to maximize your space, but have redundancy on as much of it
Re:I'm using something like that... (Score:1)
Essentially, you are doing RAID-0 (striping, no redundancy) of RAID-5 blocks. This means that if a single Raid-5 block goes out (not partity rebuild, but failed), so does all the rest of your data.
What I do is simple do software Raid5, and when I need to expand, bring my computer and a new disk in to work, dump my data onto a big frickin disk array we have there, reformat the Raid5 with an extra disk, restore, bring home, vio
Re:I'm using something like that... (Score:2)
Essentially, you are doing RAID-0 (striping, no redundancy) of RAID-5 blocks. This means that if a single Raid-5 block goes out (not partity rebuild, but failed), so does all the rest of your data.
Maybe I'm missing something, but what you're saying doesn't make any sense. How would one of the partition-based RAID-5 arrays fail? The exact same way a RAID-5 array that uses whole disks would fail: two failed hard disk drives. In either case, if you're doing RAID-5 of complete disks or multiple parallel