JWSmythe's Journal: 6.4Tb filesystem array, we have a winner

A slight update to my previous messages. I was making some mistakes trying to relate 1k blocks to bytes (silly me). I'm posting here, so hopefully people here can reference it if they need help, plus Google may spider it, and help out complete strangers.

This is our machine:
root@backup (/etc) cat /etc/slackware-version
Slackware 10.0.0

root@backup (/etc) uname -a
Linux backup 2.6.9 #1 SMP Fri Nov 12 17:11:49 EST 2004 i686 unknown unknown GNU/Linux

The machine is a dual 2.8Ghz SuperMicro 1u with an Adaptec PCI-X ASC-29320LP U320 SCSI controller. There are two IDE drives mirrored inside the 1u machine to hold the OS, os I can fiddle with the arrays all I want.

Storage is the following:

Promise VTrak 15100
15 250Gb SATA drives
with all 15 drives being a single RAID5

Configured as: /dev/sda1 1 105954 1702150947 fd Linux raid autodetect /dev/sda2 105955 211907 1702134945 fd Linux raid autodetect

(obviously the second is sdb[1-2])

I'm still doing md devices. my /etc/raidtab reads as:

raiddev /dev/md1
    raid-level 0
    nr-raid-disks 4
    persistent-superblock 1
    chunk-size 128
    device /dev/sda1
    raid-disk 0
    device /dev/sdb1
    raid-disk 1
    device /dev/sda2
    raid-disk 2
    device /dev/sdb2
    raid-disk 3

Which now gives me exactly the number of blocks I expected (6,808,???,??? 1k blocks)

root@backup (/etc) cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid5] [multipath] [raid6] [raid10]
md1 : active raid0 sdb2[3] sda2[2] sdb1[1] sda1[0]
            6808571136 blocks 128k chunks

md0 : active raid1 hdc1[1] hda1[0]
            19423040 blocks [2/2] [UU]

I wanted to use xfs, which took some tinkering with. tail -f /var/log/messages was very useful, since it was telling me exactly what the problem was at mount time mount just returned this, which was less than useful.

root@backup (/etc) mount /dev/md1 /array
mount: Function not implemented

The correct mkfs.xfs line is:

mkfs.xfs /dev/md1 -b size=4096 -s size=1024 -f

And now mount /dev/md1 /array works exactly as expected. With the array mounted up, I show 6.8 billion 1k blocks, or 6.4Tb

root@backup (/etc) df
Filesystem 1K-blocks Used Available Use% Mounted on /dev/md0 19117756 5545604 12601000 31% / /dev/md1 6808440064 528 6808439536 1% /array

root@backup (/etc) df -h
Filesystem Size Used Avail Use% Mounted on /dev/md0 19G 5.3G 13G 31% / /dev/md1 6.4T 528K 6.4T 1% /array

Now that I've done it, it seems rather painless. :) The harder part may be populating 6 TB of storage. Backing up 100+ servers may satisfy that part rather quickly. I look forward to needing to add MORE storage. :) Actually if we do it, we'd probably get another machine, since the power requirements of this and the other two machines in this rack are rather high (about 30A total), and the weight is bordering on max for a single rack. No kidding, there's about 1000 pounds in this rack. The three machines made a Chevy Tahoe squat down a few inches with all the equipment loaded in. I had to borrow that truck, there's no way my Firebird was taking 1/2 ton of cargo. :)

So you're happy with the Promise arrays?

[Saint Aardvark]

Probably a redundant question, given the last JE, but I'm in the market for a drive array and I'd like to know how the VTrak 15-bay is working out for you.

So far I'm looking at the Fastora DAS-315 [fastora.com], and what I'm pretty sure is a Premium 8600 [raidweb.com] from RaidWeb. (I'm dealing with a local reseller on that last one; used to use their arrays at my old job, and we were quite happy with them.) I'd seen the Promise arrays, but for some reason was kinda shying away from them...not sure why. But if they work for you

Re:So you're happy with the Promise arrays?

[JWSmythe]

You can't do a 15 drive array with 400Gb drives. That'd give you more space than I have on my arrays, and I just can't handle that. :)

I think I read that Seagate has released their 450Gb SATA drive. I'm pretty sure on the size, but may be wrong on the vendor. It wasn't Hitachi though.

I've been perfectly satisfied with the Promise **EXTERNAL** arrays. I have choice words for their internal cards, which are less than flattering on almost all accounts. But their external stuff is very nice.

Re:So you're happy with the Promise arrays?

[Saint Aardvark]

Yeah, I think it was the reputation of the Promise internal cards that gave me pause. We've got a couple here -- just for extra IDE channels -- and no problems so far, but I've heard stories about their RAID controllers (and have had problems myself).

I like the idea of embedded Linux. Do you ever SSH in to configure it or anything? Or maybe just hide w5r3z...:-)

Thanks for the tip about drives starting to fail. My plan at this point is two smaller arrays (RAID5 for home directories, and RAID5 or JBOD

Re:So you're happy with the Promise arrays?

[JWSmythe]

It doesn't have SSH running, and I haven't played with it enough to think about hacking it to give more functionality. I'm pretty happy with there little interface.

It does have a network connection in the back, which you're suppose to be able to maintain it with, but it makes me nervous to put something like that on the network, where someone may try something bad.

I don't have to hide anything. If I want to put someone on a drive, I just put it there. :) I'm the Sr. Admin here, so no one can s