You explained that so much better than I did!

Except FreeBSD can't do Xen + ZFS. BSD is good for a lot of things and so is Linux. A good sysadmin picks the right tool for the job. I'd like to think the BSD project benefits from more people using ZFS.

Have another look at the docs, you can upgrade capacity one disk at a time but, of course, throughput suffers because you now have asymmetric storage. You can also change level to some degree by grouping sets (eg turning a mirror into a stripped mirror)

But isn't that the whole point, ZFS is designed to avoid the most common failure modes but it relies on reducing the errors in the data it is using for check summing. Thats why ECC is important, RAM errors are the 2nd most common bit flip error and if that's your comparator, it needs to be accurate.

You could still use ZVOLs. I use ext4 on top of ZFS for some virtual machines. If you make sure to use the same block size in both the performance hit is small (approx 10%) and you will have ext4 with proper check summing and snapshot capabilities under it.

Because ZFS allows you to flexibly allocate, administer, snapshot and backup data in a coherent way. You can even use other file system on top of it if you want (zvols). Hardware level checksumming on drives is not aware of your file structure and a look at SMART info on a drive will reveal quite a few Unrecovered errors (density is a bitch). So I wouldn't rely on your drive to correct it's own errors.

You're talking about different levels. I use a zpool on SSDs for my databases AND give the databases lots of RAM. The two are not mutually exclusive but if course your RAM should be utilised as close to your query as possible to increase throughput. ZFS is for integrity first, squeezing speed out of it is certainly possible but not it's primary purpose.

ZFS-fuse is not ZFSonLinux! It uses an older version and is much buggier. Also which HDD controller are you using? This really matters cos ZFS should be talking to your drives directly and bad controllers sometimes do nasty things.

Good sane description here: http://ianhowson.com/do-you-re...

Because ZFS has far better features than BtrFS http://rudd-o.com/linux-and-fr...

Actually it's pretty friendly on resources but likes lots of RAM to perform well (1Gb per Tb of storage is a good minimum). One of my servers runs on an atom processor (8x 3TB drives in equivalent to RAID 6 gets throughput of about 200MB/sec) Adding disks is also a strength. You can grow data sets quite easily but naturally performance degrades until you update the whole drive set. A lot of RAID controllers can be put in HDA mode so you may be lucky. However the Adaptec controllers go cheap 2nd hand ($100).

One should also use SSDs with capacitors if they are backing a ZIL but the only things that are absolutely essential are decent HDD controllers (I've had good results with Adaptec SAS controllers, even using a mixed SAS/SATA drive set on separate channels)

The technical descriptions I've read say that you absolutely should use ECC because ZFS will eventually hit a checksum mismatch. This could result in valid data being flagged as corrupt. ECC RAM is not much more expensive these days but you do need a mobo that supports it.

ZFS is a layer below LVM. It's best to give it direct control over your drives (no hardware RAID). The reason for this is to allow it to do data integrity checks on the actual data being written. It's similarly fast compared to hardware RAID but guarantees data integrity in a much more compete fashion. I use a striped mirrored setup which is similar to RAID 10 (over 4x 3TB drives with caches on a pair of SSDs). If you cache like this, frequent reads don't need to go to the spindles. It also had built in compression and deduplication. The best thing IMO is instant snapshots though, that's one feature I can't believe I lived without.

I've been using ZFSonLinux for a year in production. No problems at all. It's my storage back end for Xen Virtual machines. Just make sure you use ECC RAM and a decent hard disk controller. Instant snapshots and ZFS send/receive functions are awesome, have reduced my backup times by an order of magnitude. I use a Debian Wheezy/Unstable hybrid.