We run a lot of ZFS on OpenIndiana/Nexenta, but also have some ZoL.
My favorite things about ZFS:
- Simpler volume management -- there's no more LVM layer! A little weird at first, but it really grows on you. Just zpool create, zfs create and you're off and running.
- Huge volumes -- we have a couple in production near 800TB
- Writable snapshots (think FlexClone on NetApp) -- no performance penalty. We have systems with hundreds of snaps and clones.
- Really stable (in our experience, ZFS on *Solaris has been rock solid -- the management pieces on top is where we occasionally run into issues). ZoL has already been quite stable.
- ZIL/L2ARC -- Use SSD's to accelerate reads/writes.
- Performs great with minimal tuning, but there are plenty of hidden knobs if you need them.
- Triple parity RAID options. Essential for larger drives.
Cons and Caveats:
- Memory hungry. Really memory hungry. Fortunately, RAM is cheap these days.
- Does require CPU as it wants to do all the "RAID" itself. Processors are so fast that this has never been an issue for us. Also you probably want to use disks that speak real SAS, not SATA to ensure graceful failure.
- For the *Solaris versions, picking the right hardware tends to be important. ZoL opens a lot of doors here.
- Deduplication sucks (or sucked last time we tried it). Required a ton of memory, especially if you want to use smaller block sizes to get better space savings. Very challenging to move away from deduplication once you turn it on.