What is it I can do with ZFS in Linux that is so important?
What is it I can't do without ZFS?
It does a lot, but the features I'm interested in are the protection against bit-rot. Specifically, if you set up a mirrored pair of disks in a ZFS pool, it will checksum everything on both sides of the mirror.
When the array is checked (scrubbed), it verifies the checksums. If there is a mismatch because the data has glitched on the media, the checksum won't be valid on one disk, but it will be valid on the other disk so it can repair it. If there's a mismatch in a more conventional mirrored pair, the controller wouldn't really have a way to know which one is correct.
This capability is also in BTRFS, but much has been written about how BTRFS is still experimental. Also, last time I looked, BTRFS was only available for Linux - with ZFS it would be possible to migrate to FreeBSD if Linux does jump the shark.
The other thing is that the scrubbing process is done in the background. My main data pool is a pair of 4TB disks, which was EXT4 to begin with, then BTRFS and now ZFS. The system is a desktop which is powered down at night. Every 180 boots it would run FSCK, which took something like 2 hours to run on EXT4, during which the system was unusable. With BTRFS and ZFS, the scrubbing takes place while the pool is mounted. So yes, you can do this with BTRFS as well, but ZFS is the more proven option of the two.