If you are building a server, then file system absolutely matters.
Ext4 is fine but limited and not nearly as mature or stable as XFS. When building a server that *matters*.
ZFS blows the doors off of anything the Open Source community has *ever* built (in terms of file systems). It has features that Linux users have been desperate for. For example, file system snapshots and rollbacks. Yes, you can do it with LVM2. Guess what? LVM2 sucks at it (in comparison to ZFS).