May I suggest setting up a NAS? Wherever you like (garage, or wherever). You can set it up with RAID, and use it as a backup server too if you would like. This way, you don't need to get rid of your current system but just add up. There are a few of them now with plenty of HDD space and flexible RAID configuration, so that you don't have to get all HDDs with same capacity, specifications, etc. Therefore, you can even upgrade HDDs as you go. I personally have
Synology DS1511 in my wishlist, although it is not a cheap toy.
I don't feel I know enough to recommend any other file system. ext3,ext4, NTFS are what I use currently and I am naive if I am missing anything. Similarly, I don't know much about the differences between various linux flavors and BSD, therefore I think whichever you are most comfortable with should serve you well.
x264/mkv is the format I use too, even for my blueray backups. It serves me well, and I personally don't plan to change it any time soon.