FAT32 has a 4 GB max file size.
This can be somewhat inconveniant if you have say a linux dvd ISO thats larger than that, or if you record HD video, etc.
I would suggest going with NTFS, there are ways now to use the actual microsoft NTFS driver in a wrapper with linux, then you get full native compatability if thats what you are thinking (it used to be that NTFS compatability was an issue with linux).
The cheapest solution for backup would probably be an external USB drive. And you may also want to have 2 of them that you swap out for off site storage, or something like that, depending on how important your data is to you.