Most of the comments here seem to be more about the Linux distro than Spiffyman's domain-specific requirements. Speaking from that perspective, I'd check out the Quantian Scientific Computing Environment (http://dirk.eddelbuettel.com/quantian.html). It's a complete Knoppix LiveCD distro, pre-loaded with every FOSS math and science app around, including a fairly comprehensive TeX/LaTeX set of tools, IDE's, etc.
The best part about Quantian is that you can run it as a LiveCD, boot it on an existing Windows system while you get up to speed and learn the toolset, and you don't have to immediately reformat/repartition/dual-boot your existing box.
Personally I'm a big fan of Ubuntu, and especially Edgy Eft, but I still boot up a Quantian VM anytime I'm sitting on Windows or Mac and need to use R for stats, or use some math libraries.
Hope this helps. Good luck!