I don't want to go down the rabbit hole, but without personally inspecting the source code of everything you run, you can't make any claims on the Open Source soap box. Lest we learn anything from OpenBSD's latest debacle?
Loadable libraries are available for every OS. I've been involved in writing a hooks based fault injector even for VMS. It can be used for nefarious things. Everything can be similary subverted. There are very few Neos and Trinities out there- I knew one, and he is doing amazing things for the US Govt.
You're right- of course turn off Javascript. I run NoScript and I've still let things slip- not malware mind you, just JavaScript. I've seen what Jeremiah Grossman, et. al. can do. It only takes once.
You can either use your device, or be perfectly safe. Obviously there is a happy medium, but that's different for everyone. If mutt, news and lynx work for you- more power to you.
Don't get me started on hardware. The AC below said it best.