I obsess, but don't spend $$$$ on equipment. As someone posted above, once you have equipment that's better than "on sale at Walmar!", most people won't hear any difference between amplifiers, or CD players, or the audio from Blu-ray players. (You can in a side-by-side A/B test, but what fraction of people do that in their homes?)
My obsession, and my money, went to making my listening room/home theater as acoustically perfect as the floorplan of my house would allow. I put up walls to prevent most standing waves, acoustically treated the walls and ceiling (all in consultation with a professional acoustician; I didn't just stick stuff up on the walls randomly), and positioning my speakers as optimally as possible. Made a WORLD of difference in the sound quality.
And, the other really great investment I made was building my own speakers. Designs exists on the Intertubes (e.g. http://www.parts-express.com/project-gallery, and the enormous fun and satisfaction of building them yourself cannot be overexpressed. Plus--with all due modesty--my speakers sound as good as speakers that cost 10-50 times what I spent (not including tools, heh). Friends that hear them are astonished that they are DIY speakers.
(N.B.: I am not taking credit for the designs; I have neither the knowledge nor the skills to design quality speakers. I simply implemented the designs and plans I found. There are some seriously smart people out on the web.)