Start here : http://www.ladyada.net/library/equipt/kits.html
Probably the lowest cost, best-value combination of tools and supplies.
Start with this book : http://www.makershed.com/ProductDetails.asp?ProductCode=9780596153748
Don't be afraid to blow stuff up. Hell, in all the best books/articles I've read about the very first thing the authors have you do is blow up an LED. Get used to it.
I have to second this. I just bought that book from the Maker Shed, and it is more comprehensible to someone with absolutely no helpful electronic background. I've been doing the experiments, and it's been great.
Check out a hunting supply catalog, the same device NOT sold as a medical item cost 90% less....
I worked as a nurse for ten years in the geri-psych field. Even my patients with insurance could not always afford the cost of their hearing aids. When the hunting version came out, we bought a couple dozen of them, as a facility, and gave them out as stop gap measures to our patients. It worked. They could hear, and communicate. It's not perfect. I'm a big supporter of it.
"If truth is beauty, how come no one has their hair done in the library?" -- Lily Tomlin