While everyone likes to cite the C compiler that injects a backdoor into the executable whenever it detects that it's compiling a C compiler, it's far easier just to subvert the process by releasing an executable with a back door or exploitable code in it. The former depends on you releasing the binaries, which is pretty easy if you maintain a distribution. But why even bother with that when so many people are already releasing exploitable code for you? It seems like not a day goes by where we don't see a headline here about an exploit in some popular software package. Even commercial providers like Apple can't keep ahead of all the possible exploits in the software they release -- otherwise no one would ever be able to root an iPhone.
It doesn't even have to be a specific executable we're talking about. All you really need is a library everything depends on where some guy did a unbounded copy without checking parameters. There have been several of those over the years -- compression and image libraries where some guy did an unbounded copy without checking parameters.
Of course, if someone's really interested in YOU (versus just trawling around for generic information) they could always just break into your house and plant bugs. If you browse the internet at all, it's ridiculously easy to get information on what you're up to. Sure you could use https everywhere and erase cookies, but I'm not sure how much I'd trust https. Keep in mind that a LOT of those certificates are issued by a central authority, and central authorities are easy to subvert.
With all that being said, if we were really that concerned about it we'd be making it MUCH easier to use pgp and personal private encryption for everything. We'd be making it much easier to use opportunistic encryption with self-generated keys for point-to-point communications. We'd be making it much easier to encrypt voice and video communications. Everyone would be using tor to access the internet. And we're not really doing any of those things. Hell, we volunteer so much information about our daily lives through social networking that there really isn't any need to listen in on most people anyway. I'd guess someone completely avoiding social networking sites would raise a red flag that would warrant more scrutiny.