True, you didn't built everything from source, but you were happy enough that everything traced back to "the" sources to make you feel secure. That's a lot more protection than anything from a commercial vendor, who probably just sold you formulaic encryption without any extra work to make you feel secure. Your data would have been more secure, if not actually secure, but you'd have felt it less, because really you have no way of knowing. So without somebody taking the extra time to make you feel secure, you naturally wouldn't feel it very much, if at all.
The problem is that there is no conceivable way to do what you are saying. It involves compromising or proxying disparate traffic, expertly.
And then, after all that, it would involve rooting an otherwise secure installation that is barely network connected, and using that to inject what, defects into the right sources so that the resulting binaries are weak or exploitable?
I agree that the NSA, CIA, and FBI have extraordinary capabilities, but the attack vectors that have thus far been revealed are the same attack vectors that security researchers have known and published for a long time - firmware, obscure libraries that are often used but seldom examined, zero-day exploits of popular software, mathematical flaws in encryption implementations, and physical security and chain of custody.
All of which is to say, the basic landscape of the threat has not changed much in 20 years. It is sophisticated, but as always, a strong layered defense and strong procedures and policies will minimize the possible impacts, exploits, and severity of breaches (if they occur in the first place). There are few things more secure than a well maintained GNU/Linux or OpenBSD box running in the wild.