All the crypto software I've looked into depends on big internal arrays of special numbers to do its work. If those numbers are compromised (which is what NSA contracted RSA to do, basically), then the whole end-to-end crypto channel is compromised.
And that's the problem. You can build an open-source hardware router with open-source software, to keep the possibility of hardware backdoors to a minimum, but if the basic crypto algorithm you use has been compromised from the get-go, none of it matters. I think that's going to be the next really difficult intellectual load to lift: vetting ALL of the current crypto algorithms in use today to make sure the algorithms don't have built-in compromises. Since that vetting has to be done by crypto experts, not just software engineers, that pushes the trust back up one step: which crypto experts do you trust?