I disagree. I'm pretty confident that canonical builds the binaries in a safe way. Even though there is a thirty year old "proof of principle" article on how to hide a backdoor somewhere almost impossible to detect.
The problem is that an average Unix system has many bugs at any one time. Debian and Canonical have given up on updating the kernel on every privilege escalation bug. You might forget to run apt-get upgrade for a few days.
When the NSA needs your data, they will for sure have a big list of things-to-try to get into your system. They will most likely succeed. It's the same with every other directed adversary. He'll be able to get in.
Note that I'm toning down what I read in the media a bit. When the facts state: "NSA was able to hack into some phones", the reports will change from "NSA hacked into phones" into "NSA can hack all phones" and "NSA has access to all data on your phone". They don't. Once they are interested in you, they will try to hack your phone and get what they need.
It's different for internet monitoring. Traffic analysis and automated wiretapping is something they are apparently into. So the big internet providers have automated that and they can't do much to verify the requests they get. So when Snowden says he could tap anyone anywhere, he means he could issue a "wiretap warrent" from his desk, which would reach, say yahoo as: "NSA wants access to the traffic from IP x.y.z.w, you're not allowed to ask why." And then Yahoo might try to fight that a few times, but they have already lost. So nowadays that is processed immediately, and/or automatically.