Part of it is keeping his work closed source, which is extra scary when talking about cryptography. Being asked to trust a security solution that you can't examine is insane.
Unless you're a crytpographer and a programmer... examining the source is pretty much pointless. It may give you a warm happy fuzzy to be able to do so, but you lack the qualifications to actually evaluate it.
The point, surely, is not that I am necessarily a cryptographer, but that the source is available to those who are. It's not necessary for every user to independently audit the code, because the skilled individuals who do audit the code can then communicate their findings.
"But why trust the skilled individuals?", you may ask. Answer: because I find it unlikely that all the world's cryptographers are conspiring to keep quiet about any vulnerabilities they find the code. At any rate it's a more sensible strategy than "assume that Zimmerman is both infallible and incorruptible".