The latest alpha release of the Tor Browser uses a deterministic build process for exactly that reason: users of open source software (or the small minority of users with the necessary technical skills) should be able to check that the published binaries match the published source exactly - no malware, no easter eggs, no backdoors. If someone detects a mismatch, they can alert the rest of the community.
Mike Perry, who spent six weeks getting deterministic builds working for Tor, has some interesting thoughts on why this is an important issue for security tools, even if the users completely trust the developers.
I'd like to see more open source projects following Tor's lead. Gitian is a deterministic build tool that might help - it enables multiple people to build a binary from the same source and check that they get identical results.