Assuming you have the expertise and resources to vet every single commit made to the source tree. The fact that the code could be audited doesn't mean that it is, and therefore is no protection against a well-funded state-sponsored attack to insert subtle weaknesses in the code that could be exploited by a sophisticated methodology known only to the organization that created the patches.
This idea was recently covered by Poul-Henning Kamp (FreeBSD dev) in his essay, "More Encryption Is Not the Solution"
Sure, but at least you have that option, you can outsource the expertise if you really want to. With proprietary software, you stuck with the binary as your only option, even when you have the resource to review the code