Not only in that case.
Your ability or interest might change. Like what happened now with NSA backdoors getting more publicity, the Linux code is getting more reviewed for their backdoors, even after being in use for several years. If the source were not available, it would be a lot harder to check.
About "non-code submmitting users", I don't think they are the ones who should audit it. You can trust someone who does submit code, or you can suddenly gain interest in security and start checking it, or even pay soomeone to do it. The thing is that it's possible.
This is why it's a lot better to think in terms of Free Software and not open source. Technical advantages are hard to value if they are not used. Freedom is a lot easier to understand. You have the freedom to use the code as you like, share it and whatever. To illustrate the importance of freedom vs its exercise: lots of people don't want to, or don't have the money to travel outside their countries, but we don't like to have that possibility restricted.