The reasoning goes back to old law which is based on the idea that when you mail a letter, you have no expectation of privacy regarding with respect to the outside of the envelope
Which, by itself, is already a crazy idea in a supposedly "civilized" world.
When I mail a letter, I expect only the staff of the post office can read the outside of the envelope for the purpose of delivering the letter, until it reach the mailbox of the recipient. After which only authorized persons (with the mailbox key) will be able to retrieve.
There are LOTS of privacy expectation on the outside of the envelope already, namely:
1. I expect no one but post office staff can read it, and only for the purpose of delivering the mail (i.e. no data mining, no retention of this data, no sending it to FBI/NSA)
2. To spell it out, I don't expect the general public, non-delivery post office staff, the post office manager, nor people touring the post office, etc, to be able to read the outside of the envelope during transit
3. I don't expect people to stand behind the postman while he puts the mail into the target mailbox. Nor do I expect the postman to read the "from" address intentionally unless the "to" address is wrong
4. I don't expect anyone can open and go through the mailbox, even without opening any envelopes.