There is a difference between taking an address down and reading your mail.
Exactly. The outside of a piece of mail is considered public. There is no expectation of privacy. That's, after all, why there is an envelope in the first place! Stuff inside the envelope is considered private, and protected. The outside? No.
Personally, I've long suspected that the Post Office was doing something like this because they've displayed an ability to trace a given piece of mail when requested, despite not having purchased a tracking service, albeit with substantial latency. How can they tell that a bit of mail from my Aunt Mabel did or didn't come through my local post office in the last two weeks without making some kind of record of every piece?