This is really is one of those situations that if you aren't doing anything illegal don't worry about it and if you do worry about it find another tool.
You are arguing a false dichotomy and the third axiom is the expectation of privacy from government intrusion.
Consider this scenario: Your neighbor dies a horrible death at the hands of the most gruesome killer. The police are pressured by the community to bring his killer to justice. In their dragnet, they listen in on your phone call to your mother in which you state to her that:
"My neighbor is dead, died a gruesome death and the police were all over the place.... I never really liked the guy, but it's sad to see him go that way"
They haul you in for questioning and charge you with his murder. What do you think the testimony of the officers will be in court?
Prosecutor: "Officer Jones, was there anything funny about the conversation you heard between the defendant and his mother?"
Officer Jones: "Yes there was, He stated his neighbor died a gruesome death, but the newspaper had not reported that yet"
Prosecutor: "Was there anything else peculiar about the conversation?"
Officer Jones: "Ohh yea, he said he never liked the guy."
Open and shut, do not pass go, do not collect $200. Point being, even the most innocuous of conversations can be taken out of context and used against you and it doesn't even have to be due to malice on the part of the recollecting party.