Well, the law disagrees with you. It doesn't, however, work like people here think it does. There isn't a line in the sky saying "this far, no farther". It depends on the nature and intent of the intrusion.
For example I've flown in a helicopter belonging to the Florida Keys Mosquito Control district. Those spray jockeys' job is to lay down pesticide on hard to reach places, particularly the first place a mosquito might light after crossing between islands which is likely to be a line of mangroves or bushes. They're accustomed to flying *low*. En route between Stock Island and Marathon Key we flew so low over peoples' houses I could certainly have told what magazines they left out by the pool -- if we hadn't been going over 100 mph. It's just normal business for those guys, and they're not targeting those homeowners in any way. But if we'd hovered over his house to ogle his teenage daughter, that would be an intrusion, apart from the epic noise.
This isn't really different from privacy law in general: context and intent matter. If someone is standing behind you at the ATM, that's not necessarily breach of privacy; but if they are doing it to look over your shoulder that's different. If your neighbor looks at the back of your house, it's normal. If he sits in his tree trying to peer through your back windows, it's not.
One of the landmark cases in privacy was Nader v.General Motors Corp. where GM retaliated against Nader for writing unkind things about its cars by hiring private investigators to dig up dirt and intimidate Nader. One of the things they did to intimidate him was to follow him around all day, often openly following him a few feet behind as he went about his business so he'd know he was being constantly watched. The court ruled this was an invasion of privacy. Sure the PIs had a right to be in the places they went, but they didn't have a right to be there doing what they were doing.