Most of us drone users stay well away from houses.
As I said, I live in the country.
Most ATVers, snowmobilers, boaters, hunters, etc are perfectly respectable people who go out of their way not to bother anyone, and I have no issue with them.
Those other fuckers, however... I have absolutely no doubt that drone technology will become simple and ubiquitous enough that the sort of asshole who enjoys annoying people with expensive toys will inevitably discover and abuse it.