Pretty much comes down to the privacy vs. security issue in the back of everyone's mind.
In this case, it'll probably come down to 1) where exactly was the quadcopter, and 2) does somebody have the right to destroy somebody else's property if it ends up on their property.
And note that the answer to #2 is already well known ... the answer is NO. They can remove the item, and they can even bill the person for the trouble of moving it, but they cannot destroy or take it for themselves.
drones themselves are likely a peeping tom's wet dream
Maybe in dailymail fantasy land, where every quadcopter is flown by a peeping tom looking to hover outside a girl's window and watch her undress.
In the real world, their cameras usually have have wide angle lenses and a person would be to small to be recognizable if the quadcopter was more than 75 feet away from them. The vast majority of the operators are simply taking pictures of houses, landscapes, the sky, etc. If there are people in the picture, they're incidental or just "a crowd" unless he brings the quadcopter within 25 feet of them -- and there's nothing stealthy about a quadcopter at 25 feet.