I live in an apartment block. Facing me is another apartment block and between the two buildings there's a busy footpath, then a busy road, and another busy footpath. Nothing else. If someone loses control of an RC, there's an unacceptable chance of injury and could include causing a car crash.
If someone's flying there, they deserve to have their equipment confiscated by the police and to be charged with a crime with "endangerment" or "negligence" in its name. I agree that it makes sense to restrict allowed flight areas to places that don't have heavy pedestrian or vehicular traffic, the same way that I think it makes sense to have posted "no parking" signs. "No hobby aircraft here" makes more sense to me than "No hobby aircraft, period".
Drone[s] are used by only a small group, so we're talking about restricting a small group to safeguard the safety and privacy of the many.
We're talking about restricting a small group (RC aircraft pilots) for the bad behavior of a much, much smaller group (negligent/irresponsible/criminal RC aircraft pilots). The behavior is what I'm opposed to, not the technology...so why would I want a blanket ban on the technology, rather than the behavior?
In terms of convicting someone of illegal drone use, you're right that anti-harassment could be tried. Problem is that they might or mightn't work, you might have a hard time proving it, and the case could take years. If you want to prevent the incident, it's better to have a clear law "No drones here".
In a perfect world, all crimes would be both simple and easy to prove, and criminals would get what they deserve. I don't believe in outright banning something because a small fraction of its users abuse it. I feel that on the balance, the loss due to the reduced freedom of action of the responsible users is greater than the gain of preventing an already rare occurrence.