The core issue seems to follow that same logic as the movie studios, game developers, etc. that constantly blame piracy for a drop for a low performing release or drop in sales. Ignoring the fact that their movie plot is boring or their game just sucks. The two negative issues with drones being used for a hobby that are most commonly thrown around are privacy and air safety, with a few sensationalist scenarios being thrown up to prove they are only used be evil people... Scare the public into thinking that someone will use a quadcopter to spy on you or bring down a plane.
Privacy is a right that must be protected but banning "drones" does little to protect privacy. Having real privacy laws that cover ALL filming, without permission and in a area where there is an expectation of privacy, would do much more to limit the misuse of cameras. Cameras a getting smaller and cheaper. They can be easily hidden and the are integrated into more devices. Most of the actual invasions of privacy are through low range lenses on a DSLR or a camera phone, not a drone.
Privacy is something that most people just don't have a clue about. They install apps on they phone, tablets and computers that spy on them, by sending personal information to various companies. They upload pictures to social media automatically, without considering the consequences or the privacy rights of other that might be in those pictures. What do you do when someone uploads your picture to their facebook account without your permission? No only have they not respected your right to privacy in taking the picture, they have published that picture for the world to see.
Air safety is critical and there are rules in place that require that radio controlled aircraft, including quadcopters, to not fly above a specified altitude in controlled airspace and not fly within a specified distance of a airport. Outside of controlled airspace there is generally no altitude limit. Anyone breaking these rules should face stiff penalties that include imprisonment and fines that would destroy them.
We should not forget that most quadcopters are small and light, limiting the actual risks in the event of a collision. They are no greater risk than larger birds that are often found around airports. Birds have brought down large passenger planes but a quadcopter never has. If we want to be that paranoid about air safety, kill all of those terrorist birds; leave poison seed baits and shoot down any they approach an airport. They are a greater safety risk that quadcopters. If someone wanted to crash a plane, a little imagination would show there are easier and cheaper ways of achieving their goal.
I got into radio control a couple of years ago to clean up my rusty amateur radio and electronics skills. I do have quadcopters and a couple of those have cameras. I also have a DSLR. At all times I have respected the privacy rights of others and have always operated safely which includes not flying near people, their property, roads, built up areas or anywhere near full scale aircraft. The only times you would ever see a person in a picture I have taken is with permission or in a public place where they are part of a crowd that it in the way of something I am shooting.
Most people fly safely but their are some idiots, as you find in any human activity, that need to be dealt with. The big problem now is that anyone can buy a ready to fly quadcopter, charge the batteries, and fly with no knowledge of the rules or basic safety. There are no laws that require a guide to be sold with this gear.