I know a lot of car owners that get irritated if a pebble from the road nicks their paint job.
If you owned a plane, why would you tolerate people crashing stuff into it while you fly it around? That shit's expensive.
A friend of mine did hit a bird with the driver-side rear view of his car at 45 mph. Glass and bird blood (actually, some of it was his) everywhere - window was down...
I'm pretty sure drones large enough to fly a few thousand feet up will damage a plane if they collide - maybe it won't bring down the plane, but it will push maintenance costs up. It's just something we don't need regardless of the perceived safety. You wouldn't (I hope) drive an RC car around on a busy street. Sure, it wouldn't total a car, but you'll probably get your ass kicked.