The rules for model aircraft were made in the days when big, heavy, dangerous models with IC engines were the only thing you could fly. These days we have sub-20g 'toy-grade' quadcopters and very light foam RC planes powered by small electric motors.
One-size-fits-all rules don't work any more. We need different weight/risk classes. IMHO, the smallest/safest of toys should be pretty much unregulated (max 100g at 100ft altitude, maybe?). Then we've got the medium sized consumer/hobbyist drones (usually under 1kg or so) - which start to pose more of a risk, then there's the higher-end aerial photography rigs (up to 10kg?) - potentially very nasty if it crashes into somebody. And of course, beyond that, we've got much larger military drones, which are more-or-less unmanned full-sized aircraft.