The FAA agrees with you and you still retain the ability to fly the craft as a hobbyist, but you must do so below 400 ft and within line of sight (not using FPV). You must not operate the craft for money, nor as a part of a business.
The problem is that when people take their quad out to a park and fly it using FPV, they often go above 400 ft, fly over other people's property, fly over crowded areas, fly over highways, and fly in dangerous areas (near power lines, etc). This isn't a "few bad apples" either, this is the glut of FPV users.
Ars Technica got their hands on a DJI Phantom and IMMEDIATELY did the things I mentioned and wrote about it:
link
Some emboldened quad users have been more than willing to put others at risk and fly their aircraft in the landing path of airports. In 2013 an Alitalia pilot identified a quad in the flight path at JFK International Airport, coming within 200 ft of the airplane on it's descent:
link