You're misreading the regulations. Over a sparsely populated area there is no altitude restriction, there's a distance to "person, vessel, vehicle, or structure" requirement. You can fly as low as you want over e.g., a cornfield (say, for crop dusting) or the desert or ocean so long as you don't compromise the "emergency landing without undue hazard" rule. I'd argue that it's not reasonable to expect everyone who buys a drone to become intimately familiar with 14 CFR 91's requirements, so some general rules are warranted. For instance: what if someone wants to operate a drone within a Class E surface area (which is within several miles of many airports, and explicitly goes all the way to the ground because it's for the purposes of taking off or landing) - should they be subject to visibility and cloud clearance requirements? The reason they exist is because instrument traffic coming in to land may be coming out of the clouds and VFR traffic needs to stay clear enough to see-and-avoid them. Seems fairly reasonable, right? But a drone might be below 500 feet and still kill someone if they don't follow these rules.
But that's not really the point. Something makes an aircraft subject to these regulations, right? You can't just get a plane and say you're not an airplane and as long as you fly it below "navigable airspace" you're fine - in fact that's the exact opposite of the rules. The FAA certainly doesn't assert authority over hovercraft or other "ground effect vehicles" - it seems reasonable to assume that the FAA considers its authority over aircraft to begin when they can and/or are designed to be flown into the "navigable airspace". Many drones can fly over 500 feet just fine. I suspect the FAA would be less concerned about a drone with a GPS map magically kept up to date that prevented it from flying into any airport's approach path, and kept it below the minimum altitude for manned flight (including refusing to take off from a field or other "sparsely populated area").
I think the regulation just hasn't caught up with drones, but they're working on it. It won't be the free-for-all some people want it to be, but the FAA isn't some antagonistic supervillain of a government agency. They just want to make sure these things don't get sucked into someone's jet engine or going through a windshield, and they also don't want commercial operations without a higher standard of safety. I'm a private pilot and there's all sorts of rules about money changing hands in regards to my flight, and while it's occasionally obnoxious the intent to prevent a bunch of "Billy Bob's Charter" services from popping up everywhere. If you're in a manned aircraft and you want to be compensated for flight, you need to be a commercial pilot, which has higher standards. I don't see why the same basic principle of having higher standards shouldn't apply to commercial drone operations - and neither does the FAA, which is why they're going through the rulemaking process as we speak.