Speed limits (and some other types of rules) are to enforce safety on what some of the outliers do.
"Regular people" who are just trying to get somewhere do generally act safely, and for the most part adjust. (Though there are exceptions.)
The problem is the rules are for the people that don't adjust, they don't think anything will happen, or they aren't wise enough to realize what the consequences are.
Removing the rules, subjects the "ordinary people" to the mistakes of the outliers to a much greater degree.
If you haven't ever done this, go over to Youtube and search for "car crash" videos and enjoy a couple hours of soviets and other former eastern bloc countries slaughtering each other on the roads. Then about half way through, notice the body parts ones, kid gets dragged ones, etc. aren't posted because that sort of stuff is illegal there (the videos of it) and against Youtube TOS. The really bad shit, they won't show you. In almost all the videos, you'll see normal people and then some dumbass outlier who thinks there are no rules causing problems.
You are signing us all up for that bullshit with your hairbrained plan.
Rules are not for rules followers, rules are for people who will try to break the rules. Misplaced faith in those people to "be better" punishes us all.