You could potentially walk, bike, take public transport or a cab to get to your clients.
No, I really can't - mostly I'm driving about 30 minutes at 50-60MPH average to reach them. Considering the fact that as a consultant I get paid by the hour it would cost me vast sums of money to bike to them, and probably an hour longer each way taking any public transport (I've looked into that). A cab is not a bad idea if you live in a city but I'm working between multiple areas and also take very long road trips all the time (partly for business) so it would be stupid to also spend money on a cab when the marginal extra cost of using my car is vastly less.
it's not an absolute requirement for your business
My clients disagree which is why I drive to them. If I don't have a job because I do not drive, it's a requirement.
Your argument is way, way weak. There is no "key difference". The fact is that driving for Uber and driving friends around has zero actual difference in terms of external risk or ability. That's the core argument where you simply cannot distinguish, thus either everyone needs a commercial license or no-one does.