To start with, only make it a little harder to maintain a driver's license, such as requiring people to take the test more often
Meanwhile, keep making the driving tests more strict. Not impossibly difficult, but maybe difficult and expensive is roughly the same range as getting your pilot's license.
I've been in favor of doing just that for a long time now, before self-driving cars were involved or even a thing.
It's ironic you mention the test shouldn't be impossible difficult, but it seems the primary argument for handing out drivers licenses like candy is that for way over half the US population a test that is possible to fail effectively is impossible (which never sounded like a valid reason against it to me, but alas)
A funny story that happened to me when I went to take my first driving test, many moons ago...
I was 17 and pretty nervous and anxious about taking the official test (as tends to be my nature), so I purposely scheduled it on a later day than suggested in order to get as early a slot in the day as possible.
My thinking was that a driving instructor that has had to put up with bad drivers and dumb kid mistakes all day is likely to be pissed off, even if only on the inside. That was an additional stress I didn't want, so hopefully if I was first in the day, the instructor wouldn't have a full days worth of frustration pent up to potentially be taken out on me.
After a 10 day delay I was able to get slot #2 that morning. Sure not as good as slot #1, but how bad could one student before me possibly be?
As myself, my mother, and the mother of the girl in slot #1 were all standing at the front window of the DMV watching her do the parallel parking cone test, just as they finished the instructor opened the passenger door and stepped out of the vehicle, the car lurches forward with a brief screech and runs over the instructors foot!
First to be said, the instructor was not seriously injured, though I can only imagine how much that would hurt.
EMTs came and examined him and took him to the hospital for further examination.
An employee there was out talking to the instructor before they took him away, which is where the report of "no serious injury" came from, as well as determining another instructor would need to be assigned for the day.
(It turns out I was in slot #1 for the new instructor anyway!)
The girl and her mom were still at the DMV after I completed my test and returned and her mom was still chatting with my mom, which was a little surprising as I assumed they remained there due to the accident, something I figured would be involving a lot of paperwork of not a police report or something like that.
Nope, turns out the girl passed the rest of her testing, and despite running over the instructors foot with her car, was waiting on (and issued) her full privileged drivers license!
If that isn't reason enough to fail someone and keep them on a learners permit, I'm honestly not sure what one could do to fail it if they wanted to.
The driving test was already far too easy even back then, and from what I hear lately the written test is now multiple choice where they get to choose which questions to skip or answer, the cone test is now spread out further than the parking lot line guides we used, and the driving test itself is limited to four right turns going around the block.
I understand how today it is practically impossible to live without being able to drive to and from work, to and from the big grocery stores that replaced the corner-mart, and all of that...
But I still wish they would take into account how difficult it is for the rest of us to live when they allow people like this to pilot a 4000lb block of metal without the slightest idea how to control the thing.