One reason is so that people who can't drive themselves can get around more autonomously than if they have to take public transit or taxis. For instance, my mom's vision is no longer what it used to be, and she had to give up driving a decade or so ago because she just couldn't see well enough, especially at night. Yes, she used to have the skills to have a driver's license, but now she's not going to be able to tell the self-driving car to do anything useful until about the time it hits something.
By contrast, I know some other people who have epilepsy, and would be perfectly capable of driving 99-99.99% of the time, but occasionally they're not, so maybe they could help the autopilot most of the time. Usually it's just brief inattention rather than more dramatic seizures, but it's still long enough not to be driving.
I also want a self-driving car so that when I go somewhere, I can then tell the car to go park itself, and call it up to have it come back from wherever it parked when I'm done. So there'd be no driver in those cases.
And then there's drunks. You really don't want their hands on the wheel. Maybe let them hit the brakes if they want to do that before the autopilot does.