In places where it'd make the most sense to have it drop you off at work (big cities, where parking is both limited and expensive), it won't work for 99% of the local population anyway.... where does the car park when it gets home?
99% of the population doesn't live in Manhattan. LA, Chicago, Atlanta, and many others have people drive in from Long Island (or Jersey), where people do have houses on land with driveways and (often) garages.
I didn't say 99% did live in big cities. I said that, of those in big cities, the majority there could not make use of this.
FWIW, more than 1% of the population of USA lives in LA. Nearly 3% live in NYC.
Using my car like a Taxi would be great.
For who? Do you want random people getting in your car without any supervision and doing who knows what in there? On the flip side, who wants to get in a driverless car and allow the owner to control what happens to them remotely (I'm assuming you would not allow them to choose its course, else they could easily strand it anywhere they like... talk about inconvenience). Besides, real taxi drivers tend to handle queries like, "hey, I need to get to that theater down by atlantic... you know the one?" much better than I'd expect your personal driverless car to do.
And the fuel to get it home is less than the cost of parking.
That's extremely subjective. I'm quite certain that parking is free for employees at the vast majority of employment sites. The times when it costs money, you're often in a large city (as you noted, most people don't live in NYC). Within large cities, if you have a garage, you're likely to be living outside the city. If you're making that commute, then:
a) there's a higher likelyhood of tolls exceeding the cost of gas
b) gas cost will be higher if you're that far away
c) WE'D BE DOUBLING THE NUMBER OF CARS ON THE ROAD!
How is that not a down side? (you stated there were none)
What happens when the car breaks down with no one in it?
Depends on the "break down". Most people would take that to mean the engine lost power. The car would coast onto the shoulder and wait for assistance.
Getting in the way of everyone else, and likely during rush hour. If it had a driver, he would get out, possibly ask for help, and push the car to a safe location. Without a driver, everyone else is up shit creek until the tow truck comes.
What happens when it's in an accident? (where's the owners? who is responsible? how to get it off the road? etc)
The answer to all of those is the same as any other accident where the driver is incapacitated. We managed to solve it for an unconscious driver. But you are too stupid to solve it for a missing (benevolant) driver? We've even solved it for driverless cars now (hit and runs with the car left behind).
Most accidents do not result in an unconscious driver, and hit and runs are a rare percentage of accidents. If a driver is there, even if unconscious, you have immediate access to their ID and, likely, their contacts (via their phone). Driverless car - it's just going to sit there and clog up the road way until the tow truck comes.
what's the point of having it?
To get places faster, safer, and more efficiently, ...
No need for driverless for any of these.
...while freeing up time for other activities.
Like what? Picking your kids up from school? Busses solved that ages ago. We don't need every parent sending their driverless cars around to pick up their kids individually, wasting loads of resources, clogging the roads, making it far easier for kids to be irresponsible (let along easier for the kid to take the wrong car and end up who knows where).
Parking for you? First, are you really that freaking lazy!?! Second, an automated parking attended would be much more efficient, and would work with all cars on the road today (pull into a drive through carwash type of thing). Lastly, no one wants to wait for the giant line of driverless cars that had been asked to come pick up their lusers way too early.
I predict that, should driverless cars happen, they'll become one of the most annoying things we have to deal with day to day.
Please note, none of my comments are against greatly enhanced cruise control - even fully automated. I'm only saying that an empty car should not, IMO, be driving around.