I trust computers WAY more than people for this task
Don't know about you but I work with computers and electronics ALL DAY LONG at my job and I will not mince words with you: YOU ARE A FOOL FOR BELIEVING THAT. Luckily for everyone people like you are not the ones who get to decide for everyone how this is going to play out, and also luckily for everyone this is not going to happen in our lifetimes if at all, so I suggest you keep your driving skills current because you're going to be driving yourself around for a good long time to come.
more infrastructure
Sure. Just like broadband Internet, right? Because everyone in Kansas has broadband now, right? Oh, wait, that's right, they don't! There are suburbs I've lived in where there was nothing but dialup..
The fact of the matter is: The entire highway infrastructure will have to be completely overhauled, at enormous expense, before 'self driving cars' could be a reality, or even have a chance at being prevalent. Even then there will always be roads and places where there is no such infrastructure, and people will have to operate the vehicle themselves to get where they want to go. Also, what do you do about situations where you're not even sure where you're going? At the very least, the puzzle of real, full-on human-level artificial intelligence has to be solved, I think, before anything like this can be practical. There are too many variables, you can't program an 'expert system' for all of them, something will come up it doesn't know how to handle.
I thought of something else that will be a problem with 'technology' like this, with regards to criminal activity. I've made the argument before that hacking a system like this will be as easy as hacking any other computer system, and criminals will use it to steal vehicles, or to carjack/kidnap people. Imagine this scenario: All you'd need are two or three 'attacking' vehicles, to surround and restrict an autonomous vehicle in such a way that is has no choice but to slow down, in order to avoid what it perceives as an imminent collision. From there it becomes a trivial matter to continue slowing down, and forcing the vehicle to pull over and come to a complete stop. From there extracting the passenger(s) becomes exceedingly trivial, and voila, you've either kidnapped the passenger, stolen the vehicle, or both. There's no way they're going to code an 'evasive maneuvers' routine into the vehicle, or even come up with a reliable way to have it detect that it's being gamed like this. Of course someone will say 'Nobody is going to do this!' or 'That'll never happen!', or 'All cars will be autonomous so it won't be possible', but that's as complete a fantasy as saying (in some people's dystopian future) 'guns are illegal so where would anyone get a gun?'. At best I think we'll have an advanced 'autopilot' system, as an option, of course, but people will still be driving their own cars, at least for the next, say, 50 years or so.
"How to make a million dollars: First, get a million dollars." -- Steve Martin