In general I dislike the idea of sending your empty car back home to pick someone else up. However, doing that just occasionally is probably more energy efficient than manufacturing more cars so another one's in the driveway and only slightly adds to congestion. I understand the concern about driverless bombs, but it isn't that difficult to accomplish now. Mules? FBI might be worried there won't be anyone to arrest, but the human mule isn't a decent catch anyhow, and they generally can't lead back to the source any better than a driverless vehicle. Organized crime doesn't have a hard time finding human mules. Driverless mules will be used, no doubt, but won't much affect either the crime rate or the conviction rate.
A huge advantage of everyone having their own auto-valet will be the reduction in nicks and scrapes from the cars parked in adjacent spaces (OK, I'm obsessed with that). Auto-valets will also be able to park much closer together, so parking lots will not eat as much space. One issue I anticipate is a long line of cars waiting to pick up their drivers which have been summoned too far ahead of time. We'll have to have some method of limiting the waiting time at the department store's front curb. If you order your car too soon, after it waits a minute for you it will circle around to the back of the line. Ha!
Another issue will be that some pedestrians already walk in front of cars, assuming they won't be hit. Knowing the driverless cars won't strike them, many more pedestrians will walk in front of them, seriously hampering traffic flow. Somebody's going to have to arrest those jaywalkers. Barney Fife, we need you back!
I also live in NC. I don't observe any random speeding up, only the random slowing and stopping. You've incorrectly assumed those people are otherwise normal just because there are so many of them. Self-driving cars will be wonderful. People will be able to do the same things they already do—eat, drink, apply eyeliner, read, text, etc.—but won't be bothered by occasional collisions.
Construction zones... instead of starting 1,000 projects and completing them in a month, NCDOT starts 100,000 and still doesn't complete them in 10 years. So only 1 of 10 zones actually have anything going on or even any workers present (not necessarily the same thing). Besides, someone has to test those construction zones at highway speeds.
I've appointed myself spokesman for all those who don't want to talk to their cars. We still don't want to.
MyDash carried on a phone, however, I can really get behind. I get to keep my interface regardless of the car, and it can look like anything I want. There could be a variety of customizable "themes" available. I like it. Makes driving my car to the store and back while I stay home easier.