Better ban talking to people in the car with you as well.
As it turns out, talking over a phone is more distracting than talking with someone sitting in the car.
There are multiple reasons:
1. Someone in the car with you can and will respond to the dynamically-changing environment as you do. If something unexpected happens, they will usually stop talking.
2. In fact, someone in the car may notice something important, and notify the driver (either by shutting up or pointing it out), thereby partially mitigating the distraction they cause by talking.
Why would someone in the car with you, engrossed in conversation with you and without the responsibility of driving, react to the situation faster than you? This doesn't sound at all realistic.
3. A phone conversation requires more of your attention because you have to make up for the deficiencies of the data channel (phones have lower audio quality than real life, you can't read their body language (even out of the corner of your eye, you can get a feel for a person's mood), etc.).
You can usually get a feel for a person's mood by the tone of their voice as well - I'd say a much better feel than you can get by whatever information leaks in through your peripheral vision.
4. Shared context makes communication more efficient, thus requiring less mental effort (this is why, even in this day and age, people generally want to meet face-to-face).
While this is undoubtedly true, you still haven't made any case that people can't share a context equally as well over the phone as they can with some person sitting next to them while they drive.
5. Studies have shown that it takes humans more mental effort to think/interact with people/data they believe is remote as compared to people/things they think are local. In one study, they measured reaction times and errors in a driving simulator when people were either using an "in-car GPS" giving them instructions or a "satellite data-feed" giving them instructions. Even though both sets of instructions were identical (including latency, etc.), the mere perception that the "satellite data-feed" was non-local caused people to devote more mental effort to it, which increased driving accidents. A non-intuitive result, perhaps, but human mental machinery is finely tuned not for the tasks we currently expect it to perform.
I can't argue with this, since I have no study to prove otherwise. But I know that I've been involved in conversations while driving that consumed so much of my attention that afterward I couldn't recall the process of driving home. The same has occurred when I've found something interesting on the radio. I'd like to see some studies comparing how all of these activities, which aren't outlawed, impact driving and how they compare to the impact of phone usage while driving.
6. Initiating and finishing a phonecall requires much more attention than stopping/starting a conversation with someone sitting beside you. (Unlike fidgeting with a radio, answering a phonecall requires immediate action not at a moment of the driver's choosing.)
Voice commands. "Dial wife". How hard is that? And answering a phonecall doesn't require immediate action - if I'm shifting from first to second as I accelerate from a stop, I'm certainly not going to interrupt that to answer a call. Most people who use phones while driving do so in precisely the same situations in which they fidget with a radio - cruising along, with no anticipated interruptions.