I think that the consensus is that a conversation with a passenger is different from that with some one on the phone - as the passenger is aware of the same environment and you both automatically adjust the tone/pace of the conversation depending on the current conditions. I.E. the passenger will generally keep quiet when you are performing a complex maneuver. That is not to say that passengers can not be a distraction, just that in general a phone conversation is a worse distraction that most passengers.
I would agree with you about the passengers being aware and automatically adjusting the conversation if they are adults who drive as well. But try getting your kids to stop chattering at you is quite another thing.
No hardware designer should be allowed to produce any piece of hardware until three software guys have signed off for it. -- Andy Tanenbaum