I think that the consensus is that a conversation with a passenger is different from that with some one on the phone - as the passenger is aware of the same environment and you both automatically adjust the tone/pace of the conversation depending on the current conditions. I.E. the passenger will generally keep quiet when you are performing a complex maneuver. That is not to say that passengers can not be a distraction, just that in general a phone conversation is a worse distraction that most passengers.
I would agree with you about the passengers being aware and automatically adjusting the conversation if they are adults who drive as well. But try getting your kids to stop chattering at you is quite another thing.