I think that the consensus is that a conversation with a passenger is different from that with some one on the phone - as the passenger is aware of the same environment and you both automatically adjust the tone/pace of the conversation depending on the current conditions. I.E. the passenger will generally keep quiet when you are performing a complex maneuver. That is not to say that passengers can not be a distraction, just that in general a phone conversation is a worse distraction that most passengers.