You'd think that, but I highly suspect you'd be wrong.
I fly small airplanes regularly, which means it's just me, I don't have a copilot to handle the radios. I do the most communicating with air traffic control during the most critical aspects of flight, which is takeoff and landing. While approaching the airport, I not only have to listen for radio calls for me, but also for aircraft around me to maintain situational awareness and ensure the controller hasn't just cleared someone onto the runway I'm about to land on. Often I'm not cleared to land until I'm on short final and starting the power and pitch adjustments to flare. I must acknowledge that clearance with a radio transmission. I'll often receive basic taxi instructions (asking where I'm going on the airport, giving me a ground control frequency) during rollout.
Pilots every day talk to ATC at the same time they are performing critical tasks in the airplane. "Dropping the airplane to fly the radio" is rarely cited as a contributing factor to a crash. Not to say it never is, but it is rare. Pilots receive NO training on how to split their attention between the airplane and the radio ... while we are admonished to always fly the airplane first, failure to acknowledge a landing clearance has the potential to have the FAA start enforcement action against you, so it's not optional.
I don't think the big danger in driving while holding a cell phone is because you're talking, I think it's because you've just taken a hand away from controlling the vehicle. Or you've got your neck in some weird position trying to hold it between your ear and shoulder. Sure, it takes some brain cells to carry on a conversation, and that DOES reduce safety somewhat, same as singing along with the radio or carrying on a conversation with a passenger. But I can walk and chew gum at the same time, I can land an airplane and talk to ATC at the same time, and I can drive and talk to someone at the same time, regardless if they're sitting next to me or I have a bluetooth in my ear.