Nope, you're absolutely right. Well, mostly right.
Over the last 10 years or so, I've had things on my phones that track me. Most of them also tracked what towers I connected to. I left the phones turned on accidentally a few times. Generally, in the air there weren't enough towers to attempt a conversation from. If it even connected to a tower, it would disconnect in less than a minute.
Here's a composite map of several trips in 2010. There were stops at Boston, New York, DC, Atlanta, and Tampa. I think there may have been another flight change in Charlotte.
Cell towers are tilted down slightly because that's where the customers are. That also works against making calls from aircraft. Sometimes that's built into the antenna, so you won't see it from the ground.
I was on one flight where the captain got on the intercom and asked everyone to double-check their phones, because he was getting noise on the radio. Mine was already off on that flight, so it wasn't me. :)