Mostly because we have enough arbitrary laws, and we should be working to get rid of them instead of adding more.
So we ban cellular phones in the car. What do you do if people then start using ham radio or CB? Do you ban that, too? What if people then start using data channel voice chat? If you ban things like that, then you also risk banning voice activated functions and systems like OnStar. Then maybe we ban communicating with someone outside the vehicle. Well, what if your passenger is on the phone and puts it on speaker? Does that count? Or do we ban having a phone in the car?
We already have laws about "endangering those around". We don't need "endangering those around [when using a mobile telephone]" any more than "beating someone [who happens to be gay]". We shouldn't even have "endangering those around [while driving a car]". Laws written like these are not just unnecessary, but they are almost guaranteed to be abused, and are simply wrong. You make the behavior have legal repercussions and apply it uniformly and fairly.
FWIW, I also am very strongly opposed to sentencing restrictions (mandatory minimums), three-strikes laws, and "zero tolerance". We have judges for a reason, and they should be able to do their jobs. Instead if these stupid minimums aren't warranted by what the crime really is, then any sane person has to acquit rather than convict, even when guilt is assured.
The same is true with doing this with cell phones. If your only "crime" was being unfortunate enough to be noticed talking on the phone by some guy in a police car, how does that warrant taking away your license? The punishment does not fit the crime. Driving already has enough lunacy in its punishments and laws, I certainly don't want even more.