I always find it amazing that electronic gadgets are a "distraction" yet non-electronic ones are not. You don't see legislation to outlaw paper maps, coffee cups, makeup, food, etc from vehicles.
It's a slow process, because the longer something's been legal, the harder it is to get a law passed to make it illegal, irrespective of the actual damage.
They managed to make it illegal to have open bottles of booze in a car, tied in nicely with the invention of random breath testing. Smoking too, in some places so far. Drivers eating and drinking can't be _too_ far off.
If I had to guess, I'd say it _is_ illegal to look at and/or manipulate a paper map while driving. Something about blocking the view of the road, most likely.
The highest hurdle will be making it illegal to talk to the driver while they're driving, since that was (as I recall) demonstrated in the UK as being just as dangerous as the driver using a hands-free kit, in so far as distraction was a major problem, and operating a hands-free kit was relatively minor problem.
I expect we'll have self-driving cars before that, and this ceases to be a problem... Once we have self-driving cars, just make getting a car license really really really hard, and expire them (completely) every five years.
Once a generation of teenagers ages to adulthood without the car and driver's license being a major part of their coming-of-age, it won't be a problem any more.