The trick to flying easily is a well trimmed airplane and lack of course upset from wind, thermals, changes in direction needed. Without adjustments to the flight path, a well trimmed plane will fly itself hands off. It certainly helps that except for landing, a wandering flight path poses no risk to plane, pilot or others.
A car is not such a beast. The environment that a car lives in is full of necessary adjustments. Think of road crowns, potholes, traffic lights, turning vehicles etc. A wandering driving path provides a very real risk of collision with fixed and mobile objects. Therefore the need to make constant adjustments is very real also, and it is this that is very tiring. The trick will be to take the load off the driver and make automatic adjustments for these things that upset the direction of the car.
All that said, I remember SAAB showing off their joystick controlled car in an episode of Beyond 2000, whatever year that was! So this is nothing particularly new.