Falls start to become fatal from about 50 feet, and are nearly always fatal from above 100 feet. So for flying cars to be reasonably safe, you'd have to limit them to about 50 ft altitude. Factor in uneven terrain, and that altitude ceiling means there's very little advantage to flying cars vs ground-based cars.
It's also worth pointing out that Musk's Boring idea is the same thing as flying cars, except the cars are below the ground instead of above the ground. Both increase the number of cars which can pass any 2-dimensional point by changing the altitude - up for flying cars, down for Boring. With Boring having the notable advantage of car occupants not dropping to their deaths if their engine conks out.