The assumption is that if flying cars were common, there would be vastly more locations. As they basically function like helicopters (in most conceptions - VTOL), they need only something equivalent to a helipad, not an airport. Which is much cheaper and smaller footprint than an airport.
To get to the point of allowing takeoff and landing from, say, a driveway, you'd have to have a long track record of excellent proven safety, and levels of noise reduction that current technology doesn't yet support. It's certainly conceivable in the future, but is anything but a first step for companies working on flying cars today.
I personally view flying cars as pretty much inevitable (although not around the corner) regardless of whether or not they're pursued directly at present. Namely because of delivery drones. Businesses are not going to stop pushing for them because there's such an economic case for them (not having to drive a big truck around city streets, pairing trucks with drones to not have to go down each sidestreet or stop at each location, etc), and they'll advance the technology as needed to get approval - starting small. But economics will continually push them toward making larger and larger models, and the technology to get approval for those. And eventually you'll have models large enough to carry people around, wherein the question will inherently arise, "Why, exactly, aren't they carrying people?"