Having lived in both the Bay Area and LA and commuted between them frequently, it was almost always less hassle to drive than to fly. Sure the flight itself was short, but the traffic getting to/from the airports, parking, car rental, standing in long lines, being fondled by a stranger in a uniform, etc. was no match for the tedium of sitting behind the wheel for five hours or so. Making air travel anything less than the dehumanizing punishment that it is now may be impossible, given the inertia of the entrenched interests, but I-5 isn't going anywhere anytime soon.
Traveling between pretty much any two European cities (of reasonable size), where you have efficient mass-transit on both ends and no TSA, is basically what you propose for California, and it would be truly amazing... but as you point out, it would require building a mass transit system and linking it to the airport. And if BART and the LA Subway are any indication, that is a pipe dream for a fantasy world where politicians aren't corrupt assholes.
Much like Africa sort of skipped over landlines and went right to mobile, maybe California can pioneer a new form of medium-distance travel that is--and I think this is the most compelling argument Musk makes--sustainable. If this thing can really be powered entirely by solar panels on the tubes, then it could be a nice alternative to (medium-distance) air travel, which is currently the most carbon-intensive mode of transportation.