There are two big things that can help electric cars (does Musk also include plugin hybrids?)
1) like you said, is the cost of gas. It's not going to stay at $3.50 or $4 a gallon, in 20 years, we may well have hit oil's terminal decline phase, and even if not, the cheap stuff is gone. Expect at least $10/gallon in current dollars in 20 years.
2) the cost of batteries. They're going down. Fast. see: http://www.plugincars.com/lithium-ion-battery-prices-drop-160-kwh-2025-123193.html
I actually used the 8% growth in energy density claimed above, which brings my price to $220/kWh in 2025 (not $160 like mckinsey claims). At that price, a 100 mile range battery will cost $8000, down from about $20,000 today. By 2032, prices could come down to $4500 for a 100 mile range. This assumes exponential gains in capacity/density/etc, but those gains have happened in the last ~15 years. The big question is whether they'll continue in the future. I'd say we'll get to about $250/kWh before battery scientists slow down, but improvements won't stop. That will get us to some pretty cheap, reasonable range electric cars in 20 years. Basically, the Nissan Leaf will go from $35,000 ($20k for the car, $15k for the battery) to $25,000 ($20k for the car, $5k for the battery)
[assumptions: no EV efficiency gains, still 35 kWh/100miles, which is what the Leaf, Volt, and plug in Prius achieve today.]