We're already there. The average city commuter can easily just do a single fast charge once a week, and unlike a gas car, you can just find a charger that's near something else you'd be doing anyway, like at a grocery store, mall, or down town, and your total 'charging time,' defined as 'the amount of time you, personally, have to devote to the process once you roll up to the dispenser' is 'thirty seconds to plug the car in and tap your payment card' and 'thirty seconds to unplug the car and close the charging port.'
The twenty minutes the *car* spends charging is not time *you* are spending charging. You're doing something else while the car charges. Pumping gas might 'only' take five minutes instead of twenty, but that's five minutes that *you* personally are spending.