I am happy to pay taxes to build low-carbon infrastructure. Earlier this year my old car was totaled; I didn't buy an expensive new fully EV, but bought a relatively cheap used plug-in hybrid after calculating that it would let me drive 70% on electrons.
The EV community is well aware of, and quite grumpy about, the proliferation of high-end EV's in the US. Many more affordable electric vehicles exist elsewhere that are not sold in the US -- Chinese ones, yes, but also smaller models that are not sold in the US because Detroit and importers perceive (wrongly, in my opinion) that American buyers only want large expensive models. But the average purchase price of a new car in the US is to my mind ludicrously high -- around $47K. This is higher than a Model 3, for instance. So the most common EV option is actually cheaper than the average purchase price of a new car.
The cost of climate mitigation will necessarily be borne by people who won't benefit as much -- wealthy people (who tend to live in cooler climates) need to stop burning fossil fuels so that poorer people (who tend to live in warmer climates nearer the Equator) don't suffer so much from global heating. It is perfectly reasonable to expect the wealthy to pay the cost to stop injuring the poor. The impact of unmitigated climate change on the global poor is going to be staggeringly bad; there is no reasonable path that doesn't involve mitigating it.
I think your last assertion may be true when comparing the American wealthy to the middle class, but not true when comparing the middle class to the working poor. The wealthiest people don't have fixed working hours while the middle class is usually doing a 9-5. But the working poor (who more often work shifts that are not M-F 9-5) have access to off-peak times; they are not "getting home at 6pm, then drawing 3kW for their stove and another 3kW for their car" as much as the 9-5ers in the middle class.
Time-of-day pricing lowers the cost of generation for everyone by better aligning supply with demand, reducing the amount of power that has to come from (expensive and inefficient) peaker plants.
The bigger issue is the availability of charging infrastructure in places where poorer people live, like apartments. I live in an apartment and have to charge my PHEV at work, paying 50% more per kWh than I would pay at home (and that's without time-of-day adjustments).