Drivers don't come anywhere close to paying for the roads. In most US jurisdictions, gas tax, registration fees, etc pay for less than half of road construction and maintenance costs. The rest is borne by everyone via other taxes. And since road wear scales as the fourth power of axle weight, cyclists' use of the road causes essentially zero road maintenance costs. So the cyclists are paying far more than their share of road costs and are heavily subsidizing drivers.
In many other countries, cyclists and cars share the roads just fine, with auto-bike accident rates per cyclist mile which are orders of magnitude lower than the tragic accident rates in the US. Very little of this is due to cyclist behaviors; almost all of it is because in those nations drivers know cycling is a normal mode of transport and they need to treat cyclists as fellow human beings.
The speed mismatch between a 20mph cyclist and a sidewalk 3mph pedestrian is unworkable, because sidewalks have not been designed with directional lanes, buffer zones, and passing in mind. In many jurisdictions it is illegal to ride on the sidewalk. Sidewalks, not having been designed for tires, are also inconvenient riding surfaces unless you're going quite slowly. (Did you know the reason we started paving roads was at the behest of cyclists, decades before cars became popular?)
Passing a cyclist may slow a driver's trip by a few seconds. The same is true for any slower-moving vehicle (tractors on rural roads, even horse-drawn vehicles in Amish country &c), all of which, like cyclists, have the same right to any non-freeway road as auto drivers do. People can cope with that just fine, if they don't have ignorant and arrogant attitudes.