"The more onscreen objects there are the more slowdown there is."
Even when the framerates are fully in order, that one's a kicker: How did the developers ensure that framerates would be adequate on consoles, and on average PCs? By keeping the amount of stuff on screen down. And so we have pop-in, RPGs where a 'city' has maybe 100 people (spread across multiple areas with lots of clutter to occlude sightlines, and various other deviations from either the realistic or the epic, depending on what the occasion demands.
Arguably (as with physics acceleration for destructible environments) that's the more difficult chicken-and-egg problem: If it's just a matter of how pretty things are, I can make it work on weak hardware, and if you have a nice GPU you can crank up the resolution, anti-ailias you little heart out, and set all the draw distance sliders to maximum.
If, however, it simply isn't possible to cater to people who I need as customers if my environment has 500 NPC armies clashing or if castles can be knocked down one stone at a time, with realistic friction/leverage/impact, I can't just 'dumb down' for weaker systems, I'd actually need to rebalance the game, since the weak system version might never have you facing more than 20 NPCs, and can't have any puzzles/requirements that involve destructible environments (or I need a whole separate set of game assets with scripted quest-destructables that just have hit points and 'damaged' textures). It's a different game.
This isn't to say that the effort to support all levels of prettiness is zero, I've no doubt that it isn't; but unless a contemporary rendering engine is downright broken, it should at least be possible for the gamer enthusiasts to render at substantially higher resolutions, more AA, longer draw distances, higher poly models and higher rez textures at greater distances, etc. without any changes to the core game. The same is Not true of changes that require power but are also integral to gameplay. If some people are seeing 'real' backgrounds (with actual NPCs and scenery doing their thing, and the laws of perspective applied) and other people are getting skyboxes and a few low-detail mountains, you can't let either party interact at great distances, or you'll risk changing the game.