The real problem is that most of the big themes in classical SF require vast amounts of energy. And that's not happening. There hasn't been a new source of energy in fifty years, just
marginal improvements in the old ones. This matters.
That's why space travel is a bust. With chemical fuels, it will never be more than an overly expensive, marginal enterprise. The better '50s SF writers all knew this; read Heinlein's "The Man Who Sold the Moon". They just assumed that, somehow, the energy problem would be cracked. Didn't happen. So space travel remains an expensive ego trip for countries and billionaires.
Industrial civilization is only 200 years old. 1808, the first time someone bought a train ticket on a commercial railroad and went someplace, is a good starting point. Industrial abundance, being able to make more stuff than people could consume, only goes back to WWII.
During most of the 20th century, "progress" was a big theme. We don't hear that phrase used much any more.
The number by which one measures "progress" for the average Joe, "per capita median real income for urban wage earners", peaked in 1973. (Median income, not average income; the average is biased by wealth concentration to rich people.) Back then, a guy without a high school diploma could get a job at GM and make enough to buy a house, two cars, a boat, and an education for his kids. That's over. (You don't see that number mentioned much any more. It was heavily publicized back when the US boasted "the highest standard of living in the world".)
Now we're starting to run out of energy and raw materials. Nobody serious thinks there's enough left to sustain current output for another century, let alone bring China and India up to US levels of consumption.
It's hard to write good SF about "the great winding down". It's been done, but it's not read much. The glory days of SF coincide with the period during which "progress" was a win for the little guy.
That's why SF is dead. The plausible future sucks.
I think you're right, in a lot of ways. However, I suspect a chunk of the problem is that the best path to better energy begins with that N word people are so afraid of embracing. Our society has discovered a new form of fire, and it scares us. Until we're willing to actually embrace it (dangers of use and all), we're going to be stuck in our caves.
--Jimmy