During most of the 20th century, "progress" was a big theme. We don't hear that phrase used much any more.
The number by which one measures "progress" for the average Joe, "per capita median real income for urban wage earners", peaked in 1973. (Median income, not average income; the average is biased by wealth concentration to rich people.) Back then, a guy without a high school diploma could get a job at GM and make enough to buy a house, two cars, a boat, and an education for his kids. That's over. (You don't see that number mentioned much any more. It was heavily publicized back when the US boasted "the highest standard of living in the world".)
That's why SF is dead. The plausible future sucks.
Although I would offhand agree with a lot of things you said, the US census bureau seems to dissagree with you and sees raising median real incomes:
http://en.wikipedia.org/wiki/Household_income_in_the_United_States, specifically
http://en.wikipedia.org/wiki/File:Household_income_65_to_05.png
Maybe you're over 35 too, from here on, the world will always be a worse place than when we were young :D
But seriously, there was a lot of gloom and doom in the seventies and eighties, when everyone expected we'll die from pollution or simply exterminate ourselves in an atomic war (I have several dozen novels dealing with that possibility alone). Still there was lots of great SF.
I don't think there is less good SF around, its just that the future has become more complicated. I mean, these days William Gibson is not writing SF anymore, he's writing plain novels."At some point there, we left the present and entered the future" (http://xkcd.com/652). I'd say its the abundance of progress, which makes predictions so hard. Come one, compare the youth of someone in the 80ies to todays kids: always online? always being able to chat, mail, watch porn, play, flirt, and reserach for homework? Time is moving fast.
Regards