I do not agree with your statement's overall claim.
However, in order to make you feel better, you can think of these space ventures simply as income redistribution. The "rich guys with a fantasy for space" and "wealthy people" you refer too spend money doing this. Lots of money. This money goes to high-tech jobs that pay well, that in themselves allow money to be distributed. If more of the 1% spent their money this way, there would be even more money distributed around, driving the economy that you and I derive our incomes from.
So rather than being negative on this, you should be banging the drum to make more and more of the 1% interested in this. Make those 1%ers prefer to spend their resources acquiring expensive services that feed the highest paying technical staff, We don't want those folks spending their money on things or on low-value services, we want them to spend them on services that can only be provided at great cost by technical people. Get those bank accounts spending money on high tech services provided by your neighbours !