On the contrary, a lot of people want draconian restrictions like mandatory licensing and restrictions on sale of such vehicles.
Perhaps I misspoke. I should have said that I've not heard from anyone who wants this, and I have talked about it with other pilots.
except relatively wealthy ones with a ton of time, like yourself
Your assumptions are showing.
While there are wealthy pilots, most of us are of modest means. My (small) car is paid off so I spend the equivalent of its payment on my hobby during the on season, and that amount will go down once I finish my license. I won't need to buy my own glider outright, but if I do decide to do so, there are perfectly adequate specimens for sale in the $10k-$20k range.
As for time, I fly one day per week - sometimes two, sometimes zero. On the days I do fly, I still have time to mow the lawn, cook dinner, work on household projects, and even watch a movie with family.