Private pilot (PP-ASEL) here.
I am certainly NOT the 1%. Learning to fly is not more expensive than a LOT of other hobbies. The very first time I flew in a small aircraft was as passenger with a student, who certainly wasn't rich (very old scrappy car; that was in Germany - so he didn't have to pay high tuition fees), he tried to get acquaintances to fly with him and share some of the cost of most of his flights.
Hourly costs sound high, but that is the time the engine actually runs. If you do aerobatics instead of (extremely boring) cross-country flights very few if any of your flights will be longer than an hour - and that's more than enough for the weekend, it's quite exhausting. That's roughly $200/week.
You CAN spend unlimited amounts of money, same as in sailing/yachting, but you also CAN participate in these hobbies on a limited budget.
That said, I stopped flying, part of it was THIS (article) and the noise my airplanes create. When I hike through Yosemite (or anywhere) the last thing I want to hear is small aircraft noise above, but it's sooo common these days. I just don't want to be that guy.
I also stopped horse riding, for similar reasons, but for the horses sake: I did not want the responsibility of owning one, and I don't want to support the horse industry where most animals spend almost all their time in small boxes. Those animals need lots of space on grass land, that's rarely what they get, at least not close to large cities (I did most of my riding in the SF Bay Area, some Oakland Hills stables) where there are lots of people keen on riding but not enough space to give horses a natural environment. Both flying and riding are VERY fun hobbies, unfortunately there is a considerable cost - to others.