Oh, and another reason: if you make 1000 times what the average middle class worker makes, you're alright when you're let go. I could make it 6 months without dipping into retirement, probably. If you can get fired from your job and say "eh, I'll go out on my sailboat for awhile" it's a very different experience, I'd say, even if it were embarrassing.
I think you've made several mistakes in attempting to reason away your callousness. For starters, you seem to lack the comprehension that middle class income can't even afford the property tax on upper class homes; let alone other bills such as utilities, communications, and travel expenses. I know people that earn so much more than me, that they pay more in taxes than I actually earn. However, I don't pretend that these people can afford not to work (or lose their jobs).
Now, maybe you think everyone should be forced to live on middle class wages, and never acquire more than middle class homes, cars, lives. I'm sure glad I've had the opportunity to Baron Von Munchausen myself out of the swamp of poverty and earn enough success to pay for some luxuries.