I suspect that it has something to do with the fact that the really wealthy generally spend less time whining about taxes and more time creatively structuring their assets so as to not pay them; and can also afford, if they want, to live under just about any tax structure worldwide, so their choice of residence and citizenship is based more on preference and perceived advantages than on economic necessity. They may still dislike paying whatever taxes they can't avoid; but not enough to put themselves to any great personal inconvenience or hardship about it.
The less wealthy, but wealthy enough to feel exploited by the tax system, whine more loudly because they have fewer options for lowering their effective tax rate(high-salary skilled workers, say, tend to resent getting taxed at income tax rates rather than capital gains rates); and because they are relatively poor enough that the marginal value of the assets they lose to taxation is higher(if you are really rich, taxes may offend you in some abstract sense; but they don't really change your ability to enjoy basically anything money can buy; if you are merely wealthy, taxes aren't putting you in the bread line; but they quite possibly are reducing the range of things you can afford.)
The somewhat less wealthy are also presumably less able to insulate themselves from their environment. A suitably large fortune will keep just about anything except the central government at bay(this is why so many Russian oligarchs hang out in London); but the smaller the fortune the lower the degree of cost-effective insulation you can manage. This makes Jamaica's relative poverty, very high crime, poor corruption ranking, and mediocre HDI somewhat less attractive.