I cannot do the equivalent and just move to live in a slaved up country so that I have a maid etc.
Why not?
Visa workers come here, work long isolated hours for 5 years, and then retire back home as a rich person because their cost of living is so much lower. They have options I don't.
Really, there's very little preventing you from amassing your fortune here in America, then moving to a developing country where you can buy a mansion, staff, and decent security (which you will likely want) for a few thousand dollars a year. For a few thousand more dollars, you can buy yourself the favor of the local population, by honoring the local leadership, building infrastructure, and/or investing in local businesses. Be a guest at weddings, and offer lavish $20 gifts. Visit the local bars, and make friends with the local folks. Patronize the local artists, and learn and appreciate the local culture. Don't pay your employees substandard wages, don't threaten layoffs, and don't demand absurd hours. Pretty much, become the 1% to the community, and behave how you'd want the 1% to behave here.
You get several years of luxury (depending on your savings, working capabilities, and retirement plans), and the locals working for you get high pay without having to work long isolated hours over here for 5 years. Everybody wins, as long as you're careful not to be detrimental to the local society.
Ah, but you probably don't want to. You probably want to stay near your family and friends, and be with your local culture, and living where a personal security team isn't necessary. It's not that foreign workers have options that you don't... Rather, they are willing to accept options which you won't.
The whole point of free trade is to help equalize the economies of the world. As an American, you see only the drain on your local economy, but as a citizen from a developing country, you would see the influx of money, and the improvement that can (with appropriate guidance) come with it. You can free wage slaves, cover the sewers, etc. As the vehicle for the money transfer, you have the opportunity (and in some folks' opinions, the duty) to decide how the money is directed. If you personally benefit, especially in terms of luxury or prestige, then that's the return for your time, isolation, and management work.