When they are talking about the economy 'as a whole' they may be referring to the world economy and indeed free trade is good for that. However if what you are worried about is your own piece of the pie as a highly privileged elite who benefits greatly from artificial borders and the resulting income inequality then that is not necessarily a good thing for you personally.
Maybe someone who will be happy to work for $1/hour can do your job as well as you can. Would that be good for the world economy? How could it not be? But it won't help you specifically if you want to live a good life. I think there is clearly at least some zero sum game aspect to the current world economy. What is good for Indian and Chinese people is not necessarily good for North Americans and Western Europeans and in fact may often be very bad indeed. But for the planetary economy and overall human progress it's great.
Eventually the people accepting the $1/hour wages will demand a raise and the market will stabilize at a higher level, but that level will probably be a lot lower than it is now. Probably the current mega-corporation wage-slave-worker-drone model is partly to blame for all this. It's really not a very good model for the majority of humans. Corporations have way too much power compared to individuals and behave like sociopaths.
Nevertheless I'd be interested to see what would really happen in a world entirely without borders where everyone was allowed to physically live and work anywhere they wanted. It would be an interesting experiment. I'm not sure such a world would really be that different though because most poor people don't have the money for airline tickets or other international moving expenses and don't have good enough educations to really compete with people educated in first world countries. Although presumably some of the HR drones may not be able to distinguish between well educated and badly educated applicants. And it's not like the whole world would suddenly get better at speaking English.