I think there's probably a reasonable argument to be made that a move to a foreign location, even one nominally more "native" than a zoo, is a definite hardship on an animal who has become habituated to a specific environment.
Now, if the "zoo" in question is a 10x10 concrete room with bars, then maybe the quality of life in a larger and more natural (in the sense of less confinement and concrete) environment is worth a temporary disruption.
But what about zoos that give primates large, outdoor spaces with natural accommodations like ponds, trees, shelter and primate experts who ensure their physical health and mental stimulation? A "natural" environment may be at best an equal trade and in some instances worse if it comes with a change in the fellow-species population (change in social status, loss of familiar animals or mates, etc).
I'm not always sure that "natural" spaces really are as natural as their made out to be unless it means putting the animal back in its native environment -- sure, their animals but they can become as habituated to a captive lifestyle as any animal. My dog may love to run free outside, but he seems pretty well adapted to sleeping on the couch and probably wouldn't like being made to live outdoors 24x7 after living his life indoors.