It is true that people may be content to advance at the expense and misery of others. In fact, it is rather common today, even in cultures that have a history of other-centered ethics (virtue), to glamorize such behavior (precisely because in those cultures there is an active abandonment of the foundations that produced them). And it is all done with the logic you suggest: "What is me and mine is right and good."
The problem with that is how we treat the one is how we will treat the many, and people know this. So, if you can abuse someone we both think is in our mutual "outgroup" I quietly know that you have the capability to do it to anyone, even those in your "ingroup" which includes me. "Us four and no more...and it might only be three of us."
Evil has a corrosive effect that cannot be limited to my "enemies." That's why integrity really does matter.