The justification used by the anti-spoiler group is that because most of the articles don't have the removals reverted and that this is a de facto consensus that they're not wanted. Others counter-argue that many may not have seen them yet or may be afraid to revert because of them being afraid to get into a revert war with the anti-spoiler group who has invested much more energy in this.
In some cases, reverts were performed restoring the spoiler warnings, but they were quickly reverted back. There are also allegations of users being threatened by admins when reverting.
For more information, read the mediation page and talk page on spoiler warnings.