It is, generically (not necessarily in this particular context), because of historical context. It is the same reason "Nigger", "Faggot" and "Gook" are offensive.
In this case, apparently, Tottenham's fans took a derogatory term and injected new meaning into it (for which I say "good for them", BTW).
Does that mean that I can go running around calling people "filthy chocolate-loving whiteys" and if I do it with enough conviction, I can get arrested?
Two things. First, actually reading my comment would reveal that I am actually against criminalizing derogatory names. The other thing is that you, alone, would do it, it would probably not become derogatory. If, however, it became a common way to degrade whatever group it was you meant for it to refer to (as pure guesswork I'd say it was Caucasians), then, yes, it would become derogatory.
The beauty of not criminalizing name calling is that it saves us the trouble of developing an objective test for what is derogatory, and leave the decision to take or not take offense to each individual. That is also just one of the many many problems of criminalizing name calling.