You make a compelling argument for religion - anyone who has spent time studying God more trusted than what "stands to reason" in your own judgement :)
Now, if it was simply saying "hey, some hate groups use pepe the frog, and some hate groups use swastikas!", that's one thing. But it strains even the most basic credulity to assign such a thing as "hate" to a mere symbol, stretching the claim further to "any use of pepe the frog or swastikas means you're a hate group!", which, whether ADL intends it or not, is their end effect.
To put it in the terms you used, it is irrational to give any sort of credibility to a group that proclaims words, shapes, or forms are so poisonous that the mere usage of them makes you hateful. There is incredible amount of context that is simply stripped out, that can have no rational justification.
tl;dr - just because a racist black gang uses the #13 as their symbol, doesn't make every use of the #13 a racist act. We can describe racists groups in detail without ascribing their motives to any and every use of a symbol.