"I will kill my neighbor" is illegal. "I wish all jews were dead" is not illegal. A flag could hardly be taken as a specific act against someone.
it's also usually a statement that, with or without a government that supports them, they'd encourage and possibly participate in terrorist attacks on ethnic minorities.
So? Should we throw Sarah McLachlan in jail because she does commercials on preventing cruelty to animals, and those people often join PETA and attack research facilities? A flag is an association, not a statement of actions. As such, it can't ever be taken as a specific threat.
I do think that very obvious shows of support for Nazism should be illegal (not just in Germany), simply because they cause people to fear for their lives.
And Christians bomb clinics, so I want Christianity outlawed. Muslims blow up planes. Jews shoot Muslims. Chinese shoot Tibetans. Ban them all because someone somewhere may think that the flag/cross represents hatred and violence. The choice is that or nothing. And I choose nothing. Of course, you could take the current situation where some things are banned and others not with no regard to actual effect in today's world. But that arbitrary and capricious scheme is abhorrent.