I think the exact opposite is true. A phrase is at its least derogatory when it is so ubiquitous that even those that were once being degraded are no longer aware they should be offended.
That is the point when the word or phrase has reached cultural neutrality.

A word is at its most derogatory when it is used with full malice and prejudice.

It is an unfortunate circumstance when ignorance of the original meaning of a word or phrase allows its use to become commonplace in an alternative vernacular.

You have it wrong yourself.
If you plot the political spectrum on a Cartesian plane with conservatism toward the right end of the x axis and liberalism toward the left end, the y-axis would have libertarianism toward the top and authoritarianism toward the bottom.

Extreme libertarianism is called anarchism
Extreme left leaning authoritarianism is called communism
Extreme left leaning authoritarianism is called fascism
Socialism is left leaning and toward libertarianism
The Nazis while called the national socialist party are generally considered to be fascists or conservative authoritarians.
Ultraconservatives tend to be more fascist than anything else.