You don't get to decide when a word is pejorative to a group that's historically been targeted with it. I agree strongly with George Carlin when he talks about the ludicrousness of "bad words." There are no "bad words." But you know what there *are?* There are words that have been used offensively against a minority group so often that they've become hurtful *to* that group of people.
You have a right to use those words anyway. You have a right to not care. You have a right to claim that because YOU don't find the word offensive, no one else has a right to do so, either.
You also have a right to decide that decades of discrimination against a particular group were so awful, you'll avoid using a word or two -- not because those words are "bad," but because they serve as reminders of abuse, insults, and ignorance. You have a right to decide to change your speaking habits *ever* so slightly as a way of demonstrating to this person or persons that you don't agree with the way those words were used against them.
You have a right to decide that empathy and acknowledgement is more meaningful than saying a certain collection of phonemes.
Or not to.