Communication goes two ways, and offense is in the ear of the behearer if you will. If you didn't mean to offend, then you made a mistake or you don't care. If you didn't mean to offend once but now know that you're offending, then either you don't care or you intend to offend.
Wise people will make a good-faith effort to avoid giving offense by learning and practicing diplomacy.
If your view that intention is all that matters were true, then all the racists in the world could just intend really hard that no one should be offended and we'd be peachy.