Not everything is about you. As a man, you should not be offended when those who do rape get called on it, just because they are also men.
Consider: African-American men are, statistically, far more likely to commit murder than Caucasian men are. Does an African-American man have a right to call racist bullshit if you tell him to stop murdering? Of course he does, and that in no way implies that he doesn't want those who do murder to get called on it.
In exactly the same way, men are, statistically, far more likely to commit rape than women are. Does a man have a right to call misandrist bullshit if you tell him to stop raping? Of course he does, and that in no way implies that he doesn't want those who do rape to get called on it.
I'm not saying you will, but it's possible you may surprise yourself one day.
You're just illustrated a huge part of the problem: a belief that ordinary men somehow, to their surprise, suddenly turn rapist someday. This myth is at odds with what we know about rapists: they are deliberate repeat predators with a pattern of offending from a young age and a high probably of cross-offending.
It's why the whole notion of "rape culture" around which so much of this discussion revolves is a distraction: rape is not the result of ordinary guys made confused by their culture about consent, it's the result of deliberate acts by violent assholes who know quite well what they are doing, and all the hashtags in the world won't change them.
If we actually want to stop rape, rather than have a feel-good self righteous flamewar, we need personal safety and bystander intervention training.