It's like asking a random "Christian" to justify and explain Westboro Baptist's behavior because they both think of themselves as "Christian" despite that term meaning vastly different things.
Close, but not quite. The difference is that "the assholes" in your example (WBC) aren't the only one setting dogma. Using the same metaphor, it's closer to you objecting to having "believing in the divinity of Christ" held against you, even though you don't, and you feel that calling yourself "Christian" just means "Being a decent person"
Feminism is not (just) a "philosophy", it's a proselytizing ideology: the goal is to bring people into their way of thinking, and that means that the way it's presented trumps dictionary definitions, trumps NAFALT, and trumps personal gnosis on what "feminism means". And as far as presentation goes, sorry, but the "assholes" are ruling that particular roost. "Mainstream" and/or "media" feminism is spawned in pits like Jezebel now, and I don't think anyone needs to be told how that sort of lot deals with being called on their bullshit.
So no, I don't think it's unreasonable at all, when someone claims the moniker of "feminism", to hold their tacit complicity against them.