Of course some do. Please show me a group of people without any assholes in it.
But in my experience, the BDSM scene is very big on consent and thinks and discusses it much more than vanilla people tend to do. And yes, of course I know that there have been, for example, some weird sex-cult style things going on that also used a "Gor" background, but the simple fact remains that most Goreans are not like that. Same thing as with religion, for example. Bad apples exist, yes, but most religious people aren't them.
And yes, what you say - evaluating everyone carefully - is actually what is done or at least suggested in BDSM: Vetting, covering, etc.
TLDR; Assholes exist in every group. Most kinksters (including Goreans) I know aren't.
All great ideas are controversial, or have been at one time.