To map "diversity" to skin color is superficial and reflects the bias of the viewer more than anything. As a white male programmer I've more in common with other male programmers, no matter what their color, than with male football players, male drug dealers, male prostitutes, male athletes. Skin color has literally nothing to do with it. It's cosmetics.
Gender arguably is more relevant but seriously... there is no bias against women participating in free software projects. It's literally a sport open to anyone, with as few barriers as you can imagine. Age, gender, skin color, origin, perhaps the only filter that reduces diversity is the need for reasonably fluent English.
And still, the number of women in our communities is extremely low. That means the detailed technical world of software appeals to fewer women than it does to men. That's not a problem, it's just a fact, and easily observable. It would be offensive to choose women speakers just for their gender. Tokenism is a nasty form of discrimination. At the same time it would be offensive to refuse people on any basis except their work. I don't think that was the accusation here.
Diversity simply means, different points of view, perspectives, and opinions within the group. It does not mean creating a Star Trek experience.
Then again speaking as a white male it's quite likely that my perception of this is totally biased.