Honestly, when I think of why prejudice is bad, it's not because "all subgroups of society are equal on average", that's definitely not true socio-economically and it's probably not completely true genetically. The reason prejudice is bad is because within each subgroup the range of possible qualities a person can have are huge!, where as the differences between the averages of subgroups are almost negligible. So if you go around comparing everybody by the average of their subgroups your going to be right not much more than 50% of the time. Whereas, because of the huge differences between individuals, you only need to spend a small amount of effort to make decent comparisons on individual merit, which is what people should do.
Unfortunately I think this can be a bit hard to explain, so for the most part we just say "all subgroups of society are equal on average" and hope that the result is the same. :(