One could point out that there are fewer instances of white males being miscategorized.
White males are just about the easiest faces to categorize. They tend to have short hair that doesn't obscure facial features or create oddball shapes that confuse the classifiers. Their skin tone makes photographing them and finding edges extracting features easier than it is with darker skinned people. White people have a greater variety of eye colors that can be used to distinguish among them. "White guy face" is just about the optimal case for this problem. If I had to come up with a worst case that was also a photo of a fairly "common" person, I'd go with "dark skinned, brown eyed person wtih long hair and facial hair." That's a pretty clean sweep of all of the variables that make this a hard problem.