It would be termed racist. However when you look at the situation isn't it likely that black men are better-built physically due to forced selective breeding by slave "owners" back in the day? Slave "owners" would want to maximize their "investment" in slaves, so why not force "ideal" black men and women to maximize strength and child-bearing features? While the reason for it is quite disgusting (treating people as mere cattle) the results are pretty remarkable, if horrifying (horrifying in the sense that people could be "owned" and forced to mate). This would be micro-evolution at work. Is this actually what occurred? I'm not familiar enough with the history of slavery to know the answer, but I do know of african-Americans in the Boston area (who told one of my friends of this theory) who believe this is the case.
Just food for thought. And believe me, I do not intend to offend anyone with this post.
As far as what is fair: it's not fair that I can't do strenuous exercise for long periods before my muscles cramp up (salt wasting). Should I demand to be allowed to compete since I am a very fast runner, and be given a handicap or other concession for my lack of endurance? Of course not. Athletic competitions have always centered around two things: physical fitness (usually by conforming to "ideal" physical structure) and skill. When it comes to running there is some skill involved but it's more about what one's physique is technically capable of.
If that selective breeding theory holds any water, is it any wonder that black people outperform white people on a consistent basis? Would it be fair to segregate competitions by races? To me that would not seem fair. If white people really see a disparity and are angry about it, nothing is stopping them from marrying based solely on athletic ability and hope that those desirable characteristics follow in the next generation. However, I do not think eugenics can be considered to be justifiable under any moral standard.