No it shouldn't. If gender is a predictor of ability then the probability distributions are BY DEFINITION not independent. If therefore you use the knowledge of gender after evaluating ability then you are treating them as independent variables when you combine them. This is mathematically bogus.
Actually, that's just mathematically simplistic. Here's what your reasoning does not account for: There are leanings, abilities and competencies that do not exist in isolation from other influences. Gender can be one of those. Therefore, to the extent that affect is possible, it is a valid consideration.
It could be a positive for either sex.
For instance, the air force has definitively determined that females are significantly better at maintaining more comprehensive situational awareness in complex aerial situations. This is because of a real world gender-based difference in information processing.
On the other hand, if one was hiring a bouncer, the competencies lean strongly the other way.
There will be outliers, of course, but that's why we need to think about these things rather than operate by rote. The law, unfortunately, but needfully (due to blind prejudice), specifies decision by rote. This is why many parts of the decision making process have gone missing from public view.