I'll buy that anti-discrimination laws create economic drag in a healthy society, but recall that the reason most of these laws were created is because there were widespread attempts to keep large portions of the society down.
If you refuse to hire a talented black man, and I then hire him, it will only give me an economic edge if I don't suffer a social backlash because half of the society refuses to do business with me, 'because I hired a nigger'. This is exactly the conditions that required legal addressing with laws.
Now, while we aren't in the same place now, discrimination still exists, and nobody knows exactly where the tipping point lies, in terms of the laws changing from a social enabler to a net economic drag. Since the economic drag is relatively low (compared to say, 10% of the population being destitute and disenfranchised), it makes sense to keep the laws until such time that we can be sure that repealing them won't cause widespread social problems.