Yes! And his objection to it has nothing to do with racism as you like to think it does. The federal government certainly doesn't have that right. If you can find me the clause in the constitution that suggests that they do have that right to tell private businesses who they must serve, I would retract that statement.
Understand that the Civil Rights Act was put in place in response to other government involvement in private enterprise which forced a lot of businesses to be segregated against their will (The transit system in Montgomery Alabama is a good example). All Dr. Paul said was that government should not be in the business of telling private enterprises which customers they need to or need not to serve. Let the free market prove to business owners that attempting segregation is a guaranteed loss.
If you think the Federal Government SHOULD have that right, then you should favor a constitutional amendment to take care of that.
This is why Ron/Rand Paul are difficult to connect with. Libertarianism requires greater intellectual vigour and deeper analysis to understand.