some people only count as 3/5ths of a person
Have you ever wondered why? It's because they were trying to reduce the influence of slaveholders. A default position of counting slaves as a full person for representation purposes would have led to the slaveholders (who actually voted for representation, not the slaves) controlling the federal government based on the number of slaves they held.
So the 3/5ths compromise as well as granting the power to restrict or prohibit the importation of slaves (also in the Constitution) were the Nation's first two anti-slavery measures, passed over opposition from the slave-holding States. They'd have done more, but then the slave-holding States wouldn't have ratified the Constitution in the first place, making any restrictions in it pointless.