Actually it kind of does. The elastic clause gives some leeway there but in the initial view of the country the federal government was supposed to be merely a unifying force bringing together a number of independent states. After the Civil War the power balance shifted to the federal government over the states.
The three-fifths compromise was in the Constitution along with the guarantee that slave imports would be illegal after twenty years (I forget the exact wording)