Except that it was never common to own slaves. Slave ownership was primarily among Southern aristocrats--your average white Southerner wasn't rich enough to afford one.
Would you agree with the statement "it is not common in the U.S. today to own an SUV"?
In suspect most people would disagree with this, and say that SUV ownership is common. The average number of vehicles per household is about 1.9, and the fraction of passenger vehicles on the road that are SUVs is 11%, so we can estimate that one in five households in the U.S. owns a SUV.
In the slave holding regions of the U.S. one in three families owned slaves! In the seven states of the lower south there were as many slaves as there were white people, under such conditions how could it possibly be that ownership was rare?
I submit that the statement "slave ownership was primarily among Southern aristocrats" is essentially a tautology - it is true only if you consider most anyone owning a slave an "aristocrat".