It's more complicated than that. There are those who believe in genetic diversification, and see genetic manipulation as a threat to that, much as they see GM crops along those lines. This is more of a niche ethical argument, but it's out there.
Additionally, and this is the ethical argument that Charles Krauthammer, (hardly a "spiritualist" and he's pro-choice), that it becomes an ethical dilemma if we create life simply to destroy it. At that point, there is a breakdown in the fundamental moral underpinnings of our concept of "natural law" and fundamental rights, and you encourage a very real threat from an ethical slippery slope.
One of the common arguments against cloning and genetic modification from Christians (although it is espoused by non-Christians as well) that society will become increasingly intolerant of "defects", and that people considered such (like those with Down-syndrome, or even physical defects) will be considered "sub-human".
Lastly, there is a worry that the "human" status of those cloned, despite being human, will be less than we attribute to those we consider "human".
Our "enlightened" view of humanity took a long time to achieve. A lot of people (myself included) feel that we are pushing the limits of our shared morality is capable of dealing with. The fact that a lot of people aren't even considering the implications of these scientific advancements don't do much to alleviate that concern.