By your logic, there is nothing wrong with forced abortions, because an embryo is just a mass of cells that "could potentially become a human with the right signaling factors and growth conditions." It is exactly the same as removing a tumor from a person without consent. That is just silly.
Observable facts inform our morals, but they do not decide them. You cannot derive should from is. Many people believe that because the natural course of embryonic stem cells leads to a human being, they should be considered a human being. If you disagree, you need to tell them where the dividing line between human and not human is. You know any such line is arbitrary. Until we decide on where that line is a society, doing anything within a reasonable range of that line is morally questionable. That does not mean immoral, just questionable. If you want it to be unquestionably moral, prove to a majority of the population why they are clearly not human yet.