Lots of reasons. Because polygamy generally involves powerful men taking lots of wives, and sometimes forcing young men out of the community so they have their pick of women. And no, it's not going to even out of a few Marisa Mayers take on a few "brother husbands."
Because it creates an inherent power imbalance in the relationship - if you call your wife Sue a fat bitch she can stop speaking to you until you've done a few weeks of groveling. But if you're a polygamist, you can tell Sue that if she doesn't get over it, you'll just be fucking your second wife, Molly.
Because it takes issues that are otherwise straightforward and turns them into a mess if there is no will. If you get in a car accident and end up in a coma, your wife can make medical decisions for you. If you die, she inherits everything and has custody of the kids. But if you're a polygamist who adopted, who gets the kids...Sue or Molly? Who gets the house? Which one makes the call to keep you on a feeding tube while you're in the coma?