I am not a polygamist personally, but the arguments look the same to me. This is just contract law. It seems silly to me to argue that one group of consenting adults is entitled to certain legal benefits and not others, just because some people don't believe in their choices.
Your opposition is based on the fact that some people in a group are bad, therefore, the entire group should be deprived of benefits enjoyed by the rest of society. Instead of addressing the problems, you are attacking the entire group. There are gay couples that abuse their spouses and children, does this mean that all gay couples shouldn't be able to get married? I just don't agree with this premise.
Polygamy inherently devalues women, because instead of being half the relationship, now they are 25%, 15%, 5% of the relationship while the husband always remains at 50%
As the other respondent pointed out, polygamy does not just mean one man and many women, the opposite is possible as well. Another way of looking at the relationship is as equal partners. If the marriage was legally sanctioned, then each member of the marriage would be legally entitled to equal benefit, inheritence, etc. If one member attempted to capture more than their share, then legal recourse would be possible, whereas today it is not. It seems to me that by making the relationship illegal, it removes possible legal protections to remedy this issue and helps perpetuate the very problem you want solved.
This isn't a question of decency or history, it is a question of legal contracts and free will. Shouldn't consenting women and men be free to enter into any relationship they choose (note that these relationships involve legal consent; something an animal, toaster, or child cannot do)?