I'll admin my initial post was does somewhat to act as a provocateur (though not a troll because I am genuinely interested in discussing the subject).
Its my personal view that government should get out of the marriage business. We should simply pass a low recognizing all existing marriage licenses as "Civil union licenses" and convert all marriage rights next of kin, child custody, 5th amendment testimony protections etc, to civil partner rights. Than any two people regardless of sex or gender can enter a civil union. If they are married or not is between them, their clergy, god(s), friends, and the guy who operates their car wash etc.
I find it interesting that various arguements about how having two parent house holds is good for children but the gender of parents does not have so much impact are used to justify arguments in favor of allowing homosexuals to marry. Making marriage a right implies that adultery and act that frequently makes impossible for the partner to remain in the marriage; implies that partners rights are being infringed.
Additionally the state does have pretty clear interest in promoting monogamous sexual relationships issues of morality and gender again set aside there is a clear advantage in the prevention of the spread of disease. Again since we are all "responsible" for each others healthcare costs now it seems perfectly reasonable (within that context) the state should favor policy that prefers monogamy, independent of what you call it; marriage, union, cohabitation contract.... The state has a pretty clear reason to want to encourage the formation of partnerships for raising children.
As other have pointed out we don't allow people to enter into contract of indentured servitude etc. Again I would come down on an the side of an individuals absolute right to make a contract but society has broadly chosen otherwise. A societal harm is the final underlying justification for most criminal law. Adultery is clearly harmful in the majority of cases. Its hard to escape the conclusion that if marriage is sacred enough to be considered a right that an act which harms its integrity should not be considered a crime.
I don't think your assertion that re-criminalizing adultery would negate the value proposition of marriage, union. There are lots of advantages to having a state recognized relationship (of some name). Child custody, the right to inherit property, tax filing advantages, social security, just to name a few.