Why do you presume that what someone does "in their private life" doesn't really affect others? Humans are socially interdependent. What happens in someone's private life very, very rarely _doesn't_ bleed over into the public sphere somehow. There is almost always a conflict between majority-minority rights even when the primary activity is "in private" because it drastically affects social convention outside. I don't believe you're really ready to accept the logical consequences of your statement.
Take marriage and homosexuality/heterosexuality, hypothetically. There are plenty of people who don't give two shits if you are have gay sex or want to get married. But they don't want to have to subsidize your lifestyle or be force to participate in it economically. There are various laws encoded that give special consideration to married couples with regard to insurance, social/commercial discrimination, adoption, etc. Right now those laws represent a majoritarian, heteronormative social convention and so most people don't mind them. But if you aren't in the majority, you realize that these laws subsidize people that conform to a social norm. This is in fact why so many gay people want the rights of the married, because they are unjustly excluded from these benefits.
Now look at the other side of the coin. Expand the definition of marriage, and suddenly it's subsidizing non-heteronormative behavior. Now some people realize they are paying to subsidize behavior that they don't agree with. Yeah, they are hypocrites. But it's eye-opening because you learn how society and law actually promotes some so-called "private" behavior over others.
So my question to you is, are you willing to put your money where your mouth is and dismantle a wide range of social services and safety nets that subsidize _with other people's money_ some people's private sexual lives? One's sexuality may not be a choice, but undeniably, society's decision to subsidize one definition of marriage over another is a choice, and one that forces others to participate in it economically.