If I want to marry a person of the same sex, how does that affect you?
If affects me because marraige is a social institution, by definition. If you & your partner were isolated on an island, the concept of 'marriage' would be mute. Other people (aka society) interacting with you forms part of the definition of 'marriage'. So, my answer is, of course you should be able to 'marry' any consenting adult, but you should not be able to force me to recognise your relationship as marriage.
Hypotheticals can make great arguemnts, but they also unfortunately tend to selective about reality. The fact is we ALL live under common legal systems that unite us by nation, state, etc. Your personal views may be private, but we all have to work within the common framework of law. Isolated islands where you can make up your own rules are simply not in fashion (yet).
The whole gay marriage thing is the direct and natural result of a confluence of social changes in our society. Marriage itself is a social construct that has numerous benefits. The human race is a virtual stew of competeing natural vs. social trade-offs, moreso than any other animal species on the planet. We struggle with these all the time -- what is good for the individual, the family, the neighborhood, the city, the state, the nation, the planet, ???, etc. Marriage throughout history has mostly been an issue of what is good for the extended family (royalty added the dimension of nation) in terms of economics and survival (though survival even changed from species to "bloodline" which had many more ties to economics).
But so many things have changed within the past century or two. The world doesn't need more people, so survival of the species isn't a concern. Certainly in first-world countries marriages for economic reasons are considered a desperate move. The whole dynamic of marriage has shifted to one of personal choice via the love relationship between two people rather than arrangement by parents for societal and economic benefit.
But the real clincher that sealed the deal on gay marriage was the ascendency of women's rights. Or rather, the elevated status of woman from marriage-depedent to equal partner in the marriage contract. Prior to this, marriage was a very role-concious institution with the woman being property (at worst) or domestic slave (at best) with the husband holding all the real power. Equal partners in marriage changed all that. HETEROSEXUALS took it upon themselves to strip marriage of its previous role identities in parenting, finances, sex (yes even that), household duties, inheritance, divorce, etc., etc., etc. HETEROSEXUALS strove to make the practice of marriage gender-NEUTRAL despite it only being enjoyed by gender-DIFFERENT couples.
So it was only a matter of time before other couples who were already equal partners -- namely same-sex couples -- took a look at the new gender-NEUTRAL marriage practice of benefits and responsibilities and finally said, "Hey! We already do all that! We want/deserve that legal recognition as well!" Same-sex marriage prior to this development didn't make sense because it would have cast one member to be the legally powerless "woman" and the vast majority of gay men understandably would have never considered it an option. For lesbians the issues were similar but perhaps a bit more elastic.
But when the role framework lost is sex-dependecy, the doors were opened and the course was set. Gay marriage is now all but inevitable unless there is a mass change in societal views to undo the advance of women's rights and marriage partner equality between male and female. That's why you see gay marriage fragmented across the blue-red divide. It is resisted most strongly in the places where women still struggle against their "traditional" place in society, even though they might already have equality under the law. Conservative religion, of course, tends to reinforce those roles on women and thus is one of the greatest barriers to acceptance and change to this new view of marriage.