That's a specious sophistry. Men and women both can marry someone of the opposite gender, but not someone of the same gender. The only way you can get a right men have that women don't out of that is to *define* the right itself in terms of a specific gender.
There was a similar argument for laws against interracial marriage as well - since no one was allowed to marry outside their race, it was argued, it was not discriminating against any one race. Everyone still had an equal right to marry someone, and all were equally limited to their own race. However, the California Supreme Court rejected this argument in the 1948 Perez v. Sharp
decision, which is what the court used as precedent in the recent ruling legalizing same-sex marriage. It didn't matter that the law was not singling out any one race to discriminate against; simply having race as a limiting factor was considered an unreasonable restriction on the right to marry.
case described the "right to marry" as follows:
the essence of the right to marry is freedom to join in marriage with the person of one's choice
Just as a person cannot help that the one person they choose to marry is of a different race, some cannot help the fact that the one person they choose to marry is of the same gender. Limiting one's available choice for marriage based on gender should be considered, according to CA's equal protection statutes, just as discriminatory as limiting it based on race.