Easy peasy.....we're talking a VERY small minority of people in the world that have these issues.....why should the majority get all bent out of shape for this insignificant number of people outside the norm?
You underestimate the number of people with such "issues". And why care about sex-segregated bathrooms at all? They're not even a fixture of classical Western society - they were mostly introduced post-Civil Rights era to keep the "good, noble" white women from being even potentially contaminated by contact with "black savages". Why not just have unisex bathrooms?
I really don't care what an adult does, or two consenting adults do....in private, but don't force the masses to cow-tow to them...there should not be special treatment for acting outside the norms. You shouldn't be persecuted for it, but you also shouldn't expect special and protected treatment in every day life either.
People are getting upset because this is a blatant prelude to an assault on people's freedom to do what they want in private. Pence in particular has been completely clear that the end-goal is to force everyone to behave in accordance with the gender norms that match their sex, and those noncompliant will be deemed sexual assaulters or pedophiles, and treated as criminals.
Nobody is asking for special protected treatment - they're asking for some kind of guarantee they won't be abducted and shipped away to a torture camp. Which yes, happens - overly-religious parents that see their children acting "faggy", find out about "outdoor programs" using "conversion therapy" to treat them. But rather than a fun summer camp, it consists more of forced marches, sleep deprivation and just general "make them suffer until they'll do anything to make it stop"... this has more than once resulted in deaths.
Our current Vice President describes himself as a firm advocate of these programs. If that doesn't sound faintly terrifying, I have no idea what more I can say.
I get it. You see yourself as an open-minded person, with strong libertarian ideals. You see a group that you would never really consider persecuting, out there making demands. And the only big mistake you're making is assuming that, because you would never consider persecuting someone for something like that, that nobody would - and so their demand for explicit protection comes across as a demand for special attention, which naturally rankles your minimal-government ethos.