Hypothetical situation: Let's say you travel to another country where you don't speak the language.
"I don't speak the language" isn't a Get Out Of Jail Free card. People may make exceptions for you, but those exceptions will only get you so far. You have to make an extraordinary effort to learn the language, or accept that you're going to have to be isolated with others who speak yours.
Social interaction is a lot like that. Some people may make exceptions for you based on your disability, but acting out and then falling back on your disability as an excuse is just going to rub most people the wrong way. I struggle too. I've got ADHD and social anxiety, which makes me seem aloof and distant.
Oh, and that woman who called you "sexist" for opening the door for her? She's got a problem too. I've run into people like that, and they're impenetrable walls.