You describe an extremely antiseptic environment, where nobody ever talks about anything not directly related to the code, and they only talk about it in a bland tech-manual style. That's not reality.
Reality is that people in the FOSS community enjoy FOSS, but they also enjoy community. Community often involves informal discussions, jokes, etc. and as you mention, interaction outside official channels. And that's all fantastic, and a big part of what keeps people engaged and excited about a project. Even something as simple as being informal and silly in the way you comment your code or explain your algorithm on the dev list can make the project more fun and satisfying, less like corporate work.
The problem is that the very community so many men enjoy is currently alienating for the vast majority of women. Because currently, some of those jokes, discussions and silliness are misogynistic. So while women technically can participate in the coding, the "reward" of getting to be part of a fun community and getting respect from peers is worthless or undesirable to them.