If you don't care for what other people think or their interests, why would they care about your ideas and interest?
He's not saying he cares one way or another, it was a personal observation, that somehow he has an easy time finding men who share his more intellectual pursuits, and that women who are intellectual are very rare for him to cross paths with. This is likely due to several factors, including his particular career field and geographic locale.
As an engineer, I generally see the exact same thing. However, I don't think it's because women are generally insipid morons (as I said before, it's easy to find men who are big fans of Duck Dynasty), it's because of my career and where I'm located. If I moved to NYC and worked at the NYPL (public library), I'd probably run across tons of women with zero interest in The Kardashians who would love to instead talk about all kinds of intellectual subjects (probably literature). As an engineer, I run across very few women at all, and most of them are administration or HR people, not known to be groups full of intellectuals (and HR people are, in general, just a bunch of morons, no matter their sex).