I think I can relate to that. I'm quite introverted so I need a lot of time on my own to "hear my own thoughts". I also don't like listening to music very much. OTOH, I'm kind of a musician, and I generally prefer instrumental pieces. I can feel similarities between social yapping and music with lots of lyrics. Making music is great, listening is kind of meh -- I guess it's like the difference between doing some sports yourself and watching it. (I've never understood the fascination of self-proclaimed straight men in watching other men work out.)
There are theories about introversion being a symptom of generally heightened sensitivity - we get more easily overwhelmed by sensory input, so we try to cope by locking ourselves within. As much as I like music, I easily get my daily dose of auditory input from all kinds of background noise.