What makes you think that whatever you're hearing has anything to do with that one specific variable (the sample rate) and nothing to do with any of the millions of other other variables (microphone selection and placement, preamps, mixers, A/D converters, processing, your D/A converter, amplifier and speaker system) inherent to the process of recorded music?
Typically, when a single person experiences reality in a way that the majority of people don't, we consider them delusional, yes.
If you want to not be considered delusional, pass a double-blind ABX test where A is a direct feed from an analog signal source and B is the same source being run through an A/D and D/A conversion at 44.1/16bit.