It's not about math, but about honesty. First of all, we "math junkies" don't say there is proof that there is no audible difference. That is useless null hypothesis. What we're saying is that the burden of proof is upon those who claim that they can hear a difference. And the mere statement "I can hear it, it's obvious" is also not in any way sufficient, because the currently used method to assess these kind of perceptual questions are double blind tests. If you're unwilling to conduct an honest and scientific tests you're not necessarily a liar, but at least dishonest.
On the other hand it is well known that the placebo effect exists in medicine, so it is not unexpected that it exists in audio too. The amount of people affetced by placebo in audio seems to increase with age and audio equipment cost. It also seems to be correlated with the aforementioned "dishonest" people.
What people like you hear is that the actual released records sound diferent on CD and Vinyl, and that is because of different mastering practices, especially the fact that CDs can be made much louder because the playback system is not inherently flawed w.r.t high peak amplitudes like a Vinyl turntable.