This is what I did, not on MP3 but DD vs TrueHD.
I'm a heavy movie watcher. When I got a receiver capable of lossless sound I just wanted to know if I could hear the difference. If I can't it's really no use for me. I took a movie with a great soundtrack, namely Transformers (shity movie, but one of the best soundtracks). I ripped the blueray and made a little program to play small parts of the movie 2 times, 1 using TrueHD and one using Compressed Dolby Digital (640kbps), randomly switching which soundtrack it played first. I calibrated the volumes to match until I couldn't hear any difference on most parts of the movie (dialogs etc, where the DD bitrate is clearly enough for my ears/equipment/room at least).
Then I sat down with pen and paper in front of the movie and guessed if I was hearing Lossless or compressed sound. My conclusion and my numbers from the guesswork is that Lossless vs DD is very much hearable when the action gets going. The sound is more spread out. sorry I don't know the terms for this (I'm not an audiophile) but, lower low volumes, higher high volumes, better separation beetween different simultaneously playing sound, and possibly better channel separation. My conclusion also is that the difference isn't earth-chattering on my living-room-setup and I can still appreciate DD movies . But the difference is there and it's nice.