Personally I can hear the difference of 160kbps Vorbis and 320Kbps Vorbis, and any mp3 compared to high bitrate vorbis or lossless. That said, Spotify with 320kbps Vorbis has finally reached the point where I consider it worth paying for it as a streaming service. In most cases I can't differentiate between 320kbps Vorbis and Apple Lossless, which is my default ripping format.
The few times I really want to listen to high-quality music, I put on my Sennheiser headset or turn on my toslink-connected hifi system, and play SACDs directly or lossless rips on my Mac.