No, the actual damage to the rights-holder is far less than the MSRP, because that's adding in the retailer's mark-up, the wholesaler's mark-up, and the rights-holder's marginal production costs (if the band holds the rights for music, that would probably be $0), not just the marginal profit and the proportion of the fixed costs which would have been paid by that copy. It might even be possible to argue that the actual damages were less, because you would not have bought it if you could not pirate it (I know that's true in some cases, but it would be hard (and probably pointless) to argue), which brings the actual damages down to quite a small amount per copy (for an MP3 rip of a song, the damages would be considerably less than $1 per song, since the iTunes price is obviously profitable). Then there is the fact that unless you had a very large collection, the damages would be less than the limit for the small claims court, so they would not be able to bury you in legal fees either.
This is the situation in Australia, (although Sen. Conroy has said he wants to change the law) and there are still plenty of record shops (my local mall has I think two record shops, and the nearest big one has 3 record shops and 2 DVD retailers, as well as 2 big retailers), so piracy can't be doing that much harm to the recording industry.