It's also a case of selectively reading stats without knowing the big picture.
Chrome obviously needs to decompress your PNGs and JPGs to be able to draw them to screen. While you're viewing a page, it'll keep the decompressed images in memory. If the page has been in the background for a while, it'll free the decompressed images and just keep the compressed versions in cache. When you go back to a tab you haven't used in a while, it'll decompress everything again. This reduces the memory Chrome uses a lot, at the expense of a slower load for a tab you haven't look at in a while.
If you load a few large complex sites and then look at memory usage, yeah, it'll be really high. If you've got a lot of tabs open that you look at occasionally, memory usage won't be nearly as bad. And like you said, the remaining memory used by those backgrounds tabs will get swapped out to disk if needed.
I know Firefox does this too, and I think Safari does as well.