I don't give a shit. Given that I have 1 GiB of RAM, 20 MB isn't going to impact my system. It's the 500 fucking MiB that Firefox keeps taking up because the Mozilla devs can't be bothered to optimize their code and cut their bloat, or make their config options actually do something. I need Firefox because of crappy web developers who can't understand that there's more than just the mainstream Firefox, IE, and Safari (and now coming up, Chrome).
If I compiled my Xorg to be tiny, then I might have less usage. But compiling with -O3 and -ggdb (among other options), I have a larger footprint. But, again, I don't care because of the system I have. If I had only 64 MiB of RAM, then I would be more wary about my memory.