I voted CPU, even though at the time it was more of a complete core upgrade, the only components i kept where HDD and RAM, although the mobo/vga upgrade were sort of neccesitated by moving sockets.
I upgraded from a s754 AMD 3400+ to an s939 3800+ x2, and in the process went from a single geforce 6800 to two 6800XTs in SLI (a single XT is inferior to an normal 6800, although they OCed like MAD, 325 MHz stock, got them to 475 before they got buggy). The dual core cpu was the most noticable though.
I've done the SSD thing too, which was pretty nice, and everything after using that machine, every other machine seemed old and sluggish, but by the time i got the SSD, i was pretty much out of the pc gaming scene and didnt do much intensive stuff, so performance isnt that important to me these days, my "main rig" is an IBM thinkpad t43 running linux.