I switched to Xfce because Ubuntu is switching to Unity and ditching Gnome 2.x in the next release.
I must say that Xfce is really nice, and isn't much of a step down from Gnome 2.x
In fact, I managed to style Xfce to look nearly identical to Ubuntu's Gnome 2.x in 11.04. The one missing feature in Xfce 4.8 is the ability to manually sort taskbar buttons, but they have added that back in to 4.8.5 (I'll wait for Ubuntu 11.10 for this feature, as I don't want to hassle with getting 4.8.5 on 11.04 and end up in dependency hell)
Xfce could be a little more customizable, but it is not bad and I'd rather they keep doing what they are doing and not try to please everybody and end up with a lot of cruft.
KDE is getting their groove back, but I ultimately think Xfce will be the big winner of Gnome refugees.