I've had similar experiences with Ubuntu. Its just too damn bloated.
But Linux is just the kernel and it is indeed fast. What distros pile on top of the kernel is what can slow the whole OS down if they are not careful.
Crafting a system from kernel up (ala LFS) is too extreme. There are distros that will install a base kernel and let you add whatever packages you want from repositories, like Arch. These system do not require all that much Linux knowledge and most will blow XP out of the water speed-wise.