Piracy is rampant on the DS too, and there's tons of money being made there.
Agreed. Buy a flash cart, a microSD card and you're off to the piracy races for the DS. Much less complicated than PSP modding even though that is relatively simple.
It boils down to a few things: price, game selection and allure of the hardware.
It automatically failed on price. Remember, when its price dropped the PSP received a decent boost in sales. Unfortunately, due to their sparse selection of quality games, I don't think it held much interest.
The launch of the DS was stronger. The DS Lite reinvigorated and popularized the console. Also, it provided an interesting way to play games. Now games could be made that actually interested mom and pop (read: Brain Age). The PSP stuck with hardcore technological advantage and fell on its face. Not to mention the 'nub' joystick is a pain to get used to or the disparity in battery life.
Sony has no one to blame but themselves, but honestly, they've done relatively well in an arena that Nintendo absolutely slaughters.