As said above, SSA doesn't have any sort of biometric verification of "who you are".
And, as said above, your SSN shouldn't be used as an identifier. If we need a common citizen ID number, fine, but it shouldn't be anything but identifying (i.e., effectively public knowledge).
It's the gorram 21st century. We've had public-key encryption figured out for over 30 blessed years now. Most people in the first world are carrying around several crypto smartcard devices already (EMV compatible credit cards and other smartcard tech).
Much of the world now has ID cards with cryptographic chips in them. When you open a line of credit, you prove, through RSA/elliptic-curve signatures that you are YOU via your ID chip. If you lose your ID, it gets put on the centralized revoke list, the issuing agency goes through whatever in-person process to verify you are you, and gives you a new ID. This can extend to online purchasing, online voting, etc, etc.
But, we're so freaked out about government black helicopters that we just accept the whole fraud thing as inevitable.