then use the SS#, birthdate, mother's maiden name and address info the bank was storing to compromise your
The federal government already lost control of that information, and more, for me and tens of thousands of others when a laptop (that should have never had that information on it) was stolen from a car in DC. I don't expect them to do a whole lot better with authentication keys.
And what's included in that annoyingly thorough identity test at the post office? SSN, birthdate, mothers maiden name, last 3 addresses, etc. All the information that gets stolen already anyway-- so the TFA is a convenience, but it's subject to the same sort of attacks that ID is already subject to. You can go on a two month european vacation and I can go to the post office, pretend to be you with all the information I stole, get your key revoked (leaving you kind of SOL when it comes to paying for your hotel and food, and making your re-entry into the country really fun), and get a new token and clean out your accounts. You can mitigate it to some extent with biometrics like fingerprints on the token, but I could limit my attacks to people like potters and bricklayers who wear theirs off and go in with the same blank fingers they have. And if I'm a mugger taking your token and know you need prints for authentication to revoke it, I just have to mangle your fingers. That gives me more time to clean you out while you try to prove you're really you.
These are just a few random attacks that took a few seconds to think of- someone clever with a lot of time can do much better. Things like this always seem to work great when you plan them out, but there are always exceptional cases that you have to deal with that nobody anticipated. An example of another hole in many of the ID systems is the US Passport-- when it comes down to it, all you need is another US citizen to vouch for you to get one. I know someone who grew up in NYC, never had a drivers license, no birth certificate, parents dead, had very little paper trail despite being a visible small business owner for decades, and he ended up having to get someone who'd known him for most of those decades to vouch for him when he finally needed a passport in his 40's. At the end of the day, any system comes down to the weakest link, and it will likely end up being some gaping hole like that.
And suppose I build a quantum computer and start factoring big numbers easily? Now we have your national emergency because we put all our eggs in one basket and created an awesome single point failure for authentication.