They require an active cell link and a live phone, so are bad news if you are trying to log in in the bowels of some structure, with a phone that has a dead battery, or while travelling outside your non-ridiculously-priced service area. It also tends not to be a problem in practice; but SMS is 'best-effort', so if the system is being flaky then that's just too bad. Essentially, it isn't a 'second factor' at all; but a secondary channel that is assumed not to be compromised.
Then there is the matter of the site needing your phone number. For some applications, that doesn't matter: your bank already knows way more than that about you, say. For others, I'm not so enthusiastic about providing a relatively persistent, and spammable, identifier(also fairly robustly tied to me by payment data, unless I get a burner specifically for dealing with auth issues) to any lousy little website that wants it.
Finally, I'm not terribly confident about the medium-term security of SMS if it becomes a common '2 factor' authentication method. Mobile OSes tend to be a bit more locked down than desktops; but hardly infallible, and the security of SMS gateway providers(who sites using SMS auth presumably employ to interface with the phone network) is an unknown and possibly not comforting factor.
RSA fobs are ultimately an inferior option because they cannot be safely shared across multiple systems, and carrying a fistful of the things is ridiculous(plus, the pricing is usurious); but smartcard/NFC cryptographic authentication has none of these weaknesses. The hardware is cheap, it doesn't require a secondary channel to be available, certificates are relatively tiny so you can carry an enormous number of them without issue; and you can implement certificate auth with varying levels of connection with user 'identity'. On the relatively anonymous side, the user can just generate a keypair and send the public key when they create an account. Trivially handled on the client end, no interaction with outside entities. At the other extreme, hierarchical PKI systems make it possible to robustly verify the user's affiliation with a given organization if the situation requires it. The trouble, of course, is the lack of card readers/NFC pads on a lot of contemporary computers and mobile devices. A great pity.