If your laptop contains the credit card and health information for 1 million users, yes. It should be your biggest concern. If your laptop contains sensitive corporate information trusted to you, it should be your biggest concern. If your laptop contains information you wouldn't want public, it should be your biggest concern. If your laptop contains anything about anyone THEY wouldn't want public, it should be your biggest concern. If it contains your pr0n collection, then probably not.
The stuff in your wallet is easily cancellable and easily replaced (other than the actual cash, and any information you might have written on a scrap of paper), the stuff on your laptop, once out there in the wild.. probably not.
Regarding your USB dongle, are you certain you will *ALWAYS* remember to remove the dongle and pack it separately? What happens if you are forcibly required to give up your usb key (say, when crossing a border, and the oppressive government believes you may be a dissident or spy).
Certainly, you can still be forced to give up your password if you are willing to. But it can't be taken from you, unlike a dongle. And the master keys can't taken from a TPM if the system is shut down and correctly configured. (there are some attacks under certain conditions that can lead to exposure, but that's an implementation issue which you can control through careful selection of hardware).