I'd be more worry about handing your card to waitress at the restaurant than worrying about your contactless card being read remotely.
obtaining the data contactlessly is not enough to create a duplicate of your credit card(assuming proper card implementation), and certainly not enough to create a "card not present" transaction such as internet, mail, or phone purchase. (only exception is probably using pre-play attack, and this requires some elaborated setup)
A properly implemented contactless card don't even have your name in the contactless interface.
seriously, your credit card company is worrying more on the fraudulant transaction then you, and so there are fairly good measurements deployed to ensure contactless duping can't be done.