From my understanding RFID usually don't carry that much data except for a unique identifier. Ok so I se a Hex value. However you may not know what type of RFID it is is for. Eg. Is it for your credit card or is it just that book you got out of the campus book store. Perhaps it is for your medical history that you got implanted in you skin. Maybe it is your Dogs virtual ID Tag implanted.
Say if I dropped a Passord of a vital system in the Middle of New York City and you pick it up. And that password is for only one system what is the chance you will find the system and get in.
That said we should be sure that RFID for say on Credit Cards and on other major checking systems should have additional checks to it. However for say Inventory and automatic checkouts it should be ok.
Even just a unique identifier is enough to cause a *huge* privacy concern. Not only that, but most tags do give you additional data, including their manufacturer, what kind of chip they are, and what commands if any they respond to (Some give all of this just in their ATR (Answer to Reset, which nearly all tags respond to). The biggest problem with the current implementations of RFID is that extracting data is a silent process. There's no beep, no light, no counter, nothing to indicate to the end user that their RFID tag(s) have just been read.
While US passports are actually pretty secure and do not give out any unique information without the proper MRZ data from the inside page, US passport cards are not secure at all. They're just standard UHF EPC Gen 2 tags with unique identifiers. Similarly, paypass/wave/blink/whatever RFID credit cards aren't secure at all; anyone with the proper reader can dump your card holder name and card number (though Expiration date and CVV code are not present in the RFID data iirc).
It would be trivial (and until laws are setup otherwise, legal in most places) to build a network of High gain RFID readers around a city. Not only would this let you "track" people around the city, but it would also let you build up a profile on people. You could, for example, keep a database with every tag read at a specific instance and correlate that to different data gathering points you have set up. You could then have a person object with various tag UUIDs associated with it (and if they have a credit card on them, even a name associated with it!).. Couple this with a camera that takes a picture of the people who's tags you're reading, and you have a picture too! Boom, picture, name, credit card number and unique profile of everyone that walks by your antennas, along with the time of day they walked by and their exact location. Try and tell me that's not valuable data?
I highly doubt there *aren't* companies out there doing this.. In fact, so long as it stays legal, I'm going to start up a company that does exactly this! Think about the possibilities for targeted advertising! FWIW, because the "public" at large remains mostly ignorant to all this, and companies/governments get what they want out of it nothing is going to change... ...In the case of the passport card, its even more worrisome.. Say someone sets up a checkpoint outside a border crossing with a long range UHF antenna and a camera... Boom! They now have everything they need to make a legitimate fake passport card! (This scenario is outlined by Chris Paget in his talk at Shmoocon V (http://video.google.com/videoplay?docid=-282861825889939203 ), as well as by several researchers for RSA (http://www.rsa.com/rsalabs/node.asp?id=3557).