It is a simple reductio ad absurdum argument: if you don't like the logical conclusion of the premise that anything in public is open to anyone regardless of how they use it, then you don't get to rely on that premise to defend Street View.
I see. So Google Street View implies that the anything in public is open to anyone regardless of how they use it? That is a laughable straw man.
I classed Google Street View with "candid photography taken for purposes of journalism, hobbies, and reference". In your mind, this is in principle the same as following people around and photographing their PIN codes?
There are lots of ways to use photos taken on public property that would put you at liability. You can take pictures of a someone and use the photo to imply endorsement of your product. You can take a picture and misstate the time and location where it was taken, creating a libelous accusation. You could take a photo where there was a reasonable expectation of privacy, for example when entering a PIN code, leading to financial fraud.
You can follow someone around in such a way that a judge considers it harassment. Although, for the record, few celebrities have managed to successfully prosecute such cases, so indeed it might be possible to follow someone around as you suggest.
Now, explain again how a photo taken from a van driving down the street is *anything like any of these proscribed scenarios*. Google isn't following people around, they're taking routine, non-targeted pictures from the street for cartographic purposes. It's not the same as stalking, any more than aerial cartographic photographs are.