I found this article on HackerNews a few days ago to be quite insightful in this respect: http://creativegood.com/blog/the-google-glass-feature-no-one-is-talking-about/
The main claim of the articles author is that in the past, there hasn't been any collective agency that pools information anything like Google. CCTV's, and whatnot have always been isolated from each other. The scary case (perhaps this is strawman) is that each Google Glass viewer may record and the collective samples with facial recognition can be used to track you around. Your voice can be recorded or transformed into text and stored forever to be harvested later.
This becomes a goldmine for advertisers and what not. This might include where you go, who you're with, what you look at etc... all without your approval and violates the fundamental issue of informational control in privacy.
Perhaps the technology today won't be capable of doing this but what about 3-4 years down the line? Google already works on image recognition (Google Glasses), Voice Recognition, and it knows your searches. What if the argument is that today the technology isn't capable of doing this but if these devices are allowed to saturate the market, what happens 5 years down the road?
It is true that today even cellphone carriers can totally track your location etc... but they seem to be somewhat regulated by governments. But what about Google? Which government or agency controls it? Information wants to be free.
These are purely my concerns about this sort of technology.
Thank you for your time.