In most states, you need consent to record a person. If you are doing this with hidden cameras without their consent, you are doing so illegally. If you are doing so with google glass, then when they tell you to leave they have expressed that they do not give their consent.
While I am not a lawyer, I am a photographer and have had to 'lawyer up' a few times. I've also studied the law in the area, and it seems much more familiar with it than you.
First, there is the very important concept of location. There are public places, publicly vsible places, and private places. A person in a public place, such as on public sidewalks or public parks or public buildings, does not generally have a right to that kind of privacy. Anyone who frequents /. can tell you about their favorite stores of photographs from public places (both for good and ill). Public places are public. If you are not in a private place but are still in a location that is publicly visible your privacy rights are also very limited; again we have stories both positive and negative, businesses dumping waste in a publicly visible alley or superstars making out in their back yard. Only if you are in a non-publicly-visible place can you reasonably expect privacy.
Second, there is the very important concept of commercial and noncommercial photograph use. You are right that you do usually need consent if the image will be used in commercial work, such as an advertisement or a movie. However, if the work is non-commercial in nature, such as a personal photograph, a wedding photograph, or if the image is to be used for news or social commentary or many other potential uses, no permission is needed.
So combine the two and you will see you absolutely do not need consent to record a person. You can legally record from any public place to any publicly viewable place. That gives us all the tabloid pics of the superstars naked in their back yards, from a location that is viewable from the public streets, without the permission of the superstars. That gives us all the beach photos of wardrobe malfunctions, again without the consent of the recorded. And it gives us the recordings of police misconduct, without the consent of the officers being recorded.
If you are in a private location then the rules are a little different. But as a general rule if the photographer is in a public place or has permission from the property owner on a private place, any photos they take of you can be used for just about any non-commercial purpose without your consent.
You may not like that. I can guarantee you movie stars and political figures also do not like it. But even with waves of public support (such as after princess Diana's death) the right to photograph people in public places without their consent still stands. Lots of money and effort has been put into rying to get rid of it, but it isn't going away.