I appreciate the in-depth response, as I'm still coming to grips with this concern some people have. And let me say clearly, I also don't want someone with a camera of any sort videoing me, specifically, in a private place (wouldn't be that comfortable with them following me around in a public place either).
What I still don't understand is the automatic assumption that any Glass user you see is not only always videoing everything they happen to glance at, but that they would bother to focus on you.
The former is highly unlikely, due not only to the battery expiring in a couple of hours, but because very few people care to record all the random crap they glance at, every single place they go. What would you do with all that boring video? Isn't it far more likely that people would only want to record significant moments - special events, friends, maybe a quick glance around a restaurant for context, then focusing on something they actually know and find interesting?
The other point is the apparent belief that a Glass user in particular is likely to point their lens and stare for extended times at all the complete strangers around them. That would be rude as you say, regardless of what sort of camera you use (or even without one) - so why do you think "Glassholes" would invariably lack such ordinary politeness? Because of their choice of technology? Surely if someone actually wanted to video people around them, they'd be far more likely to choose something a heck of a lot less visible than Glass?
I get that there's a potential for a Glasshole to rudely invade people's privacy in the way you are concerned about, just like anyone with a camera or smartphone or keychain camcorder, or even a bored security guard watching a surveillance camera. I don't get why people think it's so much more likely for someone wearing Glass to do this than anyone else. Particularly as it'd be so much more obvious with this weird Glass thing on your face. Glass does have a few other uses besides recording things.