Memory-impaired users of EyeTap/Glass-like devices - using them as a memory-prosthetic (a not unreasonable use of this kind of technology to aid their day to day lives) - might find themselves subject to abuse from able-bodied (in the metal sense) non-users, mistaking the disabled for geeks.
That's a real concern, yeah. With a technology like this, abuses can have a huge negative impact on people who otherwise have need of it to go about their daily lives.
I think if cochlear implants were mass-market for people who weren't born deaf, we'd probably already be facing some of the same social problems there - if it was widespread and common that people just carried around recording devices in their skulls, and anything you said anywhere could be recorded and replayed without your knowledge or permission, and it was common, that'd probably make people pretty paranoid. The introduction of smartphones and other commodity recording devices has already caused a problem here - with stories of police officers/military confiscating or damaging phones that are being used to record their acts, etc.
To a degree this isn't something unique to Glass, it's just the first time we've had a theoretically mass-market device come out that aims to make recording truly ubiquitous and non-obvious.
Just a thinking-out-loud exercise. Move along.