I am working as both researcher and developer in this field for almost two decades.
There certainly isn't a "lack of a killer app" there. However, there is one big difference - I am talking about professional market. Simulators & marketing are one thing (even though those rarely focus on HMDs but rather on projected displays - HMD is cumbersome).
Then you have training applications - machine operators, surgery training, maintenance training, safety procedures, dangerous materials handling, you name it. Of course, military training too, even though that is a complete different market.
Another category I have been involved with are various medical therapy applications - psychology and psychiatry - e.g. various phobias, even additiction therapies, PTSD is successfully being treated using VR, pain distraction applications (e.g. for severely burned people, dentists or some cancer sufferers).
And those are just the domains I have been somehow involved with or seen around me. Retail/consumer market is a different story and there I see VR mostly as niche within the larger gaming/entertainment market. Things like 360 videos and watching TV using HMDs will likely fail as that is pretty much pointless and the novelty of it fades very quickly.
Concerning AR - I am not that sure. For one, AR is very overrated. The AR applications are perhaps numerous in movies and Hollywood (everyone wants a Terminator-like HUD, right?!), but not in reality. Stupid stuff like superheroes jumping out of cereal boxes were interesting perhaps 5 years ago. There are also numerous problems that would have to be overcome first and not all of them are technical ones.
* There is AR and "AR". Google Glass was not AR but a personal HUD - if there is no registration between the image overlay and the real world, *it is not AR*. Glass was incapable of that.
* Contrast - optical see-through displays have inherently poor contrast and daylight visibility, digital see-through (using cameras) tend to suffer from lag, motion blur and poor dynamic range (big issue outdoors in sunlight, for ex.)
* Fragility - most AR displays are very fragile pieces of glass and electronics, not robust to day to day abuse. Can be solved, but at the expense of aesthetics and price.
* There is *no* robust tracking soluton for AR that would work both indoors and outdoors, have sufficient accuracy for registering overlays over objects beyond simplistic labels (think satnav instructions) and would not require enormous computational power. Projects like Tango are getting close, but still no cigar. This is actually *THE* problem we are fighting almost every week at work when clients ask for an AR application - nobody wants markers, but pretty much nothing else works with sufficient accuracy and low overhead.
* Battery life - AR needs to be mobile to be really useful which means batteries. Something iike Tango running full tilt tracking and displaying of the 3D scene would likely run out of battery within an hour or two. Or shut off due to overheating because of the heavy computation going on (see the GearVR problems with that - and GearVR is not doing any tracking at all).
And now the non-technical issues which could be the largest problem, in fact:
* Most people don't want to look like Borgs from StarTrek wearing headsets. Not an issue in the pro market, but enormous problem for consumer market. Google has learned that firsthand with their Glass.
* We are far from headsets being socially acceptable. Both the cameras and the lack of eye contact/apperance of being distracted are an issue. Perhaps this will change as these devices become pervasive (talking on a phone in public used to be a social no-no too), but until then it will be a major hurdle for adoption.