Let's be fair here: $400 jacket and $15,000 machine with engines that are a 50 year old design. Hopefully a $100 helmet too.
I think it would take only a small number of things to displace Harley in the motorcycle market. 1.) Their competitors do not have the paint quality and style that they do. 2.) Many people buy Harley's because they are American made. All of their serious competitors are foreign made. 3.) Let's stop fighting this crotch rocket vs. cruiser thing. Give me an easy chair with the power of a crotch rocket. A ridiculous amount of chrome wouldn't hurt either.
With that said, there are several groups of Harley riders. Some of them just love motorcycles and ride a million miles on theirs. Some are rich doctors and lawyers who want to look cool, and generally sit around at gas stations trying to act natural and failing. The final group of people is the dumb people, who I think you identified. These are the people riding their bikes to bars and trying to act like badasses with tatoos and skulls and crap.
Don't base perception of a manufacturer's quality based upon one of the stereotypes of their riders.
I ride an '81 Yamaha Maxim XJ550, which is an inline 4 with 4 carbs feeding dual exhaust. It is very simple with no frills, and I have personally rebuilt pretty much everything on the bike. Many people do not understand the appeal of older Japanese bikes, their simplicity, and knowing every detail inside and out is pretty cool. It is one of the few blends of early ricer / cafe racer with some cruiser style. Id love to buy a Harley, I dig their styling, but I just cant seem to bring myself to spend the amount of money they cost. When I pull up somewhere, I know I'll (almost) never see anyone else on my model of bike, and I constantly get asked questions, get pictures taken, get smiles, etc, and that beats trying to look cool on a shiny Harley.