The main reasons for failure, IMO, are lack of standards and lack of overall management. Above the kernel level is mostly chaos. The following was written over 10 years ago.

Desktop Linux

Desktop Linux (meaning GNU/Linux and the entire ecosystem) has been suitable for non-geeks since about 2004. It works well, looks good, has superior security and freedom from malware, has thousands of applications covering almost every need, and costs nothing. In spite of these advantages it has failed to win more than a 2% share of end-user desktops. WHY?

Note that Linux is the biggest player in the server space. Google, Amazon and Facebook have millions of Linux servers. Most of the world's web servers, stock exchanges and scientific supercomputers use Linux.

Here are the main reasons for low adoption by end users, in my opinion:

Fear or Hatred of Change
People hate change. Linux is easier to use than Windows, but requires learning new skills. For most folks, this is a huge barrier.

Balkanization
    + many competing "flavors" of Linux
    + competing window managers with different application interfaces
    + competing application package formats and tools
    + package naming and content variance, application dependency chaos
    + library API changes causing existing applications to fail (require update)

Application developers must be willing to support many slightly incompatible versions of Linux. "Build once, run anywhere" is not possible. Attempts at standardization have gone nowhere. Developers of popular applications refuse to deal with Linux. In contrast, Microsoft provides a stable standard that both developers and users can depend on. Backward compatibility is carefully planned, and programs from a decade ago still run on current Windows.

Credibility
No major company with clear staying power is producing and supporting a desktop Linux system. Google makes Chromebooks using a customized Linux kernel, and some Linux applications can be used on some models, if you have the necessary knowledge to install them. Google avoids using the name "Linux" because this is considered bad for marketing.

Microsoft
PC vendors are fearful of making Microsoft angry. They used to have to sign exclusive contracts, but this was ruled illegal long ago. Now they get license discounts, which means that nothing has really changed and the monopoly power continues. This is the main reason you will never find a Linux PC in a retail store. There are a few online vendors of Linux PCs, including Dell.

Conversion Costs
Organizations switching to Linux must convert custom applications and retrain technical staff and users, a potentially huge cost. A few companies and government organizations have actually done this.

Missing Applications
The main applications that most people need are available and of high quality: web browser, mail clients, document applications compatible with Microsoft Office, photo editing, and some games. World class modern games are mostly missing. Photoshop is missing. One reason for missing applications is the difficulty of development and maintenance given the "balkanization" issues mentioned above.

Free Software Culture
Free software advocates are in denial and are not facing the problems. One often reads "having many choices is good" or "a Darwinian process will select the best alternatives". This is also what they said in the 1990s. The freedom of Linux is also its downfall. Effective standards are lacking. There is no management and no road map. Every player runs his own show. Linux geeks don't worry about this - they are having fun.

No Way Forward?
Legacy Linux (GNU/Linux) will become even more irrelevant and will be abandoned by more developers. It is a shame that today's popular desktop platforms (Microsoft and Apple) have high cost, malware galore, and massive invasion of privacy. Linux could have made a difference, but lack of management and standards has ruined the opportunity. The solution would be to consolidate Linux development resources under a unified management. Sadly, there seems to be no chance for this to happen.

Gratuitous API changes Old APIs should keep working as long as the equivalent functionality is available in the new API. An adapter function should be supplied.

zombie bugs: they come back after being killed, because coder B overwrites fix from coder A