Most of the comments have been made by people who have never worked within the automobile industry and who hate the current process of buying a car. That's fine, but they're missing some important parts of the picture. The first unmentioned part is that the majority of deals involve a trade. As much as you think people hate buying cars, you will quickly discover that they hate selling their cars even more. Most people are entirely too lazy to prep their cars for sale, and are usually unwilling to invest in the repairs that will facilitate the sale of their vehicles. The second issue is that a huge percentage of the buying public has marginal to poor credit. The auto dealership essentially preps and polishes the credit application, and then finds a lender willing to buy marginal paper.
These two criteria eliminate about 85% of the buying public from purchasing directly from the factory. Really.