Exhibit A: Someone who has never played golf.
Golf is one of the hardest sports to play, requiring a precision unparalleled in any other sport. It may not be as physically demanding as say football, soccer, hockey, basketball, racing, etc. In terms if difficulty, there aren't many that are equal. I challenge you to just go to a driving range, and hit 2 balls out of 1000 strait.
Before you belittle something, you should probably try it first. I used to be like you, until I started playing golf, never in my life have I been as frustrated.
At any rate, I don't think it is the difficulty of the game that is turning away players, it is the cost of greens fees for a decent course, the other assholes on the course, either slowing you down, or rushing you, basically not allowing you to have any fun. People take it too damn serious.