Your view is common. I held it myself for a long time. I have since changed my mind. The real problem is that programming is taught very badly pretty much everywhere.
I spent 20 years as a programmer and then taught English as a foreign language for 5 years. I am now back programming, but I have started to apply what I learned as a teacher to programming. Just like programming, foreign languages are usually taught badly: the idea is to teach grammar and drill vocabulary. The students jam words together using the grammar and eventually learn how to speak. The problem is that most people are not very good at this technique, leading everyone to the conclusion that you need to have some sort of talent for learning languages in order to become fluent at a second language. I have found (like a lot of other teachers) that this is a poor way to teach. It is better to use "comprehensible input" to build fluency. The idea is that if you furnish students with a large amount of input that they can understand, they will acquire language. So instead of breaking up language by category and teaching rules that students apply, you give them language in context that they can understand and have a high amount of repetition. My experience (at high school level) is that the vast majority of my students could reach a surprisingly high level of fluency.
When I went back to programming a couple of years ago, I started mentoring some of the guys on the team who were under performing. Their main problem was that they would not understand what was going on in the code and would resort to "programming by google" (i.e. just cut and paste whatever crap from stack overflow that seemed relevant into the code). I talked with these team members and asked them, "When you read the code, how much of it do you understand perfectly". The answer was "70-80%". I explained the input hypothesis and told them that in order to reach fluency with the code, they would have to understand 95% of everything they read. Or, since it is hard to tell if you understand 95%, they should aim for 100% comprehension. After a few times where they fell back and claimed that it was "impossible" to understand all of the code (and I demonstrated that it was not impossible), they started taking more time to comprehend what was in the code base. Within a few months, their level improved dramatically.
I'm not saying that anyone can be a great programmer. But, most people who are diligent enough to graduate from college can become a half decent programmer. Our team is rather special in that we provide a lot of time and support to allow people to improve. I think the reason most teams can only get good productivity with talented programmers is that they essentially abandon them once they join the team. If you don't have the chops, then you don't deserve to be here. I've actually said that before, to my shame. I now think that most people who are interested in programming can be a programmer if they are taught properly.