OK, I started out as a Physics/ Astronomy major, and even got through three semesters of intro (all the way to elementary QM) and three semesters of math (thru diffy Q) as an undergrad.
My problem, and why I became an English major: I was in the 3rd semester phys class and the math breaks out, and I am fine until they started using bra-ket notation. (If you don't know what I mean it's stuff like and used a lot in QM) I had no idea what it was. I hadn't seen it in a math class yet. the math and physics departments evidently never spoke to one another so there wasn't ay "matching" of the curricula, so if you got to the right notation in math you were ok but god help you if it was unfamiliar. I was too embarrassed to ask about it, probably.
I didn't give up a sci major for *just* that reason. Originally I wanted to do both a liberal arts and a science degree. Yeah, I bit off more than I could chew. And I got interested in a lot of other things, like language learning (which I was more naturally talented at no question). But I did feel that I was falling behind in physics and was getting a bit frustrated I think. Even with pretty OK grades.
But all that said, math builds up from one step to the other. I think it's like bicycle riding -- a lot of things stay once burned in.
Anyhow, I did OK in my physics classes, and even the math. I was a B student and probably could have stuck it out.
Interestingly, 20 years down the line I am back in math again. And I did Vector Calc and loved the class. My prof gave a take-home exam and I loved the fact that me and other students could argue over solutions. In one interesting instance I had the answer to a problem and I had to convince 2 other people I was right. I really learned that one! I think, even though I got a B-, (I glitched on the final, blanking on L'Hopital's rule for more than one variable, for christ's sake, I was so anxious) but my teacher was so good I felt like I learned a lot. And I still remembered, with a little prodding, the calc I took 20 years ago. Funny how it stays with you.
Then this summer I was in Linear Algebra. And it was the most frustrating math class ever, for me. Lots of memorization of proofs. Abstractions way more than Vector Calc. I found it VERY hard. Much more so than vector calc even. A totally different skill set. I find that kind of abstract math more challenging for some reason. (Though I finally learned what the hell bra-ket notation meant. If someone had told me that in 1989... )
I think it's a combination of difficulty, preparedness, and the hit-or-miss setup of curricula at various colleges. And you have to have - as others here have said -- instructors who can help students with the things they struggle with. That's an art and there are no hard and fast answers or easy methods.
I'll be taking partial diffs at some point soon I think. Will have to break out my old calc book and study ahead tho. (Finishing that physics BA. I really kind of dug intermediate E&M this time around).