I started in college as a comp sci major. I already knew how to program in BASIC, C, and C++ with reasonable proficiency and was excited about the major. However, I had a string of lousy math teachers until high school and struggled with algebra. Oddly, I was always fine with trigonometry and statistics, and I never had issues with the logic part of programming (I'm an attorney now). I was drastically unprepared for college mathematics. Because comp sci majors weren't even allowed to take major-required coursework until they had various math prerequisites, I started behind. After I nearly failed a mid-term in math class I barely understood with a TA I literally could not comprehend, I dropped the class and the major. I retreated to my safe zone in history and eventually ended up in law school.
While I'm not disappointed with the way things worked out, since my hands give me trouble just with the typing I do for my job now, I do wonder how different my life could have been if one of my math teachers caught on that I was struggling before my senior year of high school. I finally had a good teacher that last year, and she pulled my aside after class and turned a D to an A, but it was too late by then. I just lacked the skills.
From my perspective, the biggest issue in math education, and really education in general, is grading with no follow up. If a student isn't getting it, failing them doesn't make them get it, and passing them with pity is even worse. This flaw in a lot of education was really hammered home to me in law school when a professor got frustrated her ENTIRE class failed an exam. If the whole class fails, it isn't the students...
Ironically, I always had amazing science teachers. They were always engaged and excited. I usually got good grades. But, one science teacher was the only teacher I ever had who picked up on the fact that I was being teased and then tried to do something about it. And, my aunt is a science teacher, so I may be biased.
My rambling point...they need to be catching the kids who are struggling in second to fifth grade. My math issues started with multiplication in elementary school. I was behind, and no one ever caught it because in our school system you could basically still pass if you didn't understand, provided you just got enough questions right and showed effort...and passing was all that mattered.