While my intuition tells me that high school grads are, on the whole, not as well prepared as they should be, there is certainly some improvement that could be done at the college level.
One problem I faced on the path to my EE degree was that in mathematics classes and some engineering classes (particularly electromagnetic fields, communication systems theory, and stochastic signal analysis -- which of course are some of the most math/calculus heavy of the EE curriculum), was that I lacked an intellectual model of what the mathematics was accomplishing. While concepts like derivatives and integrals made a degree of sense because they could be related to velocity, acceleration, position, area, and volume, when I got to the point I was dealing with eigen-this and eigen-that and hermetian-something-or-others I had lost any real-world connection, and my understanding suffered as a result.
The most frustrating and poignant instance of this was the first day of my linear algebra class, which I was taking only as a pre-req for CS class on GUIs, which only needed it to the extent that rotation, translation, and scaling using matrices was involved, and I already knew that much. Anyway, the mathematics professor walks in and announces "I do not care, even one little bit, what this material is used for in the real world. I am here to instruct you in mathematics alone." I looked around the room. In a class of about 25, I believe there were 20 science/engineering students, 4 math students, and one photography major (she was one of those brilliant types who took upper level classes in sciences, math, philosophy, or anything else just for fun). I was somewhat incredulous at the professor's utter disregard for his students' background, abilities, and interests. And just as I expected the course was utterly miserable and tedious, and then there were the bad days.
I contrast that with the math classes I took for Calculus II-IV, and Numerical Systems Analysis. The professors (thank heavens I avoided graduate students) who taught those classes were totally on top of the situation, and made it very clear what we were trying to accomplish with real world examples, or at least didn't veer too incredibly far from intuitive models. I think it helped that in Calc II-IV I had the same professor all through, and he was teaching a pilot course that integrated calculators into the material, so there was a lot of approachable material throughout. This was a stark contrast from the previously mentioned Linear Algebra as well as the Differential Equations I courses.
To this day I hate Linear Algebra and Differential Equations, and I'm 100% convinced it's due to the terrible instructors I dealt with. Which is a shame, because I loved mathematics in high school, and would go beyond my coursework to explore what I could on my own without much additional help from my (incredible) high school teacher, and I had a blast doing it. If I hadn't developed a strong interest in aeronautics and computers I most likely would have pursued a math degree.
The biggest problem I faced throughout my mathematics education, as well as many engineering classes, is that as the course would progress it was building taller and taller upon a shaky foundation. While my arithmetic was bedrock, my algebra was concrete, and my trigonometry was 2x4 construction, the rest was a lot less solid. Calculus felt a lot like building with Tinker-toys, and by the time I got to anything past that it was toothpicks stuck together with Sticky-Tack. As more and more material was piled on top, a lot of it kept slipping off because the stuff underneath it was crumbling. I would have benefited greatly from either better construction (i.e. better instruction), or a lot more hands-on experience with those shaky bits such that they were strongly reinforced.