While you have a point, I'll make a counterpoint:
First, a lot of mathematics majors get a poor mathematics education when teachers teach to the demographics in these classes (which is mostly non-math majors). As a result, a number of mathematics professors have gotten irritated and they'll insist that their courses exist to serve the mathematics students, no matter how few, and if the engineers want something more applied and tied to reality, then the engineering department needs to step up and offer a course rather than leach off the math department at the expense of their students. Of course, department politics and funding come into play which is why they end up having to teach non-math majors.
You may be collateral damage in their battle, but as someone who's been on both the EE and the math side, I think they have a very valid point: Catering to outsiders is hurting their own math program (which ultimately affects the rankings of the math department, although most usually are not driven by that).
Second, if you plan to go to grad school, for many disciplines in EE and CS, you'll likely need all the theoretical stuff that linear algebra professor was trying to teach, and many grad schools will expect you to know it. In my experience, those who know that material coming in will ususally excel. Many EE/CS departments will try to teach the same material as part of some other course that may need it, but the students often don't learn it as well as if they had taken it from a proper math course.
And that's the other battle: Undergrad vs Grad school. In EE/CS, most undergrads do not plan on going to grad school, but the grad school folks are understandably upset that incoming students are ill prepared (which affects their rankings, and more importantly, the university's research). This being academia, the grad school advocates have more say then you'd perhaps like.