Love the generalizations on both sides of the issue. In short, it's possible (as long as you don't have young children.)
Living with relatives and/or roommates, smart financial decisions (picking where you go to college, what degree you get, what food you eat, etc.) along with student loans allow people below the poverty line to go to college from a financial perspective. Depending on the circumstances, you may have to take the first two years at a community college, but, it is more than feasible. I know, from personal experience.
The issue is, it is "hard". You (potentially) have to sacrifice things like going out, having cable TV, some of your privacy, and eating out for long periods of time. However, for many people it isn't nearly as bad, especially if they have assistance from relatives (living at home, free food, etc.) In my case I didn't have that luxury, but, many do.
The debate shouldn't be whether or not it is possible, but, it should instead be:
* Is it realistic for most people to do that? People tend to take the path of least resistance and it is pointless (from a general perspective) to say "you should do XYZ", if only a small percentage do. Most will see it as too overwhelming and not even try.
* Do we as a society think it is "right" and "just" for someone to have to go through that in order to obtain a better life for themselves, or should society as a whole pay for some or all of it.
My personal opinion is based on my observations is that it really is cultural. Most people either get discouraged (don't think they can do it/get overwhelmed) after starting, or don't start at all.
If you have young children, it is a different story though.. My opinion is, just give free day-care, ensure community colleges and state colleges have a reasonable tuition, then lead by example. Make sure people know it IS possible, and offer some type of support system through the colleges to set people up with roommates, etc.