Absolutely right. I grew up in an economically disadvantaged area, went to college, and settled in one of the best-performing metro areas in the country. My classmates who skipped college are still there, driving 1-2 hours each way to the closest job they can find, and enduring the double disadvantages of lacking a college degree and living in a depressed area.
When one is living dangerously close to the poverty line, moving away from friends and family will be perceived as unacceptable risky. Only the most ambitious will leave, and most of those people went to college anyway.