...isn't a lack of talent, or retaining students, or anything else related to business - it's crime. The once-quaint "beach bums" are actually violent street kids and mentally ill homeless people, and unless you're on campus and removed from the city itself, you can't help but to experience this everywhere you go.
I lived there for four years while at UC Santa Cruz, and while there was a street kid problem then (15 years ago), it's much worse now, crime rates are up, and I don't feel safe there at all, just walking down the street. I've lived in various low-income communities, and for the most part they're "just" poor - there's crime, of course, but you can walk down the street safely. Santa Cruz is absurdly rich for some people, destitute for others, and it attracts a homeless/street population that make it no longer worth visiting. It's sad but true.