We actually *don't* ban gambling or alcohol. In many parts of the US, state governments actually run (and advertise!) lotteries, and in a few states (New Hampshire and Pennsylvania come to mind), the state government has the monopoly on consumer alcohol sales.
In recent years, commercial gambling has become legal in more places, marijuana has been legalized in more than half the country, and other narcotics are becoming decriminalized if not legalized in more and more places.
To your point, these policies *are* in fact ensnaring precisely the fragile people you alude to, and I wasn't a fan of having to smell other people's weed wafting up through the vents when I lived in a large multiunit building a while back, but whether because of freedom, greed, or both, the legal vices aren't going anywhere.
So why treat the phones differently? As if they're more dangerous than putting literal poison in your body?
I don't have to believe in spherical libertarian cows to see this as the mindless moral panic and shakedown operation that it is.