So your solution to trolls, thieves, and hackers is to just let them be? No, kick them out and don't let them back in.
If each day a thief comes into your town and steals things, destroys property, harasses people at the cafe, and kicks your dog. Are you really going to sit there and take it? Or are you the type that will just hide in your house and hope they don't bother you too?
If your town keeps tolerating that one thief, more will follow until your town is a dumpster fire
But sure, don't kick out anyone no matter what they do, since you should just let anyone do whatever they want to do to others. This is not how society remains intact.