On Twitter, even if you don't follow a person, they can still send you messages. To give another example I encountered (albeit not one involving "hate speech"), this woman online decided that I was the same person as another guy she had a problem with. Her proof? We both like photography. (She's not all there. She also claims to be a prophet of god and that god talks to her and tells her these things. Obviously, saying "you're mistaken" doesn't work.) She would harass me constantly on Twitter and, eventually, on my blog. I ignored it, but still it was annoying to come back and see a dozen messages from her. I'd block/report her and her account would be taken down, but she'd just start a new one up. (At one point, I and a few other people she was harassing found out that she had set up around a dozen accounts ahead of time for the inevitable account suspension.)
Now, her speech to me wasn't hate speech. (She was accusing me of murder/hacking/doing obscene stuff to kids/etc because god told her.) Still, she could easily have been sending me hate speech instead. Is my option in this situation "just shut down my Twitter account and don't use it anymore"? Is the only option for someone who is being harassed to leave the place where they are being harassed?
As far as deporting illegal immigrants goes, I don't see discussing it and the various policy proposals as being hate speech. It's HOW it's discussed that's the bigger issue. Saying "we should deport those illegal immigrants" is fine. Saying "all [derogatory term for Mexicans] should be rounded up and shot" obviously isn't the same thing. I welcome a rational immigration discussion. Unfortunately, there are many who use the discussion's opening to shout racist rants. This hurts both sides as the left reacts to the racists and the right has their reasonable plans drowned out by hate. (It's one reason why I'm hoping the GOP will fracture so that it can kick out the crazies and reformulate itself as a Reasonable Conservative party. I might be left-of-center, but I want reasonable options out there to keep the Democrats in check.)
Finally, I agree that I wouldn't want to see Twitter, Facebook, etc ban people for spurious reasons. People shouldn't be banned because the CEO of Twitter believes X and someone tried posting a reasonable argument why X is wrong. However, if someone is posting pure hatred and is harassing people, they should be kicked off. That's not fostering communication. That's trying to scare people into silence so the racists/bigots can force their view onto America.