However, as I understand it, the first amendment says you have freedom of speech - but it does not say that anyone has to listen. You can't get thrown out of a public park for saying hateful things (provided they are not calls for physical violence) but if you come into my house I can force you to leave if you insult me. The A&E network does not agree with what this guy said, so they kicked him out of their house. He is still free to go speak his mind elsewhere.
In other words, his freedom of speech was not infringed upon. A private entity told him they did not want to do business with him. If I went into a 7-11 and insulted the cashier I could expect much the same response.