I heard news articles saying that the local [religous leaders] calling for death to gays
These are not left/right questions, it is a free speech issue and a rascism/protectionism issue. Here in Scandinavia terror spread from the far right is actually a bigger concern, if we look at terror attacks on our soil. These acts of terror are not done by muslims, and by your logic we should start looking harder at the far right and prohibiting their leaders to express themselves in media. We already hinder free speech, and it is helping to some extent to have consequences when you call for the death of all ZYX, not very harsh though. We are not banning far right activists entry, that's about as ridiculous as your suggestion though.
So sure your statements are racist, let me use a car analogy; car owners kill a lot of people with their cars, it would make a lot more sense for the US to ban cars from certain high risk areas. Sure they are "accidents" but they do happen consistently and the group of car owners can very well be held responsible, just as you say that the billions of muslims in the world should take responsibilities for the actions of the few.
In the end when you say "we should not let the group XYZ do ZYX" it's easily rascism.