This is a very US-typical way of thinking.
In the UK, it's more of a "where is the harm" approach. If there is more perceived harm in the exercise of said speech than in allowing it, it won't be allowed. This is more difficult to administer (it means someone, usually a judge) has to make a decision about this rather than it just being black and white. It does make life more pleasant for more people.
Having lived in the UK and the US for over a decade each, I have some perspective on this, and personally I think it's worth it, worshipping at the altar of "Free Speech At All Costs[*]" is an absolute, and I tend to distrust absolutes.
[*] It's not a real absolute in the USA, you can't shout "Fire!" in a crowded theatre in the US either, for example, but it's a massively more common mindset of US people compared to UK people in my experience.