Some of the hysteria is probably because a very low-altitude plane flying supersonic can probably break windows, so people are worried about that. Of course, we're talking about higher-altitude planes here, so that isn't really justified.
The other worry is probably the frequency of the booms. One boom a month isn't a big deal, but what if they decide to make a frequent flight path over your house in the suburbs or in the country? Now you've got sonic booms every day, throughout the day. No one wants that.
They don't occur often over the continental USA because they're illegal (except for military planes), and these days because there's no supersonic passenger planes left. The Concorde was the only one, and it was limited to trans-Atlantic routes over water because of the restriction on sonic booms. I don't think it's unreasonable for people to be worried about something that isn't a problem now because of legality and current technological state; these things can and do change. Texting-and-driving 20 years ago wasn't a problem either, but now it is; it would have been helpful if someone had worried about it a little more. Drunken driving wasn't a problem 120 years ago, but again maybe someone should have worried about it and done something before it turned into a big societal problem.
Personally, I don't even see why supersonic travel should exist for passengers. It uses insane amounts of fuel, and even if that problem is fixed, it's unlikely it'll ever be as fuel-efficient as subsonic flight. No one really needs to get around that fast. If you don't like 14-hour flights over the Pacific, then don't do them; stay at home instead, or choose a closer vacation destination. Maybe eventually we'll avoid this problem altogether with some kind of sub-orbital vehicle which is both fast and fuel-efficient, but I'm not holding my breath.