... the idea of "hate crimes" still seems silly to me...something is a crime or it's not, we can't legislate hate IMO.
We don't legislate "hate". "Hate crimes" are not the criminalization of "hate", but sentencing enhancements applied based on motive or intent. And you understand that we do punish different crimes differently based on intent, right? For example, premeditated murder is usually punished more severely than heat-of-the-moment murder, which is punished more severely than accidental homicide or reckless "manslaughter", which is punished more severely than negligent manslaughter, etc. If I swing my arm without paying attention and bop you in the nose, that's bad, but it's not as bad as if I intentionally bop you in the nose... and we as a society have decided that that is not as bad as if I intentionally bop you in the nose because you're a member of a minority group I dislike.
Or, to look at it another way, if I bop you in the nose because I dislike you, that harms you, and I should be punished for that single instance of harm. But if I bop you in the nose while ranting about people of your religion/race/gender/etc., I'm doing it to terrorize or intimidate other people of that religion/race/gender/etc. - I'm physically harming you and sending a message to others like you that they should beware because I'll try to harm them in the future. Accordingly, I should be punished for that increased harm.
And remember, there has to be evidence of that intent. If I bop you on the nose because you're a member of group X, but I never say a word about that, then I'm not going to receive an enhanced sentence simply because you're X and I'm Y. It's only when I take the additional action of letting my intent be known - and as noted above, I would do that because I'm trying to intimidate other X's.
So, in short, we're not criminalizing "hate", we're criminalizing domestic terrorism. And I'm fine with that.