It's unfortunate that we call these things "hate crimes," because now it seems like simple hatred qualifies one for extra prosecutorial attention. In the U.S., the seminal hate crime was lynching. On the face of it, to the casual observer, lynching would seem like a simple murder—mob murder, perhaps. But it's more than that. Lynching was a punishment given to "uppity" blacks, and it's purpose was not merely to punish, rightly or wrongly, any particular individual but to terrorize an entire population. Lynching was meant as a public spectacle to keep the black community in line. Now, in a sane state, this would be a crime; but, back in the day, Southern law enforcement and Southern juries would look the other way and tacitly support lynching. That's why the federal government got involved. Otherwise, since simple murders are the province of the states, under our constitution, the federal government would have no business prosecuting. The federal government created this "hate crime" as a means of expanding its jurisdiction.
Essentially, properly conceived, a "hate crime" is a kind of terrorism—and it's bona fide terrorism, in this case. It's using egregious violence to intimidate a civilian population. It think, however, that prosecutors today, eager for publicity, pull out the "hate crime" card at every opportunity; and a gullible and uncritical public more or less eats it up. It's being used in too many instances. It's being misused.
Let's say two men, one black and one white, get into a bar fight. The white man is the instigator and gets the best of the black man. The cops come and arrest the white man for assault. Okay, that sounds about right. Now, let's just suppose that, in the course of the fight, the white man directs a rather unkind, racially charged epithet towards the black man. That's not nice. But, in most circumstances, it's a long way away from an action that terrorizes an entire community.
I think there is something fundamentally wrong with extending the notion of hate crimes to simple prejudice—or even gross rudeness. In most cases, even though there may be "hate," we should not be treating it as a separate crime.