Criminals should be punished commensurate with the harm their actions cause. The fact that you can imagine the same act causing less harm if some other victim were substituted isn't a defense. It's like defending yourself at a murder trial by saying "sure, I shot that guy in the face; but I shouldn't be punished for it because if the victim had been a midget, instead, the bullet would have passed harmlessly overhead. It's his fault for being so tall."
It doesn't make any sense. There's no provision in criminal law that I'm aware of for "affirmative defense by substitution of hypothetical victim who would have been harmed less by the same action."
You're right it doesn't make any sense, because thats not what I'm advocating. The crime of burning a cross on someones yard should be punished the same regardless of race, religion, sexuality, ect. If it can be proven that the crime terrorized the community then separate terrorism charges should be leveled against the offender, in addition to the original charges. The original offense of burning the cross should be treated the same regardless of race though. To treat it in any other way is racism in itself and is just plain wrong in my mind.