The U.S. so far has been using them in conflict with terrorists. Hackers/terrorists would be using them for crime or terrorism.
I know the arguments-- why is it different when the government does it? Why is it OK for the government to do it? Why is it OK that they're killing civilians? Women? Kids? And so on, ad infinitum.
The difference is that they're (currently) being used in a "war" against a non-nation state. There are civilian and innocent casualties in all conflicts, but the casualties so far have been less than in more conventional warfare. Never before have we been able to target specific individuals in a conflict except on the most limited scale (i.e. Saddam or Usama). I think it's important not to lose sight of that fact, just as it's important to continue to try to reduce civilian/innocent casualties either with or without drones, and to avoid conflict where possible in the first place. The government using them for legitimate (as far as governments attacking people is legitimate) purposes is apples to oranges compared to hackers potentially using them for whatever they would use them for.
I see where you're coming from, but it's not entirely valid. And it has nothing to do with skin color, race, religion, nationality etc., it has to do with the actions undertaken by the targeted individuals.