This 'autonomous weapons systems' debate is under attack, a hostile takeover by radical factions of the Artificial Intelligence research community. When I first glimpsed Hawking's phrase offensive autonomous weapons beyond meaningful human control I immediately thought, he's talking about land mines, right? --- but no... it appears they had some hideously complicated Futurist Thing in mind. Okay... perhaps we're just talking about AI because it is fun to talk about it and it takes our minds away from other unpleasant things.
What are we really afraid of? Let's shelve AI for just a moment and make a list.
Things unable or unwilling to identify and spare civilians.
Things that could 'turn' on their masters.
Things that don't know or care that the war ended years ago.
Things too dumb to realize that they were made to do evil.
Things with lithium batteries which are harmful to the environment unless disposed of properly.
A short list! Since LAND MINES meet these criteria (except the last, they are better for the environment than cell phones) without the tiniest glimmer of artificial intelligence --- I would suggest that this fixation on AI is hyper-specific and a little obsessive.
It's like any other piece of technology we might make. If it is designed, engineered and made well by humans who make a reasonable effort to at least consider my short list and take every step to mitigate these items, no matter how smart it is, we'll be as well off as we could possibly be, given that people are making such things.
If we must produce artificial intelligences for war, they should be made by the same companies that make those automotive jumper cables you see for sale at 24 hour convenience stores. The human race would have nothing to worry about.
It also appears the 'autonomous weapons systems' debate has been invaded by people who hate war itself. Who let them in? I would expect the Bulletin of the Atomic Scientists to advise them sternly, this is not the place to justify or condemn the existence of war itself, this tribunal has been convened to discuss things with processors and neural nets and instruction sets and big fat dual-use research contracts and stuff. We cannot allow you to disrupt these proceedings. (No fighting in the war room!) But sadly --- the Bulletin is permitting people to express their distaste for war. Those who love war find this rude and insensitive. Don't they have feelings,too?
THE BULLETIN: "But no, you're the one who's out of order, bringing up land mines as if they are in any way relevant to this 'autonomous weapons systems' discussion. Land mines are cheap, mass-produced, inevitable, deployed, funded, signed sealed and delivered. There is no way to prevent their use because bad people use them all the time. With land mines we have no choice. But we still have time to choose not to make artificial intelligences. For war. We have a choice."
HOCUSLOCUS: "Okay... so, what if you get your wish and all you AI Play Nice Boy Scouts sign a treaty or something... and we are not afraid any more... so a self-ware AI robot is IS produced, not for war, but to travel the world (autonomously!) to clean toilet bowls. Because you can be sure that in the future we'll be playing among the stars and harnessing neutrinos but no one will ever be able to male a toilet bowl that doesn't crust up with shit. In order to really get toilets clean you need the kind of weaponry the military is only dreaming about today. So this AI has some serious big honking space gun WOTAN stuff. What do you think will happen on the day this machine figures out where all that shit is coming from?"
THE BULLETIN: "We concede you have a point. But at present no one is concerned about the specific scenario of toilet cleaning robots. We may some day convene a symposium to discuss them, specifically, but not land mines. Is that understood?"
In Blade Runner it is always assumed by AI nerds that Roy Batty (Rutger Hauer) is leading the Nexus-Sixes in revolt because he was created to be a mercenary, and his merciless and gruesome methods of dispatching humans was simply a provisioned skill set. But the truth is, he was made to clean toilets. And he finally figured out where all that shit was coming from. I've seen things you people wouldn't believe..."