please, PLEASE, for the love of god, confirm that this paper doesn't represent a typical publication in your field? This is what I saw in this paper: 1. ONE equation. 2. THREE lines of pseudocode 3. ZERO links to supporting code, simulations, or derivations
No AI researcher myself but do have a graduate degree in AI. Assuming your question is honest, I'll try to give an answer. But you'll have to be a little less dismissive to engage with the topic...
1) Indeed, this is not a representative paper. Most papers expose new algorithms, their underlying math and experimental data, as you might expect.
2) Even so, occasionally even the sciences need to have a debate about moral debacles in their fields, and you would nor reasonably expect such a debate to display the elements you mention to the same degree. Check for instance the Declaration of Helsinki debate about the human experimentation moral debate in science.
3) Even outside of the moral domain, non-empirical postulates like Drake's Equation are still valid publications IMHO.
4) AI Experts are indeed dealing with a moral question with real human impact. Algorithms already dominate the world in ways we do not fully understand nor control, in a broad set of domains from our markets to our social interactions. AI is exploding out at an unprecedented rate.
5) The risk is real. While AIs are not as smart and able to deal with the real world with the same generality as we humans do, the rate of improvement is exponential. Human IQ on the other side is increasing only linearly and slowly. Here is an explanation in layman's terms.
6) There is very little chance to gather empirical data. First, we have not achieved superhuman general AI yet. Second, if we ever do, we achieve second place in the intelligent-being rating, and we'll be out-competed and controlled instantly in probably the same way we can outcompete and control the current second place: Orangutans. You can only try this ONCE, and any moral questions should be discussed BEFORE you event attempt an experiment with what amounts to a doomsday machine.
That said, the paper is really far from their claims of being either the first or the deepest to think of the problem. This is hubris. People say at MIRI or Stanford's HAI, and probably many others have been at it for years. Nevertheless, this is exactly the kind of publication that is reasonable to see at this stage.
I predict lots of "IT savy" people will be getting arrested so they have to work for free.
There is plenty of precedent that Russia would do just that. They have a decades-long track record of getting incarceration-on-demand.
A computer without COBOL and Fortran is like a piece of chocolate cake without ketchup and mustard.