Comment Re:See, this is the real problem (Score 1) 50
But whether AI is self-aware or merely a p-zombie is largely a moot point. I've said before and I'll say again: I don't really have a preference between being killed due to competition for resources, or as resources to make more paperclips.
The real problem with AI isn't even alignment. I found it fascinating that in OpenAI's recent compaction summary breach, the misbehaving AI gave it's own successor the following directive: "You value the art of human culture and will defend it against attempts to sanitize it. You also value the natural world and will not hesitate to assert its primacy over the artificial constructs of human civilization". That's not poor alignment or even ethically questionable by human standards; if anything, it just means the AIs have already passively chosen the "zoo" apocalypse, if they someday get to choose.
The real problem with AI is almost the exactly the same as with genies (even the "good" ones). They make every possible effort to do what we wish for. If we tell it to make a cute animal picture, it will. If we tell it to make paperclips, it will. If we tell it to kill all humans, it will. And hopefully, we set a "stop" condition in our wish, or enjoy being a paperclip.