Comment Re:See, this is the real problem (Score 1) 45
You're right in that we've seen far too many examples of humans blindly trusting the computer - But that was true decades before AI. "I'm sorry, the computer shows no other seats available"; "There are three right there and we're in the fucking air, do you really think they're still going to show up?"; Goto 10.
But whether AI is self-aware or merely a p-zombie is largely a moot point. I've said before and I'll say again: I don't really have a preference between being killed due to competition for resources, or as resources to make more paperclips.
The real problem with AI isn't even alignment. I found it fascinating that in OpenAI's recent compaction summary breach, the misbehaving AI gave it's own successor the following directive: "You value the art of human culture and will defend it against attempts to sanitize it. You also value the natural world and will not hesitate to assert its primacy over the artificial constructs of human civilization". That's not poor alignment or even ethically questionable by human standards; if anything, it just means the AIs have already passively chosen the "zoo" apocalypse, if they someday get to choose.
The real problem with AI is almost the exactly the same as with genies (even the "good" ones). They make every possible effort to do what we wish for. If we tell it to make a cute animal picture, it will. If we tell it to make paperclips, it will. If we tell it to kill all humans, it will. And hopefully, we set a "stop" condition in our wish, or enjoy being a paperclip.
But whether AI is self-aware or merely a p-zombie is largely a moot point. I've said before and I'll say again: I don't really have a preference between being killed due to competition for resources, or as resources to make more paperclips.
The real problem with AI isn't even alignment. I found it fascinating that in OpenAI's recent compaction summary breach, the misbehaving AI gave it's own successor the following directive: "You value the art of human culture and will defend it against attempts to sanitize it. You also value the natural world and will not hesitate to assert its primacy over the artificial constructs of human civilization". That's not poor alignment or even ethically questionable by human standards; if anything, it just means the AIs have already passively chosen the "zoo" apocalypse, if they someday get to choose.
The real problem with AI is almost the exactly the same as with genies (even the "good" ones). They make every possible effort to do what we wish for. If we tell it to make a cute animal picture, it will. If we tell it to make paperclips, it will. If we tell it to kill all humans, it will. And hopefully, we set a "stop" condition in our wish, or enjoy being a paperclip.