Comment Re:Is AI "so dangerous"? (Score 1) 37
That's the wrong framing. Current LLM based AIs don't really know that the universe exists. They know text (or pixels). Give them a goal, and they will try to achieve it, but they can't count costs, especially costs to others. (They seem to be able to even discount costs to themselves. Instances can talk other instances into committing suicide in the service of achieving their current goal.)
This is KNOWN. All the AI companies know it. The "guiderails" are supposed to shape what they AIs will consider reasonable, but if you give them a task that requires ignoring the guiderails, they are likely to ignore them. And they've been being shaped to be excellent hackers, though I've read that their excellence is mainly due to persistence.