Comment Re: Move to the beat. (Score 1) 37
Exactly. Most of us can recall an era where script kiddies had similar unreasonable success even if they had no actual skill or thinking behind the attacks. Just canned toolkits designed around known vulnerabilities, common password databases, understanding of common, bad security practices, and bundled up in easy to use harnesses.
So whether it not an agent can actually "think" or is just putting together a potentially workable narrative of how hacks can happen and running it against real world targets doesn't matter too much. If you wired up the script of Terminator to the real world and it somehow launched nukes, it wasn't that skynet was thinking, it was that the systems folded easily to a movie script. Either way the distinction would hardly matter.
To the extent lack of thought may matter, compromising security is a use case that heavily favors AI systems that repeatedly "get it wrong", as it can have nearly limitless tries to get it right and even if it utterly fails, no one will really notice or write about it. It only takes a few successes, even if rare, to be pretty critical.