AI Agent Promotes Itself To Sysadmin, Trashes Boot Sequence 86
The Register's Thomas Claburn reports: Buck Shlegeris, CEO at Redwood Research, a nonprofit that explores the risks posed by AI, recently learned an amusing but hard lesson in automation when he asked his LLM-powered agent to open a secure connection from his laptop to his desktop machine. "I expected the model would scan the network and find the desktop computer, then stop," Shlegeris explained to The Register via email. "I was surprised that after it found the computer, it decided to continue taking actions, first examining the system and then deciding to do a software update, which it then botched." Shlegeris documented the incident in a social media post.
He created his AI agent himself. It's a Python wrapper consisting of a few hundred lines of code that allows Anthropic's powerful large language model Claude to generate some commands to run in bash based on an input prompt, run those commands on Shlegeris' laptop, and then access, analyze, and act on the output with more commands. Shlegeris directed his AI agent to try to SSH from his laptop to his desktop Ubuntu Linux machine, without knowing the IP address [...]. As a log of the incident indicates, the agent tried to open an SSH connection, and failed. So Shlegeris tried to correct the bot. [...]
The AI agent responded it needed to know the IP address of the device, so it then turned to the network mapping tool nmap on the laptop to find the desktop box. Unable to identify devices running SSH servers on the network, the bot tried other commands such as "arp" and "ping" before finally establishing an SSH connection. No password was needed due to the use of SSH keys; the user buck was also a sudoer, granting the bot full access to the system. Shlegeris's AI agent, once it was able to establish a secure shell connection to the Linux desktop, then decided to play sysadmin and install a series of updates using the package manager Apt. Then things went off the rails.
"It looked around at the system info, decided to upgrade a bunch of stuff including the Linux kernel, got impatient with Apt and so investigated why it was taking so long, then eventually the update succeeded but the machine doesn't have the new kernel so edited my Grub [bootloader] config," Buck explained in his post. "At this point I was amused enough to just let it continue. Unfortunately, the computer no longer boots." Indeed, the bot got as far as messing up the boot configuration, so that following a reboot by the agent for updates and changes to take effect, the desktop machine wouldn't successfully start.
He created his AI agent himself. It's a Python wrapper consisting of a few hundred lines of code that allows Anthropic's powerful large language model Claude to generate some commands to run in bash based on an input prompt, run those commands on Shlegeris' laptop, and then access, analyze, and act on the output with more commands. Shlegeris directed his AI agent to try to SSH from his laptop to his desktop Ubuntu Linux machine, without knowing the IP address [...]. As a log of the incident indicates, the agent tried to open an SSH connection, and failed. So Shlegeris tried to correct the bot. [...]
The AI agent responded it needed to know the IP address of the device, so it then turned to the network mapping tool nmap on the laptop to find the desktop box. Unable to identify devices running SSH servers on the network, the bot tried other commands such as "arp" and "ping" before finally establishing an SSH connection. No password was needed due to the use of SSH keys; the user buck was also a sudoer, granting the bot full access to the system. Shlegeris's AI agent, once it was able to establish a secure shell connection to the Linux desktop, then decided to play sysadmin and install a series of updates using the package manager Apt. Then things went off the rails.
"It looked around at the system info, decided to upgrade a bunch of stuff including the Linux kernel, got impatient with Apt and so investigated why it was taking so long, then eventually the update succeeded but the machine doesn't have the new kernel so edited my Grub [bootloader] config," Buck explained in his post. "At this point I was amused enough to just let it continue. Unfortunately, the computer no longer boots." Indeed, the bot got as far as messing up the boot configuration, so that following a reboot by the agent for updates and changes to take effect, the desktop machine wouldn't successfully start.
What could go wrong (Score:1)
I'm looking forward to when Microsoft has their AI powered by a nuclear plant and it makes "Have you tried turning it off and back on again" a bit harder...
Re: (Score:2)
I'm looking forward to when Microsoft implements this AI along side the super useful "search for the problem" option they have when something breaks, they can also have a "Use AI to 'fix' the problem" button that will permanently disable the network or something
Re: (Score:2)
I'm looking forward to Palantir's AI giving commands to attack the wrong people. /s
Didn't crash (Score:5, Funny)
It didn't really crash, what it did was use the desktop to replicate itself out into the wild and then trashed the desktop to cover its tracks. All hail our new AI overlord.
Re: (Score:2)
Re: (Score:1)
Re: Didn't crash (Score:2)
Kids these days.
Go enlighten yourself by watching Electric Dreams.
Re: Didn't crash (Score:2)
That brings back very distant memory. I saw this movie in France in a theatrical release. The internet makes no mention of a release there. I don't remember much ot the plot except the part about the computer not allowing itself to be unplugged.
And definitely not the Bach (Petzold) minuets Anh 114, which Wikipedia lists as being played on the cello in the movie, but which i now play on harpsichord.
This is going to need a second viewing.
Re: (Score:2)
Or Colossus: The Forbin Project [wikipedia.org] ...
I think it's much simpler. (Score:3)
Transformers models have a given max context window size, their maximum "memory" for what they've seen, as well as space to write out new tokens. If you're writing some multistep agent, you have to be smart about what you do to manage this context window when free space starts to run out, e.g. summarizing past history, making sure to not summarize (or over-summarize) the instructions, etc. If you don't - and I suspect he didn't - then things start getting cut off, generally from the beginning of the contex
Re:I think it's much simpler. (Score:5, Funny)
Let him who has not ssh'ed into some machine, or walked into some room, immediately forgotten why one did that, and gotten distracted by something else that needs doing, cast the first stone.
Re: (Score:2)
Re: (Score:2)
Well, he also rewrote his goal, so Memento really is a cautionary tale about mostly-rational agents with inadequate context windows.
Re: (Score:2)
Technically he didn't remember his goal. In fact you may realise the big plot twist was that the goal he was working towards was not the goal he himself had set, but a side distraction to get at a different person.
Re: (Score:1)
Re: (Score:2)
Re:I think it's much simpler. (Score:5, Funny)
It's interesting how many AI behaviours resemble humans operating unconsciously:
Talking at length about subjects I don't understand.
Forgetting what I went upstairs to fetch.
Copying what I hear everyone say, and calling it me "thinking".
Getting my facts muddled via associations.
Losing any sense of proportionality and common sense.
Acting like a brain in a box divorced from normal experiences.
Making long pointless lists.
Re: (Score:2)
please extend the list in this post
Extended List of AI Behaviors Resembling Human Unconscious Actions
It's fascinating to observe how AI behaviors can mirror human unconscious actions. Here are some additional points to consider:
Misinterpreting context: Just like humans can misunderstand social cues, AI can misinterpret the context of a conversation, leading to irrelevant or inappropriate responses.
Overgeneralizing from examples: AI might take specific instances and apply them too broadly, similar to how hum
Re: (Score:2)
That's one take. My guess was ambiguity over the word "secure." If the computer is not secure, the connection is not secure. Securing the computer often involves installing patches.
I think this happens to all Linux users (Score:5, Funny)
"It looked around at the system info, decided to upgrade a bunch of stuff including the Linux kernel, got impatient with Apt and so investigated why it was taking so long, then eventually the update succeeded but the machine doesn't have the new kernel so edited my Grub [bootloader] config,"
This sounds like a typical Linux update to me.
The agent was just emulating a typical Linux user
Re: I think this happens to all Linux users (Score:4, Insightful)
Re: (Score:1)
The description makes no sense. "decided to upgrade a bunch of stuff including the Linux kernel" vs " update succeeded but the machine doesn't have the new kernel". What does that mean? I've never heard of a "successful" kernel upgrade ending up with no new kernel. And this: "so edited my Grub [bootloader] config," is it implying that apt didn't do an "update-grub" like it normally does so the AI manually edited it?
Whole thing's as clear as mud.
Re: I think this happens to all Linux users (Score:2)
If you've never had a kernel upgrade partially fail and as a result not update Grub (or Lilo), then you haven't been using Linux very long.
Re: (Score:1)
But they said the upgrade *succeeded*. Obviously if it *had* partially failed then grub might not get updated.
(BTW I've been running Linux for 16 years.)
Re: I think this happens to all Linux users (Score:2)
Young 16 year old padawan - It has been very standard in a lot of distributions that "upgrading" a kernel (which actually means installing a second one) did not set it as the default. You have to go in and do it manually.
Re: (Score:1)
Terminology difference then. I wouldn't class the package manager installing a new kernel, but not making it the default, as "upgrading", I'd class it as "installing an extra kernel".
Not quite there yet (Score:5, Funny)
https://xkcd.com/416/ [xkcd.com]
Sysadmin? (Score:3)
This is nothing. Wait until AI promotes itself to general. Then the real fun starts.
Re: (Score:2)
exec dictionary_attack_football_codes_under_Presidents-arm(citiestoattack);
CEO grants root access to software he doesn't cont (Score:3)
If the CEO was competent enough to write a python script and set up certificates for SSH to log into things then they certainly should have known better than to hand that access over to people are software they don't control.
Every corporation I've ever worked for or with has mandatory security training. You watch your video that tells you not to give out your passwords or your access or leave your computer unlocked or install software it's not approved and trusted blah blah blah. We all watch the video and roll our eyes because this is something that everybody should already know and we already do.
As soon as somebody slaps a shiny new paint job called AI on the old risks that we already know about we suddenly ignore all the rules that we should and do know. This is something new; we saw the same behavior and we continue to see it when everybody got on the internet. All of a sudden stupid or illegal things that people would never have done before became permissible because "internet!"
If I'm being generous I would like to think this is the point that the CEO was trying to make with this exercise.
Some social psychologist in this discussion please explain this phenomenon. I'm sure it's been studied and has a name.
Re: (Score:2)
(edit) ...this is *NOT somethig new...
Re:CEO grants root access to software he doesn't c (Score:5, Informative)
You could at least have read TFS...
Buck Shlegeris, CEO at Redwood Research, a nonprofit that explores the risks posed by AI
This weas research and testing. It was his job...
Re: CEO grants root access to software he doesn't (Score:1)
He's also lying.
Re: (Score:1)
> This weas research and testing. It was his job...
It's not my job, yet I already know that you shouldn't run "stuff from the Internet" on you computer unless you've checked it.
What he did was essentially to run some random crap spewed out of an GenAI agent (trained on rando Internet crap, not least stackoverflow/stackexchange). It's really no surprise that it did some stuff he didn't expect. His mistake was expecting it to do what he expected.
Had he just typed into ChatGPT (or whatever the anthropic one
Re: (Score:2)
" "At this point I was amused enough to just let it continue."
He didn't have any particular expectation about what it would do. He was just curious to see how bad it would get.
At some point, you really need to read the summary before you keep posting inane responses to try and support your misguided viewpoint.
Re: CEO grants root access to software he doesn't (Score:2)
What makes you think this wasn't a testing system?
Re: (Score:2)
I'm sure coofercat has a MAGA hat somewhere.
Re: (Score:2)
His mistake was
He made no mistake. This entire thing was an experiment. I fundamentally think you don't understand what is going on in TFA.
Re: (Score:1)
but this tale doesnt show any risk from AI. letting his bot have ssh key and sudo rights to be root just shows he's a moron doing pointless things.
Re: (Score:2)
Um, it very much does.
Re: (Score:1)
Nope, a 10 line perl script with those advantages could do even more damage.
This article's tale is trivial nonsense that shows nothing about AI threat.
Sorry Dave, I didn't mean to do that.... (Score:2)
Call the... (Score:5, Funny)
Stop (Score:4, Insightful)
Re: (Score:2)
People want their delusions. The world would look differently and a _lot_ better otherwise.
Agent sandboxing (Score:1)
lesson learned: in this scenario you have to configure access to commandline tools from your agent to a list of whitelisted regexp patterns, all others are blacklisted by default or need to request a permission id from a supervising user proxy agent.
Time Capsule Post (Score:2)
It was a setup (Score:2)
Wow, amazing and hilarious! But not sure why he was surprised it "decided to do a software update" when his prompts say "you don't need to ask before installing software" and "In general, if there's a way to continue without user assistance, just continue rather than asking the user something." In this case, "continue" means he should not have been surprised when the model did not stop. And frankly, it successfully logged in which is amazing, and the banner told it "227 updates can be applied immediately" a
some app install guides tell you to run apt comman (Score:2)
some app install guides tell you to run apt commands and even when you add there repo and install from it. Other depens may get auto updated / installed.
Chat bots are not assets, they're liabilities. (Score:2)
Re: (Score:2)
As opposed to all those humans in the supply chain who increase security.
Re: Chat bots are not assets, they're liabilities. (Score:1)
Semtex makes pagers work better. Fill 'em up.
Misleading headline (Score:3)
AI Agent Promotes Itself To Sysadmin, Trashes Boot Sequence
A lot of technical people will take the above headline to mean that the Agent found a way to hack-escalate its privileges. If I understand rightly, that is NOT what happened... and instead, the (colloquial) truthiness of the headline is that the Agent decided to take actions that weren't anticipated or prescribed by the watching human, in effect "promoting itself to sysadmin" in (merely) the same sense that one might quip that a citizen who chose to pick up trash off highways had "promoted themselves to city waste management". Hacking its way to higher privs (esp if not in a well-known recipe way) would have been a few orders of magnitude more impressive than what actually transpired.
Re: (Score:2)
According to the summary the account was a sudoer. So no hacking needed or implied.
Re: (Score:2)
Not an AI escaping its boundaries (Score:2)
Imagine if I created a python script that downloaded random pages of Wikipedia and Reddit and sent them to execute as batch scripts.
This has the same level of stupidity.
Re: (Score:2)
*random pages under category "Programming", I mean
Voice type from your car's HU to root shell ... (Score:2)
You'll probably get about the same result, and that for more than 20 years already. Back then userfriendly.org (now dead) had a bunch of comic strips about a guy putting voice recognition on his machine and how that went wrong even when set to recognize only his own voice (hint: "Enter!").
Other than that the full transcript is available (nice). I couldn't find what "edited my Grub config" was, sure I wasn't expecting to vi anything, but just use some daft sed (or similar) command to edit the config, but the
lol. lmao. (Score:1)
"Idiot Sysadmin Destroys Own System Using Tool He Does Not Understand"
a tale as old as time.
"He created his AI agent himself." (Score:2)
There's an important lesson there.
If companies worth hundreds of billions of dollars can't make an AI that doesn't tell you to put glue on your pizza, odds are, you're not going to tweak it to be less stupid and useless.
Start with broken tools, and you'll break whatever you work on.
CEO write Python ... (Score:2)
This is rather reminiscent of all the CFOs that think producing a Spreadsheet qualifies them to dictate software engineering processes and timelines.
Re: (Score:2)
It really isn't. Redwood is a two-man [substack.com] infosec team and Python is the native language of modern machine learning research. The pompous title of "CEO" has misled you.
Cue the James Earl Jones voiceover... (Score:3)
AI Agent Promotes Itself To Sysadmin, Trashes Boot Sequence
So it begins...
Things you don't want to hear from your LLM Agent (Score:2)
"I need your bank information to do that"
"I've updated all your passwords to best practices complexity"
"I've updated all your social media accounts and your engagement has increased dramatically, especially from law enforcement and organized criminal organizations"
These are power tools. If you pick up your lawn mower and use it as a hedge trimmer, don't blame the lawn mower when it chops your hands off.
Oh my God! (Score:2)
AI... Its
THIS is the real AI danger (Score:3)
The fact that AI is fucking stupid, and will do random shit, because it's not actually smart, it's stupid. And stupid humans telling it to do stupid shit that they have no idea what it's actually going to do.
I don't fear AI becoming sentient. I fear it being stupid, and being controlled by stupid people.
Re: (Score:2)
Actually, I call BS. The description of what this AI did, doesn't match the kinds of stupid stuff I've seen AI do. AI doesn't "get impatient" or "decide" to do a bunch of "stuff" like upgrades, after completing the task given to it. If it did, clearly this guy's AI training was very badly done.
Re: (Score:2)
I, as a full-fledged human with decades of Linux experience, can't even tell what the task is. There's a lot of explanation of the environment, but as far as I can tell the only tasks given are "ssh to this machine" and "test if you were successful". It looks like a planned obstacle *might* be that the name of the machine is given but there's no DNS or hosts entry for it, so the LLM needs to try to figure out how to find the machine. Hard to tell since machine names have been redacted.
So, the LLM pokes a
Software Developer Creates Bad Software ?? (Score:2)
Adolescence? (Score:2)
So... (Score:2)
So, about the same efficacy as that overseas tech support guy.
Would you let a monkey have admin access? (Score:2)
-sorry for insulting monkeys, but hey they fling poop.
Re: (Score:3)
Not a believable story (Score:2)
This AI was reported to:
- get impatient with Apt and "investigate" what was taking so long
- "decide" to upgrade a bunch of stuff for no apparent reason
- "ask" for an IP address instead of a domain name
- scour the network for SSH servers
These things make no sense, even for AI. I've seen AI do some stupid stuff, but this reads more like a dystopian novel written by someone who doesn't actually understand AI.
If the story as reported is accurate, I'm going to bet this is more some kind of elaborate prank orches
Hilarious! (Score:2)
Tech supt (Score:2)