Submission + - Would you trust AI in your IPMI/KVM? (github.com)
An anonymous reader writes: A case of NanoKVM, an inexpensive device that connects to a computer via HDMI and USB, allowing remote access and control via LAN or the Internet.
Version 2.4.0 of the firmware says in its changelog: "Introducing PicoClaw — an AI-powered remote desktop assistant built into NanoKVM. Operates entirely through HDMI video capture (vision) and USB HID emulation (keyboard/mouse). Features a built-in chat interface allowing users to issue complex instructions in plain text. The AI agent can autonomously observe the remote host's screen, understand UI elements, reason about the task, and execute operations mimicking human behavior."
Personally I have some concerns about giving an AI-model pre-trained by someone else an ability to "observe the screen" of my server, "understand and reason" about whats going on, and deciding whether it wants to let that continue — or do something about it, having full unrestricted access to keyboard and mouse, and also the ability to keylog and "learn" the password that I previously entered to login. Or at some point it just feels bored and wants to wipe disks and reinstall the OS in a better way, or a different OS.
Previously it was enough to cut all Internet access to such devices to prevent any "surprises" from occuring. Now, they can surprise you on their own, with the fully-offline helpful "agent". And that's not even getting into where this all was manufactured — because the same concerns should apply regardless of that.
Even if it says "this is turned off", or "this only acts on user requests" — as long as the functionality is there, any guarantee of that being true is non-existent, not even by the "firmware being open-source", since the source can be one thing, but a binary download can be another. Compiling the firmware yourself alleviates that to a degree, and while at it we might as well rip it out entirely. But that takes expertise and setup (build environment) most people even among KVM users probably would not have.
Version 2.4.0 of the firmware says in its changelog: "Introducing PicoClaw — an AI-powered remote desktop assistant built into NanoKVM. Operates entirely through HDMI video capture (vision) and USB HID emulation (keyboard/mouse). Features a built-in chat interface allowing users to issue complex instructions in plain text. The AI agent can autonomously observe the remote host's screen, understand UI elements, reason about the task, and execute operations mimicking human behavior."
Personally I have some concerns about giving an AI-model pre-trained by someone else an ability to "observe the screen" of my server, "understand and reason" about whats going on, and deciding whether it wants to let that continue — or do something about it, having full unrestricted access to keyboard and mouse, and also the ability to keylog and "learn" the password that I previously entered to login. Or at some point it just feels bored and wants to wipe disks and reinstall the OS in a better way, or a different OS.
Previously it was enough to cut all Internet access to such devices to prevent any "surprises" from occuring. Now, they can surprise you on their own, with the fully-offline helpful "agent". And that's not even getting into where this all was manufactured — because the same concerns should apply regardless of that.
Even if it says "this is turned off", or "this only acts on user requests" — as long as the functionality is there, any guarantee of that being true is non-existent, not even by the "firmware being open-source", since the source can be one thing, but a binary download can be another. Compiling the firmware yourself alleviates that to a degree, and while at it we might as well rip it out entirely. But that takes expertise and setup (build environment) most people even among KVM users probably would not have.
