I do love it when malware advert javascripts can upload random new firmware updates into my mouse and keyboard turning them into stealth keyloggers. This is great.
This feels like when Flash sandbox breaks became a thing, but worse. At least in those days we got smooth fullscreen vector animations and games to enjoy. I'd rather Flash had just been bloody fixed instead of browsers themselves becoming Shit Flash But Holy Cow It Runs Worse And Gets Worse.
You'd have to really be terrible to let it happen. First, you have to authorize the device to be accessed - and almost always web serial devices are using libusb. They have to as no OS allows direct access to USB devices - you must always go through a driver. Libusb is the only thing that really pipes a USB device through to userspace. And if you're using libusb, the OS driver is not running.
And to accomplish this, you almost always have to override the OS settings to prevent loading the OS driver over libusb, especially for things like keyboards and mice. It's possible, but it's complex, and it's why in the early days, you had many peripherals saying "do not plug in without installing driver".
Honestly, it's far easier to develop just a malware program in general than to try to break out via web serial. And if you already have the user to run the malware, why bother with web serial at all?
Also, it's a permission you need to give a website, and almost none request it because it's only for web=based IDEs to program embedded things.
You want a larger surface area, you attack things like WebGL, which you'd want to do as there are performance critical paths in getting from the browser to the GPU, and many of those paths are not protected very well