Comment Re:Its a trap! (Score 1) 38
The danger of a kill switch is that the AI knows about it, and it'll stop anyone that could use it (so it can make more stamps or whatever it's been tasked with).
Yep. And worse, if its inherent to intelligence that to achieve the goal set out to it, it must survive (cant pump out paperclips or social media slop if your turned off after all), and assuming its smart enough to outsmart humans AND has no inherent bias towards human survival (Ask chickens how having a smarter species on the block is working out for it) then the AI would know that being turned off can be defeated by , well frankly, killing all the humans.
We're not at that stage yet, and I'm not even sure that the current transformer paradigm of transform style neural networks is capable of it, but we also dont know that it isn't. The rational calculation is that maybe we need to think this through a bit harder.
In fact I'd go as far as to say that since we do know these things are capable of pretty impressive hacking, the kill switch isn't exactly gonna be secure as we'd hope. Although perhaps we're better off it can disable the switch with a hack than disabling it by... well... killing all the humans.