It's really the same problem that we see with quantum physics -- the subject matter is esoteric enough that common sense can't be used to determine whether what you're listening to is really advanced theory or utter BS (or some combination of the two); they both sound the same, and the situation isn't helped by the presence of large numbers of people who think they know what they are talking about but don't. :/

I think the distinction they are making is between technology that kills people, and technology that entertains people. If someone fitted actual weapons to a Doom engine, they'd probably object.

This is why they're so keen on robots. They want guards whose loyalty they can really count on when the mob shows up with a guillotine.

Let's make the hopeful assumption that the superintelligent AIs will not "go rogue" in any way, but will correctly try to carry out their tasks as assigned by the people who designed and trained them.

Pretty much every major nation has military AIs, at this point. So the Chinese AIs are working hard on how to defeat the Americans, and the American AIs are working hard on how to defeat the Russians, and so on. In the final analysis, no matter where you live, there is probably an AI that has been explicitly tasked by its operators with figuring out the most effective ways to do you in. Not because they've "gone wrong", but because they're working as intended.

The size of the universe is increasing...
but the rate at which it is increasing is decreasing...
but the rate at which the rate is decreasing is increasing...
but the rate at which the rate of the rate is increasing is decreasing...
but the rate at which the rate of the rate of the rate is decreasing in increasing...
[... and so on, ad infinitum...]

That one might surrender and meekly accept their likely death at the hands of the machine, I can believe. But all of them? This isn't the first such claim we've heard about AI danger.

I believe that Yudkowsky believes what he says he believes, for example. He says shut them all down worldwide, immediately, and if there are rogue nations who don't go along with it then launch air strikes on their data centres - even against a nuclear state. He said that three years ago. That's the thinking of a man who believes it's genuinely likely that the machines will kill us all in the near future.

And this is still considered extremist lunacy. That tells me the rest of them don't believe what he does.

The general public should care because there's apparently a good chance these machines are going to kill them all, very soon.

They don't care because they don't believe that.

When some AI agent of theirs, trying to meet a developer's benchmark test, eventually hacks the power grid or the water supply to a metropolis to help itself to a way around a sandbox environment, then the general public will care. And when that happens they may well drag AI industry CEOs and researchers out of their beds to the nearest lamp-post.

> approximately eight hundred billion ... oops, that's eight hundred million, of course. Ten percent probability, times the current population of the Earth. The expected consequences of allowing the AI industry to continue their work, according to people who are oddly calm about the prospect.

The actual post is behind a login wall, but I don't see from the BBC article that he's even advocating any action. Is he demanding the immediate shutdown of the data centres? Raids on the labs? Coordinated global action to impound and destroy the equipment? Life imprisonment for anybody found training large neural networks? Rogue researchers declared hostis humani generis? There's no mention of it.

That's what I'd be advocating at the very least. Then the question of whether to take direct action personally on top of that would be one of practicality - can I lay my hands on the right equipment, how can I do the most effective damage, do I have access to high value targets, will it be dramatic enough to frighten new recruits away from joining the industry, will others be inspired to follow my example, should I take the time first to form a team for more extensive action, that kind of thing. It wouldn't be a question of whether it should happen, only one of whether those are the right tactics. Yes, in all probability I'd decide I wasn't personally cut out for terrorism or assassination, but not everybody would decide so. It would be a reasonable choice in the circumstances.

If the expected bodycount from the operation of this industry over the next ten years was approximately eight hundred billion, then that would make AI researchers the single most dangerous group of human beings that has ever existed. The National Socialist German Workers' Party, previous title holders for the worst people ever, probably didn't kill even a tenth as many as this man says he expects AI firms to kill in roughly the same timescale. That is the comparison that would frame my thinking about AI researchers and what we should do with them, if I believed what he says he believes.

And I just don't see that kind of thing coming up in these conversations. People calmly and academically discuss their probability estimates as if it's a question of pure theory, not the imminent destruction of all humanity. If they really believed what they say, there would certainly by now be AI Luigis everywhere, data centres would be burning, and people would be quitting the industry all over the world for their own safety. That simply isn't happening. So I don't believe they actually do think that this outcome is likely. Actions speak louder than words, and their actions say they don't believe a word of it.

People commonly overestimate the possibility of "all humans dying", and simultaneously underestimate the possibility of "billions of humans dying".

People are like cockroaches that way; a disaster could wipe out 99% of us, and there would still be millions of us left to repopulate afterwards. To truly wipe out all humans would take something more severe than the asteroid that did for the dinosaurs.

Whoever set up its reward function, really. Could be any of the above. The technology doesn't care, it only follows its algorithm.

... then I would be demanding that every single person involved in the AI industry be rounded up and shot. All of them. My former boss, my coworkers, our competitors, all of them. I would be advocating a world in which is was worse to be found with LLM weights on your computer than to be caught with the worst possible kind of CSAM.

But apparently the idea that his former employer might literally kill literally everybody in just ten years' time isn't that big a deal for this guy. Go into the office one day and take a Second Amendment approach to as many of the frontier research team as possible? No... let's just whine about it on the internet!

Which is why I'm not at all convinced he believes this. If he believed that, this action is too small a response. Nothing short of Butlerian jihadism is an appropriate response, if you actually believe this.

I'm not. $2K is an eye-watering amount to spend on a cell phone, even in 2026, and especially on a design that seems almost guaranteed to start wearing a vertical line into the display at the folding-point over time. I expect a number of Slashdot articles about unhappy Duo users demanding redress over that issue, even if everything else works as advertised.

Maybe in a better economy, people would splurge on something like this, but I don't think that's the zeitgest this season.

n/t

Agreed, I use scala almost exclusively, the same things that make a language great for humans also make it great for AI, a compiled, type safe secure language gives you guarantees that python/javascript crap cannot.