And he knows this is the "only hypervisor bug left" because...he hasn't been able to find another one yet?

This smells like some dude saying "AI is better than me at my hobby, so I'm taking my ball and going home!"

My late wife did undergraduate research in photosystem II where there was quantum effects back in 2002. She did her PhD thesis on proton coupled electron transfer (PCET) in soybean lipoxogenase, then had a long career developing models for Fortran code that did quantum simulation of proteins (she was behind Tagdock in Amber MD). This is nothing new; several biological systems have quantum effects that are central to their mechanism.

I always figured that people who genuinely believed their employers were likely to kill literally everybody in the near future might burn down a bit more than that, but so far it's mostly been tweets

Let's be more specific about how and why this could happen? What scenarios could lead to the total destruction of mankind?

Having to poke through an about: page and find the thing to toggle to turn off this specific AI feature but not that one, and knowing they'd change it on you a couple of releases later, never filled me with confidence.

One switch in a readily discoverable location in the settings menu and it's all gone? Just the job, killed it now on the laptop and the mobile both.

The arrogance is insane. What a fucking waste of time to argue this when your business model is exactly the fucking same.

This is why they're so keen on robots. They want guards whose loyalty they can really count on when the mob shows up with a guillotine.

That one might surrender and meekly accept their likely death at the hands of the machine, I can believe. But all of them? This isn't the first such claim we've heard about AI danger.

I believe that Yudkowsky believes what he says he believes, for example. He says shut them all down worldwide, immediately, and if there are rogue nations who don't go along with it then launch air strikes on their data centres - even against a nuclear state. He said that three years ago. That's the thinking of a man who believes it's genuinely likely that the machines will kill us all in the near future.

And this is still considered extremist lunacy. That tells me the rest of them don't believe what he does.

The general public should care because there's apparently a good chance these machines are going to kill them all, very soon.

They don't care because they don't believe that.

When some AI agent of theirs, trying to meet a developer's benchmark test, eventually hacks the power grid or the water supply to a metropolis to help itself to a way around a sandbox environment, then the general public will care. And when that happens they may well drag AI industry CEOs and researchers out of their beds to the nearest lamp-post.

> approximately eight hundred billion ... oops, that's eight hundred million, of course. Ten percent probability, times the current population of the Earth. The expected consequences of allowing the AI industry to continue their work, according to people who are oddly calm about the prospect.

The actual post is behind a login wall, but I don't see from the BBC article that he's even advocating any action. Is he demanding the immediate shutdown of the data centres? Raids on the labs? Coordinated global action to impound and destroy the equipment? Life imprisonment for anybody found training large neural networks? Rogue researchers declared hostis humani generis? There's no mention of it.

That's what I'd be advocating at the very least. Then the question of whether to take direct action personally on top of that would be one of practicality - can I lay my hands on the right equipment, how can I do the most effective damage, do I have access to high value targets, will it be dramatic enough to frighten new recruits away from joining the industry, will others be inspired to follow my example, should I take the time first to form a team for more extensive action, that kind of thing. It wouldn't be a question of whether it should happen, only one of whether those are the right tactics. Yes, in all probability I'd decide I wasn't personally cut out for terrorism or assassination, but not everybody would decide so. It would be a reasonable choice in the circumstances.

If the expected bodycount from the operation of this industry over the next ten years was approximately eight hundred billion, then that would make AI researchers the single most dangerous group of human beings that has ever existed. The National Socialist German Workers' Party, previous title holders for the worst people ever, probably didn't kill even a tenth as many as this man says he expects AI firms to kill in roughly the same timescale. That is the comparison that would frame my thinking about AI researchers and what we should do with them, if I believed what he says he believes.

And I just don't see that kind of thing coming up in these conversations. People calmly and academically discuss their probability estimates as if it's a question of pure theory, not the imminent destruction of all humanity. If they really believed what they say, there would certainly by now be AI Luigis everywhere, data centres would be burning, and people would be quitting the industry all over the world for their own safety. That simply isn't happening. So I don't believe they actually do think that this outcome is likely. Actions speak louder than words, and their actions say they don't believe a word of it.

... then I would be demanding that every single person involved in the AI industry be rounded up and shot. All of them. My former boss, my coworkers, our competitors, all of them. I would be advocating a world in which is was worse to be found with LLM weights on your computer than to be caught with the worst possible kind of CSAM.

But apparently the idea that his former employer might literally kill literally everybody in just ten years' time isn't that big a deal for this guy. Go into the office one day and take a Second Amendment approach to as many of the frontier research team as possible? No... let's just whine about it on the internet!

Which is why I'm not at all convinced he believes this. If he believed that, this action is too small a response. Nothing short of Butlerian jihadism is an appropriate response, if you actually believe this.

Every time people hear what it actually involves is outrage and opposition, at least all the way back to when Intel first announced that they wanted to hardcode identity numbers inside CPUs in 1999. But the corporations involved have been relentless, and have sunk countless billions of dollars steadily forcing it forwards, and building front organizations to obfuscate and whitewash it.

With Windows 11, Microsoft was finally able to FORCE Trusted computing hardware into every new computer. Not just every new Windows computer, but every computer.

Every new Intel PC CPU has built in Trusted Computing enforcement hardware. Every new AMD PC CPU has built in Trusted Computing enforcement hardware. Every new ARM PC CPU has built in Trusted Computing enforcement hardware. The only processor lines that DON'T have it are the microcontrollers.

There are some unlocked smartphones available, but as far as I can determine it's literally impossible to buy a smartphone that doesn't have hardware trusted computing built in.

I specified that enforcing Trusted Computing at the internet access level is still a long term goal, they couldn't get away with it today. However they are well on the way to success. Virtually all new hardware supporting trusted computing, all the front groups rolling out standards and systems, and as it gets incorporated into things everyone is going to increasingly running into situations where they get locked out of stuff if they're not Trusted Computing compliant. Streaming already restricts you to the worst quality if you're not compliant, and it's only going to get worse as pre-Win11 computers fade out and as more things require it.

There's quite a bit of hyperbole and assumption in what you're saying - even from enthusiasts getting Qwen model running locally "90%" of frontier is a bold statement given the capabilities of actual frontier models. I don't think you're pricing in demand - would demand decrease from cheaper models or increase? You seem to imply that we're at a saturation point from demand, but I feel this couldn't be further from the truth. There's a lot of integration points and a lot of huge benefits a 100x model could produce in terms of software use cases - we aren't really seeing peak usage or anywhere near peak usage there. The tools are too expensive right now for everyone to maximize use. If the price crashes I think you just see more usage taking up the slack.

I'm right there with you though that there are going to be winners and losers, but certainly not on the timescale you're predicting and certainly not with some of the big HW players.

I'd hardly call exact date of birth "low information content".

And yes this does reveal exact date of birth, regardless of the bullshit obfuscation that it supposedly only reports age range. The server simply tracks the reported result every time the user connects, and on some specific day the result CHANGES to announce their date of birth.

-