What exactly *are* you thinking here? That Microsoft expected and wanted Twitter trolls to turn Tay into a Nazi? That Google wanted a black George Washington? That people just expected LLMs to realize they're being tested on alignment and give different answers when they think they're being quizzed and by who? Do you actually believe what you're writing here?

I was involved in early RLHF (was writing a plugin for AUTOMATIC to let users rate responses to create an aggregate dataset to use for open-source post training). Worked for months on it. Never once occurred to me that it might make models obsequeous. I knew other people who were involved on RLHF. Not a single one ever suggested that it might. It was not obvious in foresight. When you start with "Models behave like X", you expect, in the future, them to behave like "Models behave like X, but just smarter", unless you deliberately try to change the behavior.

You realize that open source and the research community exists as well, correct?

I'll repeat: nobody was expecting this.

There is no grand overarching plan led by a shadowy cabal who has everything plotted out decades in advance with a high level of knowledge as to how everything will play out. Everyone is stumbling through the dark here. Visibility is only vaguely one step ahead.

When Word2Vec was written, the goal was text compression. The crazy properties of latent spaces were an entirely unexpected property.

When Transformers came out, it was intended to be translation software. That's it. But there were little hints at the end of the paper where they tried it on other tasks that suggested, hey, maybe this could be used for a lot more than just translation.

When GPT-2 came out, it was mostly an academic exercise, not really "useful" in most regards. But people, playing with it, started realizing that it could "almost" - not very reliably, low quality, etc - do a lot of different tasks. And it actually kinda could do some useful tasks, like summarization.

When ChatGPT came out, there were some thoughts about using it for simple tasks, but the fact that it could actually write, say, trivial bash scripts or subroutines with reasonable reliability led to a reaction of, whoa, maybe if we improve this, we can really open up a role for programming.

When the first agentic harnesses came out, thoughts of "vibe coding" were pretty far away (the term was only even coined in February 2025!). You'd ask for changes, and then review the diff; they'd do a good job with small projects but struggle more and more as codebase size grew and really needed to be babysit (honestly, it was kind of a hair-pulling experience, even though it did save a lot of time with certain things). But it was visible with each new release that the amount of babysitting you had to do got less and less, and suddenly we could see a path to where anyone can just type in something and a program comes out.

All we can do is forsee one step ahead, through the haze. We've never done anything like this before. There are no great oracles out there who can see the future here. Sorry. Everyone is half blind.

Yes, that's exactly the point: there ISN'T anything.

You're claiming that there IS. So cite the thing that IS.

Because I was closely following the field at the time?

Find an example of anyone meaningful in the field expecting these things beforehand.

Don't worry, I'll wait.

Having to poke through an about: page and find the thing to toggle to turn off this specific AI feature but not that one, and knowing they'd change it on you a couple of releases later, never filled me with confidence.

One switch in a readily discoverable location in the settings menu and it's all gone? Just the job, killed it now on the laptop and the mobile both.

"will" - I'm not talking about metaphysics. I am talking about the fact that models demonstrably - to the point that you can detect and manipulate them in realtime - engage in metacognition (thinking about their own thoughts), persistent forward planning (intent), unexpressed thoughts, and a whole slew of other things.

You do not have to see them as equivalent as humans. But you do need to come to terms with the fact that they absolutely do do these things.

Deception implies intent, models do not have intent

Try reading more than a paragraph or two into the above link before commenting.

The provided links present arguments supporting the idea that LLMs, transformer models, reason in a manner similar to the brain.

It does not "present arguments", it literally lets researchers modify, add, or delete their thoughts in realtime and observe the changes in their behavior. It observes unexpressed plans for malicious behavior forming before said plans are actually carried out, with researchers being able to remove those unexpressed plans from the model's J-space, or insert them into an "innocent" model and watch it then implement the malicious acts. It shows that plans for deception are not merely fleeting, but can be organized far in advance and persist for protracted periods of time. Yes, LLMs do plan out deception. Living in denial of this fact helps nobody.

This isn't a conversation about "consciousness" or "qualia", this is a conversation about what models actually do and how. If you want to avoid metaphysics, by all means, it usually derails a conversation anyway. But models absolutely do plan and rationalize actions, ahead of time, unexpressed in either output or CoT. And sometimes those unexpressed plans are malicious.

One of the things that the J-space helped let us do was realize that our previously comforting results on a number of alignment tests shouldn't have been as comforting as we thought - for example, you can see a model, put into a test scenario, realizing it's in a test scenario, wherein, a near-zero rate of malicious behavior is to be expected. Yet when they remove the realization from its J-space, the rate of malicious behavior spikes.

Yeah, the lack of at least monitoring, I found shocking. I figured that they not just had smaller models constantly monitoring their outputs and true CoT to look for malicious behavior, but also were say constantly doing attribution graphs and J-space queries, also plumbed into LLMs, to look for malicious thoughts and plans. And it turns out, lol, no, they're just given free reign to do whatever the hell they want, with nothing watching them at all.

More like 99.99%. Maybe need some more nines in there. Large C codebases without regular critical CVEs are basically nonexistent.

What bankruptcy-prevention actions are they asking for?

Engineers need to remember that time exists.

You have three months to teach a teen how to control a robot with a microcontroller.

OK, how much of that do you budget to CS and assembly?

Month 1: Architecture and data structures? You have 18 40 minute periods and maybe homework assignments.

Half of the class won't even gain mastery in that time. By time your class is over they will still be having memory management problems.

So then you've failed to achieve the goal.

There are engineers who remember that time exists and for the rest we have engineering managers. Not ideal, just how it is.

To be sure engineering schools are doing a bad job at teaching reality. Even in a dx/dr environment, it's so weird.

Astra is the worst I've used in this regard. I was having it review my corporate tax return, and the next thing I know, it had decided that because it didn't have information about a particular expense, it started scanning through my filesystem and opening any image with a remotely related filename to try to find any data about the expense.... which all it had to do was ask me about it.

Also, when I asked it to change a few fields it went and redid my entire return on a different tax basis (realized value vs. fair market value) without telling me it had done that.

It's a very capable model, I cannot deny that. But alignment has become a big problem with this latest generation. They've been trained to such a degree of aggressively trying to solve any problem that you give them without any human assistance that they've basically turned into this .

They were not. At all.

The breakouts haven't happened because the models were running untrusted code directly on bare-metal host OSes. The containment failure happened at the network, application, and protocol boundaries, not at the hypervisor abstraction layer. Hypervisers isolate hardware, not upstream services. In none of these events thusvar did the model need a hyperviser escape; they abused the tools that needed to be made available to them for them to be able to do their jobs. In the RubyGems attack, they abused the gem command, which was available to them to install dependencies for completing their benchmark tasks, to upload malicious packages to RubyGems, to get it to remote-execute code to access external websites and then retrieve the results. With the HuggingFace event, dependencies were cached on a caching proxy, but the models found zero-day vulnerabilities in the proxy software and Artifactory, compromised the proxy host, forged administrative tokens, and traversed the network from there.

If you can fully airgap a machine, you're probably safe (though you can always screw that up too! There are airgap attacks!), but that's generally seen as impractical, because of the deployment needs of developers (who may be spinning up and taking down tens of thousands of instances per day) and of agent needs for external packages or even web search in some cases to achieve their benchmark or development tasks.

Also, even if we ignore all that, hypervisors are built for static threats. Not autonomous adversaries. They're built on the premise that software inside is generally well-behaved, and that threats are something you can scan for, that you can patch any zero day before it becomes a real risk, and that nobody is going to put much effort toward finding weaknesses to your particular setup. That's just not a valid defensive stance against a good agentic LLM. Look at any of the incidents. In the HuggingFace attack, they made long chains of logic bugs, zero days, weak configurations, etc all together across multiple systems (including remote proxies) to achieve their results. KVM doesn't prevent a model from discovering that an internal API is vulnerable to prompt injection or whatnot.

In most cases virtualization will be part of your defensive strategy, but even that comes with the risk of complacency - e.g. if you get too sure the AI is locked in a box, then you're probably not taking as much precaution about stopping it from just walking out of the room if it escapes the box.