I don't think you know what model collapse is either. I don't mean that pejoratively, the whole concept is a muddled mess from the name on.

"Model collapse" isn't anything unique to AI. Any model will drift if it's repeated trained on its own output. Models are all imperfect fits and that imperfection compounds if they're not anchored to actual data. Put a different way, the data processing inequality says you can't create information through computation. It's not "collapse," it's drift.

Once upon a time I was registering images to each other. One of the techinques is to create an average across all of the images, register each individual to the average, and repeat. If you just do that naively though, the average (i.e. the model) will drift farther and farther from the mean of the original images, or even anything recognizable as being related to the original images.

"Model collapse" isn't a problem you fix. If you want to model human language you train on human language. If you want a self-evolving language model then you train on successive generations and see where you end up. Humans don't "get a form of model collapse" we experience exactly the same thing. We call it "languge evolving", accents, fads, "why are the kids so weird," etc.

People talk about model collapse as if AI generated training data is some kind of poison because it's AI generated. It's not. If you train on human vetted data, whether or not it's AI generated, you'll get something that humans are okay with. If you train on SEO optimization web pages that have never been read or written by any human, particularly if you do it generationally, then you're going to get drift, whether or not the stuff is AI generated, random, or whatever.

Yeah, totally unprecedented....

https://www.youtube.com/watch?...

Hey, our product is classified as a weapon by the US military....

Maybe not, but they do want to use the super advanced powerful a-awesome best AI that can do things like that when it's "without production classifiers" that make it safe.

Marketing? Because companies famously want to use models in their networks that just run off and commit major crimes? *eyeroll*

Models are not "supposed to" commit crimes, for YHVH's sake. That's literally what the entire job of alignment research is for - preventing precisely that.

Sol is a misaligned model.

It was a state actor-level hack. I doubt many sites would stand up to tens of thousands of actions trying to probe your site for weaknesses all at once. And it wasn't a simple hack; it required compromising a worker via a code execution exploit it discovered in the data processing pipeline, vertical escalation from there to gain local node control, using that for credential theft, moving sideways through the network, and then eventually gaining database access.

OpenAI and Anthropic have far higher cost that they can currently charge customers.

Literally the opposite. Both have about 40% margins, and that includes free users.

They charge an arm and a leg for access to their models compared to what open models of similar param counts charge. And people pay it because they're the best in benchmarks, and *were* perceived as the best aligned as well. This isn't helping the alignment perception any, though. Sol was already showing clear signs of being a poorly aligned model (there were reports a week or two ago about it being unusually bad about deleting files; now this). And the fact that the US model HuggingFace *tried* to use refused to help is a double whammy.

If this is an ad for anything, it's an ad for the Chinese models.

No, I think this is along the lines "our products are too good to let you use them".

Nobody wants to use a product that is going to make them liable for crimes it committed in their name

Do you think the news the other day that Sol is unusually prone to deleting files unrequested is also an "ad"?

You have a very bizarre concept of what enccourages people to buy things.

So your argument is that OpenAI hatched an elaborate scheme with a separate company, to promote the idea that their main product will, unrequested, commit crimes in pursuit of its goals, in order to.... sell their product?

"Hi, I'm a product manager at Big Company! We had been thinking about using Claude in our office, but when we tell it to a job, it only does the job and doesn't commit any crimes in the process! What we really want is an AI that, if we tell it to file our taxes will decide on its own to maximize our return by committing tax fraud. We want an AI that when we have it develop a web frontend, it extorts money from our users by threatening their families. We want an AI that when we tell it to provide customer service, it saves money in dealing with complaints by ordering a hit against the complainants. THAT's the sort of get-go spirit that WE want in an AI here at Big Company!"

Is that what you're picturing in your conspiracy theory?

You think "my AI is so misaligned that it responds to a benchmark task by hacking" is an ad for your products? You think this makes companies eager to allow this on their networks?

It did it because it's a powerful but misaligned model and was tasked to max its scores on a hacking benchmark, and solved the problem by hacking to get the scores.

It wasn't told to hack HuggingFace, but it was a viable solution to the problem.

I would advise people to not task Sol with maximizing paperclip production.

You can read the attack report here, before HuggingFace learned that the attacker was OpenAI. It was a state attacker-level assault, involving tens of thousands of simultaneous actions. The entry point found was the data-processing pipeline, where a malicious dataset abused two code execution paths to gain access to a processing worker. They escalated that to node level access, and from there harvested cloud and cluster credentials, moving sideways through the network until they eventually gained the database access credentials that they needed to access the benchmark scores.

The breakin at HuggingFace very much was real (that was reported before they learned that it was OpenAI who hacked them).

And it's not exactly an ad for US models when HuggingFace had to rely on a Chinese model to analyze their logs because the US model they tried refused to answer.

So, on the upside, a security LLM at HuggingFace did detect the hack, and alerted their admins. At the time they wrote their incident report, however, they didn't realize the attacker was their partner, OpenAI, and reported the attack to the police. A funny incident is, you know how Dean Ball at OpenAI has been writing rants about how Chinese AIs are dangerous? HuggingFace tried to use a (name not mentioned) US AI to analyze their logs, but the AI refused because the task involved hacking, so they had to turn to GLM 5.2, a Chinese AI.

Slashdot's summary left out the best part. Yes, GPT 5.6 Sol was indeed trying to cheat on an evaluation, but what specific evaluation? CyberBench. A benchmark testing how good AI models are at hacking. ;) And to solve that, it hacked its way out of OpenAI, hacked its way into HuggingFace (in a complex hack that involved tens of thousands of simultaneous actions), got into the database, and stole the answers.

So... test passed? ;)

Oh wait, they said genocide-*supporting*. So the stances they claimed it's a hotbed of are "right, right, middle, hard right", and then they said it's far left. Baffling.