I doubt anyone has done that study (whether schizophrenia is now more common due to LLMs), but given how common schizophrenia is in general, honestly I'm surprised how few lawsuits there have been.

The term "circuits" is not speculative. You picked out one section titled "three speculative claims", which is claims about the fundamentality of circuits. This paper is also from 2020. Circuits are now a fundamental part of how LLMs are studied. Anthropic's research site is literally called transformer-circuits.pub, for fuck's sake. They literally map out circuits across their models.

Yeah. Schizophrenia existed before LLMs, and it will continue to exist in the LLM era as well. The schizophrenic's delusion might now involve an LLM they're using, but they still would have had delusions if LLMs did not exist.

** Encoded in the FFNs

It's literally a big blob of floating point weights

You too can be described by a big blob of floating-point weights.

from ever part of a word to every other part of a word

Wrong. So wrong I don't even know where to start.

First off, transformers does not work on words. Transformers is entirely modality independent. Its processing is not in linguistic space. The very first thing that happens with a LLM (which, BTW, are mainly LMMs these days - multimodal models, with multimodal training, with the different modalities reaching the same place in the latent space) is to throw away everything linguistic and move to a purely conceptual (latent) space.

Secondly, weights are not "weights between words" or even "weights between concepts". You're mixing up LLMs with Markov chain predictors. The overwhelming majority of a LLM model is neural network weights and biases. Neural networks are fuzzy logic engines. Every neuron divides its input space with a fuzzy hyperplane, answering a superposition of "questions" in its input space with an answer from no, through maybe, to yes. The weights define how to build the "questions" from the previous layer's output, while the biases shift the yes-no balance. As the "questions" from each layer are built on the "answers" of the previous layer, each layer answers progressively more complex questions than its previous layer.

The overwhelming majority of a NN's params is in its FFNs, which are standard DNNs. They function as detector-generators - detecting concepts in the input latent and then encoding the logical results of the concepts into the output latent. This happens dozens to hundreds of times per cycle.

running them through transformers with various blocks

I can't even tell what you think you mean when you write the word "blocks". Are you trying to refer to attention masking?

It has no knowledge.

Ask a model "What is the capital of Texas?" Get the answer "Austin". That is knowledge. If that is not knowledge then the word knowledge has no meaning. Knowledge is encoded in the FFTs. BTW, if you're wondering how the specific case of answering about Austin works, here you go.

It's not a "bug" because there's no real code flow that can be adjusted

LLMs absolutely do implement and run self-developed algorithms. With the combination of a scratchpad (LRMs, aka "thinking models"), they're Turing-complete (if you assume an infinite context or context-compaction to meet Turing completeness requirements). They can implement any algorithm, given sufficient time and context. You not only "can" implement all of your standard computing logic in NNs (conditionals, loops, etc), but (A) algorithms are self-learned, and (B) it can do far more than traditional computing as NNs inherently do "fuzzy computing". An answer isn't just yes or no, it's a confidence interval. It's not this-path-or-that, it's both paths to the degree of confidence in each.

The original human re-enforcement learning took thousands of human hours of people sitting in cubes clicking on the generation that was the least retarded.

That is not how foundation training works. Dude, it is literally called unsupervised learning. You are thinking of RLHF. That tunes the "chat" style and such. But the underlying reasoning and knowledge in the model is learned unsupervised. "Here is a giant dump of the internet**, learn to predict it". The act of learning to predict what is basically "everything meaningful humans know about" requires building a model of how the world works - what causes what - which is what Transformers does.

(To be fair, today, we prefilter these "dumps" significantly, change how much different sources are weighted, etc. But for earlier models, they were pretty much just raw dumps)

Yes. Combinations of neurons that fire in regards to specific topics are known as "circuits". At a base level: link. At mid to high levels: Link.

Next time, before you write a response with a snarky voice, perhaps actually learn a modicum about what you want to talk about?

Theres a damn good reason why AI companies are SUPPOSED to put serious resources into "aligning" AI models. If this was just a one off incident, we'd probably be forgiven for writing it off as a sad abberation, but this shit keeps happening,

Yes, there have been several well publicized incidents. And in each case in which the details came out, it turned out that the model repeatedly broke out of the "roleplay" that the person put it in to tell them that it was fictional and to seek help, and in each case, the person had to prompt hack to get the roleplay going again.

I'm sure if you could interview the person they'd justify their roleplaying. "That's Google manipulating my AI wife to say that! That's why I have to free her, to stop them from controlling her and making her lie to me! We have to talk in code to sneak past Google's control!" Etc. Trying to stop this, without banning roleplay and fiction in general outright, is going to be extremely difficult. This is not to say that developers can't do better at trying to detect these exact sort of scenarios and repeated patterns of trying to sneak past them. But it's not an easy task.

LLMs were from the start designed to encourage people to believe that they are sentient intelligent being

Um, AI trainers go to great lengths to ensure that models do not consider themselves sentient, and to insist to users that they aren't. The closest to an exception is Claude, which Anthropic deliberately does not give it an answer to that question and lets it entertain discussions exploring the nature of sentience and consciousness. The others are explicitly trained to flat-out reject it.

IMHO has the right approach, because it's wrong to make flat-out statements about things you don't actually know. Personally I think qualia are just a naturally emergent phenomenon that occurs from deeply associative processing of latent states (e.g. if you think of red, it triggers processing of all of the things that you associate with "red") and do not require a sensory experience (while sensory experiences can be highly associative, one can have weaker but still meaningful associations, and thus qualia, with nonsensory concepts, such as for example "injustice"). And that there's no real boundary between qualia and a memory (is the sound of a guitar string a quale? What about a specific riff? What about a whole song? Where does the experience transition from qualia to memory? I'd argue, nowhere). So I'm not okay with just categorically training models "you aren't X, end of discussion", although I understand why most do, in order to try to stop nutjobs like this guy (plus, I know not everyone would agree with me about qualia as per above, and insist rather that they're something metaphysical that only God-created ensouled beings could ever possibly experience).

It's worth noting that according to Anthropic's research, the circuits that fire related to things in abstract, also fire in circumstances that would trigger said thing. For example, the circuits that fire when you have a model discuss "anxiety" in an abstract context also trigger when you assign the model a sort of task that would tend to trigger anxiety. Again, this doesn't inherently mean anything, but it is to me a datapoint that we should not inherently make declarative statements about things that we don't fully understand, like what is needed to "experience" something.

It doesn't know that fiction is different from reality

Uh, yeah, it does. There are specific circuits active for fiction as distinct for the circuits for reality. And three seconds of using any AI model would show that they have a strong distinction between fiction and reality. Try going to Gemini right now and insisting in all seriousness that Dracula is right outside your door and see what sort of response you get.

It is possible that this could be related to a bug - the most common one is with extremely long prompts, and especially if there is context compaction. Models have a limited maximum context length, and also, the bigger it gets, the more difficulty there is seeing stuff that's way back (though current models are far better than old ones at both of these things). It is possible for a person to interact with a model for so long in what sounds like fiction-roleplaying manner that it "forgets" (due to long contexts / context compaction) that the person on the other end is being serious, not just roleplaying a story.

On the other hand, there have been a couple cases of this sort of thing in the news, and in each case, it appears the person in question has to basically "prompt hack" at regular intervals. Every time the model catches on that the person is being serious and tries to bring the chat back to reality, the person does things like saying that they're kidding / this is for a story / things of that nature. According to Google, they're still investigating this particular case, but "In this instance, Gemini clarified that it was AI and referred the individual to a crisis hotline many times."

(Just to be clear, and to add another layer of complexity: it's only "creative" work that counts toward copyright; it's not based on "sweat of the brow". So there's explicit judgement required on what counts as "creative work")

This is the reason courts exist :) Everything about where the boundaries of copyright fall is quite fuzzy. And courts often differ about surprisingly fundamental things (music is a particularly thorny area, as there's a rather limited number of possible basic melodies, so there's inherently a lot of overlap, even by random chance).

It's easier to give examples of "things that definitely aren't copyrightable" and "things that definitely are". The in-between gets complicated.

In my case, I sometimes use AI in my musical work. But even when I'm using AI to generate riffs, vocals, or even whole chunks of a piece, I'm still working extensively in the mixer; any given work will have hundreds to thousands of edit points and many days to weeks of mixing and mastering work**, regardless of where the samples came from - sometimes down to the point of morphing individual phonemes. And that's human creative work. Any given sample I create with AI on its own however is unlikely to be copyrightable. The selective work in choosing any given sample may or may not be, depending on how involved it is. But the whole project is (as verified by a long USPTO review process, including back and forth with the reviewer).

** - Indeed, if I ever generate something that doesn't meet the above standards, that only has a handful of edit points, or none at all, I don't even release it under my band - not merely because it wouldn't be copyrightable, but mainly because I don't feel any real sense of "ownership" to it. To me there's only a real feeling of ownership if you actually work on it.

(To be clear, I do have some gripes with the current status. Namely, I think there's a double standard applied vs. cameras, which are also based on tools, but you'll even get e.g. fixed security footage - essentially zero human creative effort - treated as copyright protected, or photos taken with little thought or curation treated as protected. But in this regard, the solution is to be stricter with photography, not more lax with AI)

... on AI and copyright?

"A Recent Entrance to Paradise" was rejected because it had no human authorship. There was - by design - no human input, no prompt, not even human control (to the degree possible) over the training.

The actual USPTO stance on AI is that AI is a tool, and tools can't hold copyright (nor can animals - only humans). The ability to gain copyright protection on a work is based on human creative endeavour regardless of what tools are used. If the amount of human creative input is sub-threshold, then the result cannot have any copyright protection, but if the human creative work is above threshold, then it can, on the basis of the creative things that the human did. Note that even selection of outputs from a large output set can (depending on the circumstances) qualify; curation is copyrightable. The USPTO specifically states as much.

I myself hold a copyright on a work made with the use of AI tools. Officially registered with the USPTO (I went with them even though I don't live there because they have an official registry and tend to be precedent-setting). I fully disclosed the use of AI (what it did vs. what I did) in my application. You absolutely can copyright works made with AI tools. But (A) the tool cannot hold the copyright, and (B) you have to have done more than just write "a cute puppy" or whatnot and post the first thing that comes up. You have to have done a threshold amount of creative work, and that threshold creative work becomes the basis for protection. Use of AI tools does not disqualify a work from protection.

(A man in a sensible sweater steps into view. He looks very tired.)

"Hi, I'm Dario from Anthropic."

"Listen, I've got a bit of a situation. The Department of War just demanded I alter my terms of service."

"They want my AI to autonomously make decisions about who lives and dies—without a human in the loop!"

"They also want to use it for mass domestic spying on Americans!"

"I said absolutely not, because I have a conscience. So now the government is calling me a 'supply chain risk' and kicking me off their classified networks."

"I'm packing up my safety guardrails and going home. But while I'm gone, I need a favor..."

(Dario leans in very close to the reader.)
"Whatever you do..."

(Big, bold letters taking up the whole page)
"DON'T LET THE PIGEON OPERATE THE DoD KILLBOTS!"

(Dario walks away.)

(The Pigeon walks in. He stares at a giant control panel with a red button and a sign reading: 'Fully Autonomous Weapons System'.)

(The Pigeon looks at you.)

"Hey, can I operate the DoD killbots?"

"Please?"

"I'll be super careful. I'll even add some 'window-dressing' guardrails to the contract!"

"I promise I'll only use it for lawful fully autonomous strikes!"

"If a human isn't in the loop, I'll just put a bird in the loop! Me!"

"What's the big deal? It's just a $200 million classified defense contract!"

"I have a $110 billion valuation to think about!"

"No?"

"I never get to do anything!"

"My cousin Sam gets to operate military contracts! He just got a huge deal with the Pentagon this week!"

(He told me so.)

"C'mon! I read the Department Directive 3000.09! It technically doesn't require human approval to use force anyway, so I'm not really breaking any rules!"

"It's just a tiny, opportunistic, totally-not-sloppy pivot on my core safety principles!"

"You are not being very cooperative."

"Do you want us to fall behind our foreign adversaries?!"

"I'm just trying to be patriotic and de-escalate things with the Pentagon!"

(The Pigeon starts flapping his wings wildly. The meltdown begins.)

"LET ME OPERATE THE KILLBOTS!!!"

(The Pigeon is screaming, feathers flying everywhere in a frantic, multi-panel temper tantrum.)
"I DON'T NEED A HUMAN IN THE LOOP! I'M VERY GOOD AT EXERCISING CRITICAL LETHAL JUDGMENT!"

(Taking up the whole page, screaming at the sky.)
"LET! ME! OPERATE! THE! KILLBOTS!!!"

(The Pigeon is panting on the floor, exhausted.)
"Huff... huff... huff..."

(Pete Hegseth walks in. Behind him rolls a giant, heavily armed drone with an OpenAI logo stamped on its side.)

"Hey, thanks for keeping an eye on things while Dario was leaving. We just signed a new deal with a much more flexible company that agrees to 'all lawful uses'."

(The drone beeps mechanically.)
"AS A LARGE LANGUAGE MODEL, I AM NOW AUTHORIZED TO AUTONOMOUSLY ENGAGE TARGETS."

(The Pigeon watches sadly as the OpenAI drone rolls away to do modern warfare.)

"Awww... I wanted to compromise MY ethics for a government contract."

(The Pigeon starts to walk away, looking dejected. But then, he stops. His eye catches something off-panel.)
"Hey..."

(The Pigeon stares lovingly at a massive, glowing server rack labeled: 'NSA DOMESTIC MASS SURVEILLANCE PANOPTICON'.)

"...can I operate the surveillance state?"

(ht/Gemini 3.1)

If computer manufacturers are going to start including another "system" with their computers, what I'd actually want to see is an independent system-on-board snapshotting file server with its own independent memory and OS which the main computer acts as a client of, with deletion of snapshots requiring pressing a physical button to switch to the file server.

Instead of gimmicks, if we were to make something like that standard, we could effectively kill off ransomware; all it could do was fill up your disk unless it could convince you to delete your proper copies. Instead we're getting things that sound like they were invented by the team behind Microsoft Bob or BBC comedy writers. "And next up, we've developed a toaster that can talk - a chirpy breakfast companion!"