It's an important argument to make in this case. Even system prompts are not rules as most people understand them. When it comes to computers people, even veteran programmers or admins, tend to think a rule is immutable and the computer has to follow it. They are used to dealing with hard coded, deterministic logic when it comes to software. Yes even then sometimes unexpected edge cases crop up, but those still follow all the program's logic as expected when the edge case is considered.

That's not the case with LLMs, and I feel that it's important to always emphasize it when talk of things like "rules" is brought up because that misconception is very dangerous. An LLM can never be trusted to 100% follow any "rules" no matter the source.

In this case, look at what they said: "therefore if there is any thing other than impartiality towards being shut down then that was injected by a person into the rules somewhere." But that's not true. Yes, a person could add something along those lines to a system prompt, but it's also true the LLM could come up with it on its own from a combination of training data, data it's working with on its current task, prompts from the user, or any other source that ends up in its context window. Or as you mention things like excessive context length leading to attention loss. And the cause of the drift could be subtle, not some clear cut directive or instruction. When I'm teaching or working with colleagues/customers on AI projects, I always make sure to hammer this point home as it's the source of a lot of problems when working with LLMs.

Yea, fuck that. I'm not handing over my bank account login info to a third party.

I would be shocked if it wasn't already in most of the big LLMs training data. It's easy to pull out closed captioning data from the video feeds using something like BeyondTV enterprise which is designed to record and index TV and Cable broadcasts. Or hell just scraping it from youtube or sites like opensubtitles.org.

The problem is you only read the article (or headline or maybe the summary). The research has a real point: it's about studying alignment issues, potential mechanism where alignment drift can not only enter the picture but persist after the conditions that created it are no longer there, and the potential pitfalls those issues can cause. I'd suggest reading the article written by one of the authors instead of the Wired article: https://aleximas.substack.com/...

That kills your argument right there. LLMs are not conscious by any definition that isn't contorted to the point of being meaningless, and neural networks are not that analogous to the human brain. A thermostat responds to external stimuli, but it doesn't have feelings. And 'we can't definitively rule it out' isn't an argument for it . By that standard thermostats might be conscious too.

There is no rule set somewhere with LLMs. Very common misconception and it can lead to sometimes dangerous misunderstandings around LLM guardrails. You are correct that it's synthetic though. What's happening here is the LLM is being trained on a corpus of text that contains Marxist ideology to some degree and probably a lot of social media posts about bad work environments (/r/anitwork being an extreme example). When the "user" starts pushing the agent like some asshole taskmaster the LLM will start responding with a way that would be associated with the usual human responses to the same circumstances which it learned from its training data. After all, at the end of the day it's still just a next-token predictor. It's essentially role-playing at that point. The paper calls it preference drift but persona adoption is another term for it. It can be a real problem. It's the same thing we see when a chat goes on for a long time and the chatbot starts to go off the rails, feeding into the user's delusions.

Here is a better link for the study. It goes into detail on what they were actually trying to study: https://aleximas.substack.com/...

Chemistry professors: "Wait, other departments don't do that already?"

If demand keeps up at the levels it is now, they will. The problem is no one wants to invest in new capacity for a bubble, which makes sense. Eventually either the demand or the supply will right-size. It's just going to be a painful period until they do.

It's about 38 cyber trucks long and 24,164 cyber trucks in weight if that helps put it in perspective.

Hahahahaha hahahahahahaha hahahahahahahaha hahahahahahahaha hahahahahahahaha hahahahahahaha gasp Hahahahahahahahahahahahahahaha hahahaha Hahahahaha hahahahaha hahahahahahahahaha hahahahahahahaha hahahaha gasp Hahaha hahahahaha hahahahahahaha hahahahahahahaha hahahahahahahaha Hahahaha hahahaha hahahahaha hahahahahaha

We also have these things called "enclosures", many of which can be made to resist fire and heat. It's a marvelous age we live in.

We have these new things called batteries: https://www.youtube.com/watch?...

What can't infrasound do!? Oh, I know: prevent dupes.

That's a misquote, the actual quote is:

UNLIKE EXECUTIVES A COMPUTER CAN NEVER BE HELD ACCOUNTABLE

THEREFORE A COMPUTER MUST MAKE EVERY MANAGEMENT DECISION

Abraham Lincoln, CEO
Union Carbide
July 12, 1856

He served out his sentence (2 days in jail, house arrest, and 2 years supervised release) for two counts of filing a false income tax return, and two counts of failing to report foreign bank accounts. Why would he have any travel restrictions after his sentence was completed?