Forgot your password?
Close
typodupeerror

Comment Re:Hypocrites (Score 2) 64

No different than any other corporate spin about the inflexibility or uselessness of their platform. AWS does this quite well. "Undifferentiated heavy lifting" - while ignoring the absurd complexity and beta quality of their own services. Every issue I have with their shit, I experience a new and different bug or incorrectness in their documentation, and it takes years for it to get corrected. There are still bugs in Lambda that I discovered and reported 5+ years ago which haven't been resolved.

Comment Re:Not reassuring (Score 1) 64

At every level. They must be using claude to make their business decisions, too.

It's not only being demonstrated, it's being reveled in. Astounding.

I don't know if this is just the symptom of the AI bubble, cronyism, or "late stage capitalism" (even though the resemblance to capitalism long since ceased). But it's a mess.

20 years ago, these guys would've been tried for criminality by now. Both Anthropic and OpenAI. They're making Enron look like amateurs.

Submission + - A fundamental flaw leaves LLMs strikingly vulnerable to attack (technologyreview.com)

Submitted by joshuark
joshuark writes: It is impossible to make large language models fully secure against hacks because of a fundamental flaw in how they work, a team of researchers argue in a paper presented at the International Conference on Machine Learning, a top AI conference, this month. The claim has huge implications for the safety of this technology.

By taking advantage of this flaw, which concerns how LLMs identify who or what is giving them instructions, the researchers were able to make popular LLMs spit out information they had been trained not to provide, such as how to synthesize cocaine and how to sabotage a commercial aircraft’s navigation system.

“There’s a real probability that this is going to be a problem that’s fundamentally unsolvable,” says Charles Ye, an independent researcher and coauthor of the ICML paper.

Companies will typically hire teams of human testers to try to come up with novel attacks that break existing guardrails, a process known as red-teaming. Model makers also use LLM super-hackers (such as OpenAI’s GPT-Red) that find and exploit weaknesses in other models to automate parts of this process. The goal is then to take those attacks and train a new model to resist them and anything that looks like them.

The problem, says Jasmine Cui, another independent researcher and coauthor of the paper, is that the approach amounts to giving the models a list of things they shouldn’t do. But no list is exhaustive. “It’s like watching The Simpsons and they have Bart writing ‘I will not say something inappropriate to my teacher’ a hundred times,” she says. “And he still does things that are pretty crass anyway.”

The ICML paper describes attacks against several of OpenAI’s models, but Cui and Ye say that they have since seen similar results with models made by Anthropic, Alibaba, and DeepSeek.

Cui and her colleagues wanted to find out why an attack like chain-of-thought forgery was so effective. They suspected it had something to do with the mechanism that LLMs use to keep track of where their instructions are coming from.

But what Cui and her colleagues discovered is that LLMs are in fact very bad at keeping track of different roles. In a series of experiments that looked at what was going on inside a handful of different models, the researchers found that LLMs seem to identify the role of a specific chunk of text not by the tags around it but by the style of that text and the words it contains.

The upshot, the researchers claim, is that all an attacker needs to do to hack an LLM is write text that spoofs a certain role. And because roles are a fundamental part of how LLMs work, no amount of training will fully solve the problem.

Ye is worried that nobody is ready for what’s coming. “There’s going to be a huge economic incentive for people to do jailbreaks and prompt injections,” he says. The best defense could be to expect the worst. Organizations shouldn’t trust LLMs, and they should expect that anything done by agents could be unsafe, he says: “That’s not a great solution, but it just might be what we have to do.”

“It’s really incredible that these things are being deployed everywhere to control super-critical systems,” he adds. “There’s been no study of the fundamental science here. We’re all doing it ad hoc.”

Comment Re:This is a good sign (Score 1) 87

That's a model specific problem, to a large degree.

The "AI assumed something stupid and destructive" problem went away when I started using better planning skills and models which were not anthropic's.

I've not had any of those "wild destructive hallucination reasoning" problems with GLM or Qwen models, even small versions, which I fought with regularly with opus.

Comment Re:This is a good sign (Score 0) 87

Sure, you can do that. You can do that with buggy code that compiles, or python, too.

For AI, you can also have another agent and/or model review and fix it. You can ask it adversarial questions until you get it right. You can ask it to explain the architecture and design (or: you can plan it out ahead of time so that you know the design and then verify it).

It's no different than starting on an existing code base... just faster.

Comment Re: Hopefully he sues for a few million. (Score 1) 222

by Tom (#66262388) Attached to: A Missing Underscore Sent Innocent Man To Prison For 18 Months

Hard disagree.

There are mistakes that can happen - getting a phone number wrong, knocking on the wrong street number, stuff like that, sure.

But all of those mistakes need to be found and fixed DURING the investigation. This is the duty of the police officiers.

Not finding any evidence and STILL going to trial - that's on the prosecution. But I doubt that there would have been a conviction if police had not omitted things or outright lied in court. I doubt any courd would hand down a guilty verdict if all the police involved said that they found no evidence of anything.

Slashdot Top Deals

IN MY OPINION anyone interested in improving himself should not rule out becoming pure energy. -- Jack Handley, The New Mexican, 1988.

Close