Forgot your password?
Close
typodupeerror

Comment Re: No it was not (Score 1) 45

by drinkypoo (#66266042) Attached to: Catastrophic MoD Data Breach Caused By Lack of Training On Excel

Actually, it's exactly the same issue. Just using a different program.

It's not the same issue.

The pdf redaction folks drew black bars over extant text and hid it that way.
The excel chap clicked on "hide this tab".

There is no evidence that anyone was trying to redact anything when they hid the tab, they may have just been trying not to confuse the recipient. The person who hid the sheet is not the person who shared the document. Any similarity which did exist would be irrelevant because format shifting to a PDF would still solve this problem.

Comment Re:nothing's impossible (Score 2) 64

by drinkypoo (#66265978) Attached to: A Fundamental Flaw Leaves LLMs Strikingly Vulnerable To Attack

The example might have been chosen because synthsizing cocaine from raw materials is hard, so revealing a process that begins with "obtain illegal drugs" is unlikely to do harm.

In many cases, coca leaf is not illegal. You can buy coca tea in Peru. People frequently bring it back into this country because it just looks like tea. It's a reasonably mild stimulant, similar in effect to smoking a cigar. I understand that you need quite a bit to make any significant amount of cocaine, but I've never tried so I wouldn't really know.

Comment Is it a Master or a Copy? (Score 4, Insightful) 37

The way this is being described is asinine. You lose a master and you're fucked, lose a copy and you still have the master. It's like words don't mean anything to these people. Additionally, fuck the studio for not encrypting it or exercising any sort of security on it at all.

Pretty much everyone in this story is a moron

Comment tldr but human brain has same vulnerability (Score 1) 64

by mattr (#66265446) Attached to: A Fundamental Flaw Leaves LLMs Strikingly Vulnerable To Attack

tldr but the two examples are pretty dumb as anyone in those fields could doubtless figure it out. If you give a model sufficient data, even if it is just axioms, understanding of basic physics and chemistry, has holes, etc., it will be able to fill in the gaps or tell itself a story or roundabout logic walk that gets there if possible. As far as not being able to tell where instructions come from, the rules they try to implement are flimsy and less grounded than the massively interconnected data they have. You would have to not tell them about the concepts of drugs, poison, sabotage, aircraft, etc. and not let anyone tell them about it. It just isn't feasible. Humans have even less internal security, which is why media brainwashing works. In fact you can get incensed about something that is fake and even after looking up to see if it was real or not, still be upset about the hypothetical situation. And people can love AI created "animal came to my door looking for help saving its mom" videos, even if they know it's AI generated the response is "I don't care, it's so cute!" It's also why willingful suspension of disbelieve and works of fiction sell. I do have an idea about a solution (not enough space to write in the margin here, lol) but it is not going to depend on a single LLM policing itself within the turn.

Comment Re: No it was not (Score 1) 45

by drinkypoo (#66265340) Attached to: Catastrophic MoD Data Breach Caused By Lack of Training On Excel

That's what I was talking about in another thread in this discussion when I said PDFs have their own issues but they wouldn't apply here. That's a problem when redacting PDFs, WHICH NO ONE SHOULD EVER DO. They should either go back to the source document, redact that, and create a new PDF; Or they should create an image from the PDF, redact that, and then they can publish that as a new PDF.

Comment Re: I was wondering when FOSS projects would notic (Score 1) 96

by drinkypoo (#66265192) Attached to: GCC Adopts Policy Rejecting Significant AI-Generated Code

What LLMs do can not reasonably be called learning, or even frankly training, though the latter has unfortunately stuck.

On the other hand it's also not storing the original data (although the training corpus is that) but rather data about the data.

However, the standard for a derivative work is that it contains substantial recognizable elements, not that it's a 1:1 copy, so that distinction may be irrelevant, or at least not mean what you think it means.

Comment Re: Copyright (Score 1) 96

by drinkypoo (#66265172) Attached to: GCC Adopts Policy Rejecting Significant AI-Generated Code

"I'd advise you you to shut the fuck up and work on a project that's now possible where before you needed a whole team."

Why, so they can replace me with someone cheaper next week? Or do you mean I should create some project in hope of a buyout when the organization that could buy it from me can just prompt their own version without me?

Comment Re: Are they using the same call center (Score 2) 26

"There's actually plenty of examples on YouTube of Tesla's FSD avoiding collisions"

And there's actual court cases about Tesla's FSD mistaking the side of a trailer for the sky and driving under it, or going full speed towards a barrier and then disabling itself a fraction of a second before the collision so they can claim FSD didn't cause the accident because it wasn't in control at the time. Well guess what, neither was anyone else, and Tesla set them up for that.

Submission + - A fundamental flaw leaves LLMs strikingly vulnerable to attack (technologyreview.com)

Submitted by joshuark
joshuark writes: It is impossible to make large language models fully secure against hacks because of a fundamental flaw in how they work, a team of researchers argue in a paper presented at the International Conference on Machine Learning, a top AI conference, this month. The claim has huge implications for the safety of this technology.

By taking advantage of this flaw, which concerns how LLMs identify who or what is giving them instructions, the researchers were able to make popular LLMs spit out information they had been trained not to provide, such as how to synthesize cocaine and how to sabotage a commercial aircraft’s navigation system.

“There’s a real probability that this is going to be a problem that’s fundamentally unsolvable,” says Charles Ye, an independent researcher and coauthor of the ICML paper.

Companies will typically hire teams of human testers to try to come up with novel attacks that break existing guardrails, a process known as red-teaming. Model makers also use LLM super-hackers (such as OpenAI’s GPT-Red) that find and exploit weaknesses in other models to automate parts of this process. The goal is then to take those attacks and train a new model to resist them and anything that looks like them.

The problem, says Jasmine Cui, another independent researcher and coauthor of the paper, is that the approach amounts to giving the models a list of things they shouldn’t do. But no list is exhaustive. “It’s like watching The Simpsons and they have Bart writing ‘I will not say something inappropriate to my teacher’ a hundred times,” she says. “And he still does things that are pretty crass anyway.”

The ICML paper describes attacks against several of OpenAI’s models, but Cui and Ye say that they have since seen similar results with models made by Anthropic, Alibaba, and DeepSeek.

Cui and her colleagues wanted to find out why an attack like chain-of-thought forgery was so effective. They suspected it had something to do with the mechanism that LLMs use to keep track of where their instructions are coming from.

But what Cui and her colleagues discovered is that LLMs are in fact very bad at keeping track of different roles. In a series of experiments that looked at what was going on inside a handful of different models, the researchers found that LLMs seem to identify the role of a specific chunk of text not by the tags around it but by the style of that text and the words it contains.

The upshot, the researchers claim, is that all an attacker needs to do to hack an LLM is write text that spoofs a certain role. And because roles are a fundamental part of how LLMs work, no amount of training will fully solve the problem.

Ye is worried that nobody is ready for what’s coming. “There’s going to be a huge economic incentive for people to do jailbreaks and prompt injections,” he says. The best defense could be to expect the worst. Organizations shouldn’t trust LLMs, and they should expect that anything done by agents could be unsafe, he says: “That’s not a great solution, but it just might be what we have to do.”

“It’s really incredible that these things are being deployed everywhere to control super-critical systems,” he adds. “There’s been no study of the fundamental science here. We’re all doing it ad hoc.”

Slashdot Top Deals

"A great many people think they are thinking when they are merely rearranging their prejudices." -- William James

Close