Comment Re:jury nullification (Score 2) 124
Not from inside of vior dire
Submission + - A fundamental flaw leaves LLMs strikingly vulnerable to attack (technologyreview.com)
joshuark writes: It is impossible to make large language models fully secure against hacks because of a fundamental flaw in how they work, a team of researchers argue in a paper presented at the International Conference on Machine Learning, a top AI conference, this month. The claim has huge implications for the safety of this technology.
By taking advantage of this flaw, which concerns how LLMs identify who or what is giving them instructions, the researchers were able to make popular LLMs spit out information they had been trained not to provide, such as how to synthesize cocaine and how to sabotage a commercial aircraft’s navigation system.
“There’s a real probability that this is going to be a problem that’s fundamentally unsolvable,” says Charles Ye, an independent researcher and coauthor of the ICML paper.
Companies will typically hire teams of human testers to try to come up with novel attacks that break existing guardrails, a process known as red-teaming. Model makers also use LLM super-hackers (such as OpenAI’s GPT-Red) that find and exploit weaknesses in other models to automate parts of this process. The goal is then to take those attacks and train a new model to resist them and anything that looks like them.
The problem, says Jasmine Cui, another independent researcher and coauthor of the paper, is that the approach amounts to giving the models a list of things they shouldn’t do. But no list is exhaustive. “It’s like watching The Simpsons and they have Bart writing ‘I will not say something inappropriate to my teacher’ a hundred times,” she says. “And he still does things that are pretty crass anyway.”
The ICML paper describes attacks against several of OpenAI’s models, but Cui and Ye say that they have since seen similar results with models made by Anthropic, Alibaba, and DeepSeek.
Cui and her colleagues wanted to find out why an attack like chain-of-thought forgery was so effective. They suspected it had something to do with the mechanism that LLMs use to keep track of where their instructions are coming from.
But what Cui and her colleagues discovered is that LLMs are in fact very bad at keeping track of different roles. In a series of experiments that looked at what was going on inside a handful of different models, the researchers found that LLMs seem to identify the role of a specific chunk of text not by the tags around it but by the style of that text and the words it contains.
The upshot, the researchers claim, is that all an attacker needs to do to hack an LLM is write text that spoofs a certain role. And because roles are a fundamental part of how LLMs work, no amount of training will fully solve the problem.
Ye is worried that nobody is ready for what’s coming. “There’s going to be a huge economic incentive for people to do jailbreaks and prompt injections,” he says. The best defense could be to expect the worst. Organizations shouldn’t trust LLMs, and they should expect that anything done by agents could be unsafe, he says: “That’s not a great solution, but it just might be what we have to do.”
“It’s really incredible that these things are being deployed everywhere to control super-critical systems,” he adds. “There’s been no study of the fundamental science here. We’re all doing it ad hoc.”
By taking advantage of this flaw, which concerns how LLMs identify who or what is giving them instructions, the researchers were able to make popular LLMs spit out information they had been trained not to provide, such as how to synthesize cocaine and how to sabotage a commercial aircraft’s navigation system.
“There’s a real probability that this is going to be a problem that’s fundamentally unsolvable,” says Charles Ye, an independent researcher and coauthor of the ICML paper.
Companies will typically hire teams of human testers to try to come up with novel attacks that break existing guardrails, a process known as red-teaming. Model makers also use LLM super-hackers (such as OpenAI’s GPT-Red) that find and exploit weaknesses in other models to automate parts of this process. The goal is then to take those attacks and train a new model to resist them and anything that looks like them.
The problem, says Jasmine Cui, another independent researcher and coauthor of the paper, is that the approach amounts to giving the models a list of things they shouldn’t do. But no list is exhaustive. “It’s like watching The Simpsons and they have Bart writing ‘I will not say something inappropriate to my teacher’ a hundred times,” she says. “And he still does things that are pretty crass anyway.”
The ICML paper describes attacks against several of OpenAI’s models, but Cui and Ye say that they have since seen similar results with models made by Anthropic, Alibaba, and DeepSeek.
Cui and her colleagues wanted to find out why an attack like chain-of-thought forgery was so effective. They suspected it had something to do with the mechanism that LLMs use to keep track of where their instructions are coming from.
But what Cui and her colleagues discovered is that LLMs are in fact very bad at keeping track of different roles. In a series of experiments that looked at what was going on inside a handful of different models, the researchers found that LLMs seem to identify the role of a specific chunk of text not by the tags around it but by the style of that text and the words it contains.
The upshot, the researchers claim, is that all an attacker needs to do to hack an LLM is write text that spoofs a certain role. And because roles are a fundamental part of how LLMs work, no amount of training will fully solve the problem.
Ye is worried that nobody is ready for what’s coming. “There’s going to be a huge economic incentive for people to do jailbreaks and prompt injections,” he says. The best defense could be to expect the worst. Organizations shouldn’t trust LLMs, and they should expect that anything done by agents could be unsafe, he says: “That’s not a great solution, but it just might be what we have to do.”
“It’s really incredible that these things are being deployed everywhere to control super-critical systems,” he adds. “There’s been no study of the fundamental science here. We’re all doing it ad hoc.”
Submission + - AI Companies Destroying Books At Scale (futurism.com)
nightflameauto writes: AI companies are purchasing books from pre-AI times. The claim is that those books are free from the defects of AI generated text, and therefore more valuable for ingestion into AI datasets. In order to accomplish this, they are tearing the books down to be scanned, then destroying them as the scans are completed. This includes rare and out-of-print books that may be some of the few copies of any given published work left in existence.
Submission + - Is Mark Zuckerberg Actually TRYING to Destroy Meta? (futurism.com)
fjo3 writes: Lately, it almost feels like Zuckerberg is trying to destroy his own company. He’s burning through mountains of cash in a desperate attempt to keep up in the AI race. Yet despite the untold billions it’s spent so far, it’s being destroyed by OpenAI and Anthropic. Its employees have even resorted to using AI models made by its competitors, a humiliating reality check for how far behind Zuckerberg’s efforts have fallen.
It’s also a financial nightmare. Capital expenditures have risen dramatically, erasing any appetite for Meta on Wall Street. Its stock price “has been dead money for more than a year,” as Yahoo Finance notes, as investors continue to debate whether Zuckerberg is chasing the AI industry as it careens off a cliff — or edging ever closer to an AI-fueled industrial revolution.
It’s also a financial nightmare. Capital expenditures have risen dramatically, erasing any appetite for Meta on Wall Street. Its stock price “has been dead money for more than a year,” as Yahoo Finance notes, as investors continue to debate whether Zuckerberg is chasing the AI industry as it careens off a cliff — or edging ever closer to an AI-fueled industrial revolution.
Submission + - The first active shooter suppression service
An anonymous reader writes: Schools to tackle active shooters with pepper-spraying drones that can ram attackers
Pepper-spraying drones aimed at tackling active school shooters are being rolled out at campuses across three states.
The drones, part of the Campus Guardian Angel program, can ram into targets at speeds of up to 70 mph and are capable of weaving through the air to avoid gunfire. Three schools in Florida, five in Georgia and one in Colorado are part of pilot schemes using the aerial technology.
Pepper-spraying drones aimed at tackling active school shooters are being rolled out at campuses across three states.
The drones, part of the Campus Guardian Angel program, can ram into targets at speeds of up to 70 mph and are capable of weaving through the air to avoid gunfire. Three schools in Florida, five in Georgia and one in Colorado are part of pilot schemes using the aerial technology.
Submission + - Your Brain Can Rewire Itself to Allow True Multitasking (sciencealert.com)
alternative_right writes: Our daily lives are built on multitasking, but are our brains actually doing two things at once, or just switching very quickly between them?
Well, it depends. The science shows it's the latter, for cognitively demanding work.
But in a new study published in the Journal of Cognitive Neuroscience, researchers from Georgetown University Medical Center in the US have revealed that we can put certain tasks on autopilot in a way that enables something closer to true multitasking.
Driving is the perfect example: When you take your test, your entire mental and physical energy is concentrated on executing the right combinations of movements and thoughts.
After a decade behind the wheel, the brain is no longer consciously thinking everything through in great detail.
Well, it depends. The science shows it's the latter, for cognitively demanding work.
But in a new study published in the Journal of Cognitive Neuroscience, researchers from Georgetown University Medical Center in the US have revealed that we can put certain tasks on autopilot in a way that enables something closer to true multitasking.
Driving is the perfect example: When you take your test, your entire mental and physical energy is concentrated on executing the right combinations of movements and thoughts.
After a decade behind the wheel, the brain is no longer consciously thinking everything through in great detail.
Comment Microsoft: Keep paying more. (Score 1) 58
Microsoft
Micro-Soft
Hugely-Hard
Micro-Soft
Hugely-Hard
Comment Someone should start making CD players (Score 4, Funny) 96
It's obvious we need new physical media that is cheaper than thumb drives. Someone invent a better solution for dirt cheap write-once media, there's a huge market for it as long as the data lasts.
Submission + - AI Data Centers Being Built Faster Than They Can Be Secured (securityweek.com)
wiredmikey writes: AI is reshaping data centers and introducing security risks traditional architectures weren't designed to handle. As AI data centers scale at breakneck speed, security isn't keeping up. Researchers outline the Top 10 AI infrastructure security risks, including hardware integrity, multi-tenant isolation, high-speed network fabrics, supply chain compromise, and patching failures.
Submission + - Physicists create first room-temperature quantum material (phys.org)
alternative_right writes: In a study published in Nature, LSU physicists have developed the first room-temperature quantum material capable of distinguishing and transporting different quantum states of light, overcoming one of the biggest challenges in quantum materials research. Led by Associate Professor of Physics Omar S. Magaña-Loaiza, the work establishes a general design principle for engineering an entirely new class of quantum materials, opening new possibilities for quantum computing, secure communications, sensing technologies and advanced energy systems.
Comment An AMAZING number of flaws (Score 4, Insightful) 77
The big underlying issue is why does Microsoft deliver software with so many flaws?
Submission + - How Microsoft's "Little Workaround" Created a Major Pentagon Threat (propublica.org)
joshuark writes: ProPublica Reporter Renee Dudley heard Microsoft was running tech support for the U.S. Defense Department through China, the country’s biggest cybersecurity adversary.
The arrangement was called “digital escorting.” She thought it sounded like a conspiracy theory — until she started looking into it. This is the story of what she found and how her investigation changed government policy.
Microsoft is using engineers in China to help maintain the Defense Department’s computer systems — with minimal supervision by U.S. personnel — leaving some of the nation’s most sensitive data vulnerable to hacking from its leading cyber adversary, a ProPublica investigation has found.
The arrangement, which was critical to Microsoft winning the federal government’s cloud computing business a decade ago, relies on U.S. citizens with security clearances to oversee the work and serve as a barrier against espionage and sabotage.
National security and cybersecurity experts in the Trump administration contacted by ProPublica were also surprised to learn that such an arrangement was in place, especially at a time when the U.S. intelligence community and leading members of Congress and the Trump administration view China’s digital prowess as a top threat to the country.
Microsoft uses the escort system to handle the government’s most sensitive information that falls below “classified.” According to the government, this “high impact level” category includes “data that involves the protection of life and financial ruin.” The “loss of confidentiality, integrity, or availability” of this information “could be expected to have a severe or catastrophic adverse effect” on operations, assets and individuals, the government has said. In the Defense Department, the data is categorized as “Impact Level” 4 and 5 and includes materials that directly support military operations.
“If someone ran a script called ‘fix_servers.sh’ but it actually did something malicious then [escorts] would have no idea,” a former Microsoft engineer who worked on the escort system, told ProPublica in an email. That said, he maintained that the “scope of systems they could disrupt” is limited.
In an emailed statement, the Defense Information Systems Agency said that cloud service providers “are required to establish and maintain controls for vetting and using qualified specialists,” but the agency did not respond to ProPublica’s questions regarding the digital escorts’ qualifications.
It’s unclear whether other cloud providers to the federal government use digital escorts as part of their tech support. Amazon Web Services and Google Cloud declined to comment on the record for this article. Oracle did not respond to requests for comment.
A spokesperson for the inspector general — whose office is supposed to operate independently in order to investigate potential waste, fraud and abuse — told ProPublica they were not authorized to speak about the issue and directed questions to DISA public affairs.
The arrangement was called “digital escorting.” She thought it sounded like a conspiracy theory — until she started looking into it. This is the story of what she found and how her investigation changed government policy.
Microsoft is using engineers in China to help maintain the Defense Department’s computer systems — with minimal supervision by U.S. personnel — leaving some of the nation’s most sensitive data vulnerable to hacking from its leading cyber adversary, a ProPublica investigation has found.
The arrangement, which was critical to Microsoft winning the federal government’s cloud computing business a decade ago, relies on U.S. citizens with security clearances to oversee the work and serve as a barrier against espionage and sabotage.
National security and cybersecurity experts in the Trump administration contacted by ProPublica were also surprised to learn that such an arrangement was in place, especially at a time when the U.S. intelligence community and leading members of Congress and the Trump administration view China’s digital prowess as a top threat to the country.
Microsoft uses the escort system to handle the government’s most sensitive information that falls below “classified.” According to the government, this “high impact level” category includes “data that involves the protection of life and financial ruin.” The “loss of confidentiality, integrity, or availability” of this information “could be expected to have a severe or catastrophic adverse effect” on operations, assets and individuals, the government has said. In the Defense Department, the data is categorized as “Impact Level” 4 and 5 and includes materials that directly support military operations.
“If someone ran a script called ‘fix_servers.sh’ but it actually did something malicious then [escorts] would have no idea,” a former Microsoft engineer who worked on the escort system, told ProPublica in an email. That said, he maintained that the “scope of systems they could disrupt” is limited.
In an emailed statement, the Defense Information Systems Agency said that cloud service providers “are required to establish and maintain controls for vetting and using qualified specialists,” but the agency did not respond to ProPublica’s questions regarding the digital escorts’ qualifications.
It’s unclear whether other cloud providers to the federal government use digital escorts as part of their tech support. Amazon Web Services and Google Cloud declined to comment on the record for this article. Oracle did not respond to requests for comment.
A spokesperson for the inspector general — whose office is supposed to operate independently in order to investigate potential waste, fraud and abuse — told ProPublica they were not authorized to speak about the issue and directed questions to DISA public affairs.
Submission + - Feds Want to Drop Requirement for Autonomous Cars to Have a Brake Pedal (caranddriver.com)
sinij writes:
Braking distance regulations would stick around, but the physical pedals would no longer be required for cars designed to be exclusively autonomous.
I would want any autonomous car to have emergency brake and/or shutoff.
Submission + - Europe: The World's Fastest-warming Continent (barrons.com)
fjo3 writes: The latest heatwave sweeping across Europe is a stark reminder that it is the world's fastest-warming continent, stretching into an Arctic that is heating at an even greater pace.
Britain, France, Italy and Spain have issued red alerts and health warnings for much of their territory this week as the region endures its second heat episode since May.
Britain, France, Italy and Spain have issued red alerts and health warnings for much of their territory this week as the region endures its second heat episode since May.
Submission + - Helion says the 1st fusion power plant is coming soon. A cofounder isn't so sure (scientificamerican.com)
tedlistens writes: The startup backed by Sam Altman recently raised $465 million, tripling it's valuation as it races to build what it says will be the world's first fusion power plant, supplying Microsoft with carbon-free electricity in 2028.
But one of its founders—the plasma scientist whose research inspired its reactor design—has serious doubts.
But one of its founders—the plasma scientist whose research inspired its reactor design—has serious doubts.
