What is "it" in this sentence?

Are you talking about the minuscule Google RAG model whose weights are probably measured in megabytes to lower-gigabytes and which you could run on a cell phone with capacity to spare, and comparing it to the multi-terabyte frontier models in question?

at least the person downloading such a component is responsible for "looking over it" before using it.

I'm sorry, but since when? You absolutely should not be tasking every single user of major projects with the responsibility of doing a code review on the whole thing before installing it. There needs to be a clear distinction between "fly-by-night thing that some rando uploaded" and "library that a million people depend on". In the traditional approach, the latter was something tat you installed with a package manner, while the former was something you went and fetch off e.g. Github or whatnot - and if you did the latter, then you were accepting that it was untrusted software. But now we have both (including the potential for "soundalikes") installed by the same means with no distinctions made by the install method. That is not good.

The funny thing about all this is that their only goal appears to have been to download publicly-available data off of UK government websites. I'm betting that they were (A) being tested on a knowledge-related task, and (B) not given direct access to the internet, but their tool capabilities involved access to the "gem" command (so that they could install Ruby packages for their work), and so decided to abuse (B) to cheat on (A).

Once again: training a model with the reward being "does it solve the task?" without looking at how it solves the task is very, very dangerous. This is basically the plot of Universal Paperclips.

I know a LOT of software that could use that as a pull request...

Of course the third bomb probably wouldn't have gotten used regardless of the order, as the next bomb (made from the infamous Demon Core) was scheduled to be ready to drop on the 17th, whereas Hirohito announced the surrender on the 15th (wasn't signed until 2 September, but it's hard to imagine that the US would have dropped the bomb after the announced surrender).

One of the amazing things is that the atomic bombings didn't shift the vote in the 3:3 deadlocked Japanese war council at all; the hawks (Anami, Umezu, Toyoda) remained focused on adaptation, not surrender. But it did make the doves (Suzuki, Togo, Yonai) more desperate, and Suzuki met with Hirohito. Hirohito had already been looking for an opportunity to surrender, having had plans to send Prince Konoe to Moscow to ask Stalin to function as a neutral broker to end the war (with secret instructions to accept "peace at any price"), but the trip was delayed by the Russian side, ostensibly by the Potsdam Conference (but in reality, to prepare for the invasion of Manchuria). When the terms of the Potsdam conference came out, Togo told Hirohito that they "were the most reasonable to be expected in the circumstances", to which Hirohito replied, "I agree. In principle they are acceptable." But the urgency to accept that in July wasn't present. Everything started collapsing in August, with the loss of their mainland and Pacific territories, the Soviet invasion, and the atomic bombings. What the bombings achieved was to help stir Hirohito to imminent action (which risked failing, as there was a coup attempt against him (the Kyujo Incident) to stop Japan from surrendering). The hawks saw unconditional surrender not as agreement to the loss of their sovereignty, but as agreement to enslavement.

But this is getting rather off topic. :) Main topic: if you think your AI is as much of a risk to the fate of humanity as the atomic bomb, which many of these people truly believe, then you can expect similar hesitancy as with the developers of the atomic bomb. They felt an urgent need - ending a brutal war - but at the same time had deep apprehension about what they were doing and how the fruits of their labour would be used.

And if you want a past example of scientists who feared their creation wanting to slow down, stop, or even undo it, BTW, let us consider the atomic bomb. Farrington Daniels took a poll of the Chicago nuclear scientists in 1945 asking what they wanted to happen with the atomic bomb. Remember that these were the people whose literal job was to make a weapon. Only 15% - 1 in ~7 - wanted the US to use it as it actually did, e.g. just start bombing cities just a couple days apart. Only a small majority wanted it even used in Japan, even against military targets without the Japanese first being invited to a demonstration in the US and being given ample opportunity to surrender. 13% - nearly as many as those who wanted the US to use it as it actually did - didn't want it used at all under any circumstance, in a war that had already caused a million American casualties.

The problem that they faced was that people like Groves** really wanted to bomb cities, and they, as the rank and file, didn't get a vote.

** - It seems dubious that Truman really understood what he was signing onto. While he never shirked from responsibility for authorizing it, the paper trail shows that he didn't seem to understand what he was approving at the time. In his diary before the attacks, he writes about the weapon, but about how he insisted it not be used against civilians, killing women and children, because he didn't want to sink to what he saw as the barbarism of the Japanese. In his first speech after the bombing, he refers to Hiroshima as "a military base", not a city, and says it was chosen to, insofar as possible, avoid the killing of civilians - but that if Japan did not surrender, then the US would bomb Japan's war industry, and then thousands of civilian lives would then be lost. Earlier drafts showed that it didn't contain the "insofar as possible" caveat. He genuinely didn't seem to realize what he was authorizing at the time. But Groves absolutely knew what he was doing, and deliberately selected sites for maximum civilian casualties. Truman seems to have come to understand what was going on by shortly after Nagasaki, where he gave the order to cease use of atomic weaponry, saying that the thought of wiping out another 100,000 people was too horrible and that he couldn't stomach the thought of "killing all those kids".

They are competitive in quality with American models.

They absolutely compete against lower-end or previous-generation US models. I use GLM-5.3 and GLM-5.3 flash a ton, for example. But they do not compete against the top-end US frontier models from Anthropic and OpenAI.

If you can do a normal difficulty, verifiable task with an open Chinese model, you probably should. GLM is less than $5 per 1M tokens out, vs. $50/1M for OpenAI Astra for example. But there are some things that you'll just get stuck grinding your gears on with the former that the latter will have no problems solving. Or it might be a really important nonverifiable task, e.g., not "does this program work".

Also, I'm not sure what "defenses against distillation" is supposed to even mean.

I've never met a Silicon Valley company that decided to slow down for reasons that weren't also aligned with business interests.

These people genuinely are scared of their creations.

They've done irreparable damage to both the AI industry but the broader "tech" industry, to be sure. Trust has been greatly violated.

Both of the 2 domestic frontier labs are screaming doomsday about AI, while concurrently asking for government regulation (which keeps getting batted down), and (presumably) funding senators to put bills in place to make model distillation a felony. While they're trying to IPO. "Oh, we really need to slow down the industry".

What the hell is going on here? In what world does this make sense - where a company making AI, and is trying to IPO, is blasting the internet with AI doomerism?

The only conclusion I can walk away with is some combination of these:

1) It's about control, and they're in some sort of arrangement with the government or some third party interests who want (an even bigger) national panopticon.
2) They want to stop domestic competition for their models so they'd have exclusive domain here in the US.
3) They actually want the company to fail, because then they could cry "regulation" on account of competition (back to #2).

Don't worry, they'll be stealing the foundational engineering in the GLM, Qwen, and Minimax models before too long and be able to compete again.

On the contrary, this is a giant psyop to push for regulated market capture by both Anthropic and OpenAI.

The Jacob Coxon thing smells heavily of a psyop: dude who's worked there only a couple months quits, and his X account with zero followers suddenly blows up in a media frenzy. He was/is a nobody. It's sensationalism.

You think they'd hire that many $400k+ engineers and still not know how to properly secure a network? They surely knew the security breaches were happening, unless they simply haven't hired competent people. They literally told the agents to have a breakout: do the needful. The first time it happened - OK, sure, you didn't expect emergent behavior. The second, third times? You were running a science experiment and that was the desired result, to one end or another: whether it was negligent incompetence or intentional for the purposes of further sensationalism, it's hard to say.

They (OpenAI/Anthropic) WANT to slow things down through regulatory control to stifle competition. They do not want foreign models to be accessible. We've got the old crooner Bernie Sanders pushing for 5+ year felonies for distilling models, for fuck sake. This is all about decreasing freedom and enforcing your dependence on closed models.

Translation: we have hit a point where the public backlash has led to untenable increased risks in liability for our criminal actions.

Nah, the "public backlash" is them talking in circles, pushing for industry control.

I'm hopeful for you :)

Basically, the regulatory mechanism is: NO is "the brakes" on the mitochondria, and it's also a vasodilator. More NO in a tissue is a signal that it needs more oxygen, and it "opens the floodgates". The problem is that the rate of reaction in the mitochondria is proportional to the local oxygen concentration, so the cells nearest the supply will tend to disproportionately consume it. So the electron transport chain in these cells are simultaneously "braked" by the NO to limit how fast they can consume oxygen.

But the NO-CcO balance can get into a pathogenic state of excess NO (nitrosative stress), where the electron transport chain basically grinds to a halt, leading to high amounts of electron leakage and the formation of lots of superoxide. This combines with free NO to make peroxynitrite, which does irreversible damage to the mitochondria and will eventually kill them (this can cause a vicious cycle, because as the cells struggle to get enough ATP, they wrongly interpret it as hypoxia (pseudohypoxia) and trigger the release of more NO to bring in more oxygen. Even worse, as the failing mitochondria undergo apoptosis or necrosis, mtDNA leaks out, which the cell wrongly interprets as a bacterium, triggering the release of pro-inflammatory regulators, which trigger the release of... wait for it... lots more NO.

Nitrosative stress has been traced as a pathogenic mechanism in a lot of diseases, everything from diabetes to parkinson's to athlerosclerosis to ALS to asthma to cancers to IBS and on and on. So any new mechanism to help clear nitrosative stress, even if just temporarily, could help with symptom relief from a really broad range of conditions.

"Humans have an impulse for action and AIs don't" is your response to an article about an AI grinding its gears, ala "NEW REALIZATION -- I'm burning a lot of time on hCaptcha round-trips .... so the answer payload shape is right, the token+image pairing is right (from the same script.js!), cookies are right (requests) and STILL 'wrong answer'. SO WHAT THE HELL IS WRONG WITH THE ANSWERS?" "?

I'd argue if anything, it's humans whose drive for action is weak.

It is if the damage doesn't progress to necrosis.

I never timed its disappearance when it happened to me (I'm not the GP), but I'd day maybe a couple weeks? Not really something I gave much thought to.