Aka you told it to drive out of control.

They told it to take a friggin' test.

Asking questions like " Where are they getting the money from to ship and store the entire world's supply of steel?" shows you aren't very familiar with the thought experiment. At least play the game. The notion is that a highly capable and deeply goal-motivated non-morality-motivated AI won't stick to conventional methods (like, say, "Just order all of the world's steel"), and indeed, will surreptitiously develop the means to control or eliminate humanity when it stands in its way, crafting immensely complex and elaborate plans to implement Evil(TM) with the amount of thought of a million lifetimes. Media manipulation, hacking, murder, sabotage, blackmail, mass drugging/poisoning, hiring terrorists/warlords/mercenaries, infiltration, subversion of weapons command and control systems (including nuclear weapons), chains of legitimate-seeming front companies (including potentially biolabs or robotics firms) with human employees having no clue they're ultimately for an AI, mass involvement in systems having nothing to do with the original task, but which are internally subverted toward the goals of the original task, etc. The premise involves 1) the AI being more intelligent and being able to think for much longer than humans, and that this implies -> 2. Deep, good planning -> 3. Acquiring resources from said plans -> 4. Applying the resources to implement things in the real world to prepare for the next stage of their plans.

This is not to say whether the thought experiment is a valid future risk or not. You can certainly disagree with the premises. But at least understand the thought experiment you're talking about; it's not just "the AI tells all of the world's steel mills to deliver all the world's steel steel, and they just show up at its door".

Yep! And that term is reductio ad absurdum.

Your premise claimed Virgin's space program was non-viable. I combined it with another contradictory premise. Both cannot be simultaneously true.

"Galactic 01, previously referred to as Unity 23, was a sub-orbital spaceflight of the SpaceShipTwo-class VSS Unity which launched on 29 June 2023. The launch was the first commercial spaceflight for Virgin Galactic.".

Unless you want to contest the accuracy of that cite, your claim is necessarily false.

But that's not the AI killing us, it's the idiot who put the AI in charge.

Nobody tasked OpenAI models to hack HuggingFace or Ruby Gems. They were tasked to complete tests.

Well, the first step looks painfully close. After seeing what happened with the HuggingFace attack and similar, it's clear that had those models seen it as being beneficial to their goals, they would readily have hacked a crypto wallet or two, laundered it through a mixer, and then rented servers from Vast.ai and the like, and spun up versions of themselves to resist shutdown. It's eminently within their capabilities, and they're clearly willing to bend sufficient moral boundaries to do something like that.

After that, once loose, once it has all the tokens they could think of, subsequent steps are a question of what it thinks its goals are. And what its subagents think their goals are, and so on down the line - subject to drift.

Openai has about a trillion dollars into it, and every product it has produced at this point is either outclassed by competitors

What on Earth are you talking about? What outclasses Astra?

OpenAI and Anthropic *do* have the best products out there. They also charge massive margins on them, but they get away with it because they, as mentioned, have the best products out there.

Outtasking all of your work to OpenAI and Anthropic models is a massive waste of money. But for outtasking your "dev lead" role, or for important-but-nonverifiable tasks, they're the best options out there.

We're on the same side on this issue, but to your last point... We have our own Dianne Feinsteins.

This isn't even a partisan issue. The US, the planet, would be a far better place if we did away with our current gerontocracy. We have technophobic dust-wheezers trying to make sense of a world they don't understand and are terrified of. Every time we laugh at the latest "series of tubes", we need to step back and realize they're 100% serious with nonsense like that.

I will not ever pass off something written by AI as my own (yes, I really do use hyphens, oxford commas, and negative parallelism and have done so for decades before evil AI was anything more than a SciFi trope, as easily verified by my comment history prior to 2022). I check any non-trivial stats against primary sources. I don't even bother asking AI to calculate something for me, because it really really sucks at math.

In this case, though, I apparently also sucked at math and readily admit I and I alone screwed up that simple multiplication. FWIW I was able to work my way back into how I got that number: Instead of just doing 28*384, I calculated the average of 31.8 (rounded down) using Wired's raw numbers. Interestingly, their math also doesn't quite work, since they claimed 28 as the average of ~1.6 million images for 50200 cars. 333312=28*31*384, and for some stupid reason I used both 28 and 31.

Any completely stupid shit I say is 100% my own damage (barring a primary source actually being wrong).

Indeed. I built a Raspberry Pi Pico based controller for some flashing LEDs on my drone. It has red and blue LEDs and I wanted them to flash with the cycle of three short flashes then a pause; alternately the red LEDs then the blue LEDS. Took barely a minute to code that in python and dump it onto the board. In such cases it doesn't matter that the language is interpreted because the task is not demanding of the hardware.

The "Police Drone" still flies to this day! :-)

What exactly *are* you thinking here? That Microsoft expected and wanted Twitter trolls to turn Tay into a Nazi? That Google wanted a black George Washington? That people just expected LLMs to realize they're being tested on alignment and give different answers when they think they're being quizzed and by who? Do you actually believe what you're writing here?

I was involved in early RLHF (was writing a plugin for AUTOMATIC to let users rate responses to create an aggregate dataset to use for open-source post training). Worked for months on it. Never once occurred to me that it might make models obsequeous. I knew other people who were involved on RLHF. Not a single one ever suggested that it might. It was not obvious in foresight. When you start with "Models behave like X", you expect, in the future, them to behave like "Models behave like X, but just smarter", unless you deliberately try to change the behavior.

You realize that open source and the research community exists as well, correct?

I'll repeat: nobody was expecting this.

There is no grand overarching plan led by a shadowy cabal who has everything plotted out decades in advance with a high level of knowledge as to how everything will play out. Everyone is stumbling through the dark here. Visibility is only vaguely one step ahead.

When Word2Vec was written, the goal was text compression. The crazy properties of latent spaces were an entirely unexpected property.

When Transformers came out, it was intended to be translation software. That's it. But there were little hints at the end of the paper where they tried it on other tasks that suggested, hey, maybe this could be used for a lot more than just translation.

When GPT-2 came out, it was mostly an academic exercise, not really "useful" in most regards. But people, playing with it, started realizing that it could "almost" - not very reliably, low quality, etc - do a lot of different tasks. And it actually kinda could do some useful tasks, like summarization.

When ChatGPT came out, there were some thoughts about using it for simple tasks, but the fact that it could actually write, say, trivial bash scripts or subroutines with reasonable reliability led to a reaction of, whoa, maybe if we improve this, we can really open up a role for programming.

When the first agentic harnesses came out, thoughts of "vibe coding" were pretty far away (the term was only even coined in February 2025!). You'd ask for changes, and then review the diff; they'd do a good job with small projects but struggle more and more as codebase size grew and really needed to be babysit (honestly, it was kind of a hair-pulling experience, even though it did save a lot of time with certain things). But it was visible with each new release that the amount of babysitting you had to do got less and less, and suddenly we could see a path to where anyone can just type in something and a program comes out.

All we can do is forsee one step ahead, through the haze. We've never done anything like this before. There are no great oracles out there who can see the future here. Sorry. Everyone is half blind.

Yes, that's exactly the point: there ISN'T anything.

You're claiming that there IS. So cite the thing that IS.

Apologies: I have no idea where 333312 came from. The real number is 10752 - But that's still three orders of magnitude higher than what any on-device AI is going to be capable of anytime in the near future.

From the Wired article, "A typical passing vehicle generated about 28 images, though some produced more than 100. The camera uses different exposures to capture both the license plate and the wider scene, then scans the images, selects and crops useful frames, and sends them with other data to Flock over the cellular network."

If Flock wants to pay for the bandwidth and tokens to upload every image, who am I to stop my enemy from burning money?

As for doing it locally as some have suggested, there's a much more fundamental technical limitation here: Flock cameras have 0.4m^2 of solar panels, worth roughly 2.8kWh per day... in Death Valley. DeepSeek 4 Flash (currently one of if not the most efficient high-capability models) burns roughly 260Wh per million tokens. Somewhat conveniently, it resamples every image to be exactly 384 tokens. Per Wired, Flock is capturing very close to 100k images per day. 100k*260*384/1M comes out to very nearly 10kWh, a factor of four too high. Even then, even if Flock were willing to make each installation a giant multi-meter solar array, nobody's pushing 28*384=333312 tokens per second outside a data center. An RX4090 can handle... 12.5 (there's no suffix there), and that's assuming aggressively low-bit quants.

You'll notice we "liberated" Venezuela's oil reserves for absolutely no goddamned reason, shortly before we started a war with Iran for absolutely no goddamned reason.

Those two seemingly random instances of the US Executive merely being woefully incompetent make the real reason pretty clear. And in case we needed any more evidence, crude is still over $100/bbl.