Deliberately eradicating a species is remarkably difficult - humans have done it a few times by over-hunting, but not against any species that can fight back. Attempts to eradicate invasive species or pests like mosquitoes usually fail.

However, there are already all sorts of ways someone evil enough could kill a LOT of people, both with and without AI assistance. Is is just total eradication we should fear?

The total eradication of humans would be a tall order, but I suppose an AI that had already taken over the world would have a lot of options. Maybe just kill everything living - poison the planet?

Right - these "AI researchers" are just programmers, and not very good ones at that. They are not doing research - they are doing grubby engineering work trying to reduce cost. maximize benchmarks, etc.

This RSI they want to get everyone riled up about is just automating this grubby engineering work.

Yeah, if people are genuinely concerned about the safety aspect of AI, not just the fact that it'll put most people out of work, then this is the wrong thing to focus on.

The danger that exists today is far more mundane. It's that AI has become extremely capable of hacking into online systems, and that can not only cause a lot of economic damage (deleting data, shutting down systems, etc), but can in fact kill people too, as simple as shutting down a 911 response system, or messing with a hundred other types of system.

AI is indeed being rolled out far to quick, especially this needlessly dangerous type that these companies have chosen to develop, so let;'s focus on the danger that exists today. Tomorrow we can focus on tomorrow's problem.

You've gotta admit it is a good name!

I think what they are, in effect at least, trying to say is that you do own (not license) the model output, but as a pre-condition for using their service you agree not to use the output to compete against them.

It's basically as if Microsoft said you can't buy our compiler if you are going to use it to build a compiler (or a clippy, or anything else that we do).

How they work, and/or how they are trained?

Anything specifically?

Distillation is the word used to refer to the practice of using the outputs of one LLM as inputs to train another, often smaller one.

The big US companies like OpenAI and Anthropic have usage "terms of service" that forbid you from doing this, but Gary Tan is saying this is unreasonable and you should be allowed to use the outputs of an LLM in any way you choose.

Specifically, Tan is hoping that, if allowed to, some US companies will choose to do this - train their own models with the help of outputs generated from these large/expensive OpenAI/etc ones, then release these new models in "open weights" form (i.e. downloadable, so you can run it on your own computer if you want to).

The "open weights" business model is interesting. The companies doing this can still make money by making larger customers pay, or by selling services (e.g. model customization), just as a Linux vendor can make money selling/providing open source software.

It does seem a bit weird.

When you pay for API usage, are you buying the model's output or are you licensing it under some restricted usage terms?

It seems what companies like Anthropic are saying is "you are buying the output, but we won't sell it to you unless you agree not to use it to compete with us".

It'd be like Microsoft saying we'll only sell you a C compiler if you agree not to use it to write a compiler.

The headline is questioning what Gary Tan is advocating for, namely that:

1) The US frontier labs shouldn't be allowed to impose terms of service that restrict how users can use the model's outputs

AND

2) Some US labs should take advantage of 1) to distill a, presumably smaller, model from the outputs of these frontier ones, and should release the resulting model in open weights form

I don't think they lose money on everything - it seems their coding subscription plans are probably not proifitable when maxxed out, as many people do, but at current full API rates it is likely quite a profitable business. It's not so hard to figure out, since the cost to provide the service is basically the cost of the hardware amortized over it's useful lifetime.

It does seem that AI is basically a commodity, and when cheap is good enough then the cheapest providers win, just as happened to PCs. No need to pay for a Ferrari when all you need/want is a Ford.

Yes, the AI companies, especially OpenAI and mostly Anthropic, are lobbying for government regulation, partly to exclude Chinese and open weights competition.

It is complicated though.

The guy, Jacob Coxon, you are presumably referring to, may well be regarded as part of Anthropic's own fear-mongering campaign and push for at least partly, maybe largely, self-serving regulation, but at the same time there are many developers working for these companies who say that the internal fear of what they are developing is almost universal.... and yet they keep working on it for whatever reason(s), whether that is pending IPO-riches or some justifcation about needing to develop AI to keep up with an arms race (which they seem to be the ones accelerating).

It seems what we need to do, and perhaps actually will, hopefully before it's too late, is to distinguish AI that is useful for everyday use like coding, and AI that is ridiculously overpowered for things like this and anyways dangerous, that indeed should be government regulated.

It's not an either/or matter of regulate or not - there needs to be some intelligence put into what is regulated or not, just as we allow people to own guns, but not fully automatic assault weapons or rocket launchers, allow people to drive cars, but do not license top fuel dragsters to be used on public roads.

> Controlling superintelligent AI is a fantasy.

Perhaps, but what we have today is something much dumber, yet arguably equally dangerous, in the form of these RL-trained LLMs.

It's like trying to beat DeepBlue or Stockfish as chess - it may not be intelligent, but by automating something and applying brute force compute to it, it becomes highly capable.

Many of these RL-trained LLMs have been specifically trained to be good at hacking and exploit generation, and also just for finding software bugs, which is only one step-removed. Take a hacking-trained LLM and apply brute force compute to it and you get the hacking version of Stockfish - you can call it dumb if you like, but it'll be able hack into many/most of the systems that have a vulnerability to exploit.

Could we at least control these hacking experts? Perhaps, or minimally we need their release to be slow enough to give time for all the potentially vulnerable systems (esp. infrastructure - air control systems, power/water plant control systems, etc, etc) to be checked and hardened against attacks.

To be fair, I think most people are just token predictors.

Yes, for anyone interested a lot of the details are here:

https://cims.nyu.edu/~tristanb...

Estimated around $20M at API prices.

Just to scoop a human researcher's work, then threaten to "ruin his career".

https://venturebeat.com/techno...

How is OpenAI morally right?!?

Best case for OpenAI is that they heard rumours that a researcher was closing in on a NS breakthrough using a specific method, so they instantly set a team on the problem giving them with a massive amount of resources. Their internal team then specifically guided the model to use the same very specific direction the researchers was using, something they (nor the model) would ever have known to do without the rumours. A detail that OpenAI conveniently left out of the narrative.

And that's the best case scenario. The worst case is that because the researcher was using ChatGPT to work on drafts of the paper and those drafts ended up in the training data for the model. So rather than discovering anything the agents just implemented / finished off the work they already had access to.

It's insanely sketchy. The NYU researcher is honestly being extraordinarily calm about the whole thing, typically in a situation like this people would be getting close to a fist fight.