You've gotta admit it is a good name!

I think what they are, in effect at least, trying to say is that you do own (not license) the model output, but as a pre-condition for using their service you agree not to use the output to compete against them.

It's basically as if Microsoft said you can't buy our compiler if you are going to use it to build a compiler (or a clippy, or anything else that we do).

How they work, and/or how they are trained?

Anything specifically?

Distillation is the word used to refer to the practice of using the outputs of one LLM as inputs to train another, often smaller one.

The big US companies like OpenAI and Anthropic have usage "terms of service" that forbid you from doing this, but Gary Tan is saying this is unreasonable and you should be allowed to use the outputs of an LLM in any way you choose.

Specifically, Tan is hoping that, if allowed to, some US companies will choose to do this - train their own models with the help of outputs generated from these large/expensive OpenAI/etc ones, then release these new models in "open weights" form (i.e. downloadable, so you can run it on your own computer if you want to).

The "open weights" business model is interesting. The companies doing this can still make money by making larger customers pay, or by selling services (e.g. model customization), just as a Linux vendor can make money selling/providing open source software.

It does seem a bit weird.

When you pay for API usage, are you buying the model's output or are you licensing it under some restricted usage terms?

It seems what companies like Anthropic are saying is "you are buying the output, but we won't sell it to you unless you agree not to use it to compete with us".

It'd be like Microsoft saying we'll only sell you a C compiler if you agree not to use it to write a compiler.

The headline is questioning what Gary Tan is advocating for, namely that:

1) The US frontier labs shouldn't be allowed to impose terms of service that restrict how users can use the model's outputs

AND

2) Some US labs should take advantage of 1) to distill a, presumably smaller, model from the outputs of these frontier ones, and should release the resulting model in open weights form

I don't think they lose money on everything - it seems their coding subscription plans are probably not proifitable when maxxed out, as many people do, but at current full API rates it is likely quite a profitable business. It's not so hard to figure out, since the cost to provide the service is basically the cost of the hardware amortized over it's useful lifetime.

It does seem that AI is basically a commodity, and when cheap is good enough then the cheapest providers win, just as happened to PCs. No need to pay for a Ferrari when all you need/want is a Ford.

Yes, the AI companies, especially OpenAI and mostly Anthropic, are lobbying for government regulation, partly to exclude Chinese and open weights competition.

It is complicated though.

The guy, Jacob Coxon, you are presumably referring to, may well be regarded as part of Anthropic's own fear-mongering campaign and push for at least partly, maybe largely, self-serving regulation, but at the same time there are many developers working for these companies who say that the internal fear of what they are developing is almost universal.... and yet they keep working on it for whatever reason(s), whether that is pending IPO-riches or some justifcation about needing to develop AI to keep up with an arms race (which they seem to be the ones accelerating).

It seems what we need to do, and perhaps actually will, hopefully before it's too late, is to distinguish AI that is useful for everyday use like coding, and AI that is ridiculously overpowered for things like this and anyways dangerous, that indeed should be government regulated.

It's not an either/or matter of regulate or not - there needs to be some intelligence put into what is regulated or not, just as we allow people to own guns, but not fully automatic assault weapons or rocket launchers, allow people to drive cars, but do not license top fuel dragsters to be used on public roads.

> Controlling superintelligent AI is a fantasy.

Perhaps, but what we have today is something much dumber, yet arguably equally dangerous, in the form of these RL-trained LLMs.

It's like trying to beat DeepBlue or Stockfish as chess - it may not be intelligent, but by automating something and applying brute force compute to it, it becomes highly capable.

Many of these RL-trained LLMs have been specifically trained to be good at hacking and exploit generation, and also just for finding software bugs, which is only one step-removed. Take a hacking-trained LLM and apply brute force compute to it and you get the hacking version of Stockfish - you can call it dumb if you like, but it'll be able hack into many/most of the systems that have a vulnerability to exploit.

Could we at least control these hacking experts? Perhaps, or minimally we need their release to be slow enough to give time for all the potentially vulnerable systems (esp. infrastructure - air control systems, power/water plant control systems, etc, etc) to be checked and hardened against attacks.

Yes, for anyone interested a lot of the details are here:

https://cims.nyu.edu/~tristanb...

Estimated around $20M at API prices.

Just to scoop a human researcher's work, then threaten to "ruin his career".

https://venturebeat.com/techno...

He surprisingly knowledgeable about AI, and started an AI-based film studio and sold it for $500M.

An LLM is just a stack of Transformer layers - they are great at what they do - prediction - but that is all they are built to do.

Humans are not just intelligent predictors. As a social species we have also evolved to (mostly) peacefully co-exist and co-operate, and a lot of our intelligence seems in fact to be collective intelligence.

If we want LLMs to behave in more human ways, then we need to make them more brain like, and start adding all the extra moving parts that evolution has equipped ourselves with.

Yeah - makes no sense to me either, but notable that this "you'll all be on UBI (government welfare)" is the best outcome he can think of.

Dario Amodei (Anthropic CEO)'s "Machines of Loving Grace" was the first of these, and if effect explains why others like Altman and Zuckerberg are chiming in.

Amodei's prediction:

1) AI will take ALL the jobs

2) Since people won't have any jobs, the government will have to provide UBI (universal basic income), aka food stamps

This is the most positive spin Amodei can put on the future that he is rushing to create, and not surprisingly people are not happy. The public increasingly see AI as an unwelcome technology that will create more problems than positive benefit.

This is at least part, a large part, of the reason that others like Altman and Zuckerberg are coming out with their own spin on how wonderful the AI future will be - because they realize public perception is massively against them (and also they are a bunch of arrogant pricks who think they are creating god, and are entitled to tell us how grateful we should be for it).

Well, I guess you could call a cruise missile a "mobile" application.

Bottom line it's a Russian weapons system enabled by an American electronics component.