I think what they are, in effect at least, trying to say is that you do own (not license) the model output, but as a pre-condition for using their service you agree not to use the output to compete against them.

It's basically as if Microsoft said you can't buy our compiler if you are going to use it to build a compiler (or a clippy, or anything else that we do).

Understood, and yeah single tasking seems like a good workaround given the situation. Or switching to an LLM with a larger context window, if possible/practical. I was able to get about 7X more context window "runway" using the AST approach, but it's a relatively small code base to begin with.

How they work, and/or how they are trained?

Anything specifically?

Distillation is the word used to refer to the practice of using the outputs of one LLM as inputs to train another, often smaller one.

The big US companies like OpenAI and Anthropic have usage "terms of service" that forbid you from doing this, but Gary Tan is saying this is unreasonable and you should be allowed to use the outputs of an LLM in any way you choose.

Specifically, Tan is hoping that, if allowed to, some US companies will choose to do this - train their own models with the help of outputs generated from these large/expensive OpenAI/etc ones, then release these new models in "open weights" form (i.e. downloadable, so you can run it on your own computer if you want to).

The "open weights" business model is interesting. The companies doing this can still make money by making larger customers pay, or by selling services (e.g. model customization), just as a Linux vendor can make money selling/providing open source software.

It does seem a bit weird.

When you pay for API usage, are you buying the model's output or are you licensing it under some restricted usage terms?

It seems what companies like Anthropic are saying is "you are buying the output, but we won't sell it to you unless you agree not to use it to compete with us".

It'd be like Microsoft saying we'll only sell you a C compiler if you agree not to use it to write a compiler.

The headline is questioning what Gary Tan is advocating for, namely that:

1) The US frontier labs shouldn't be allowed to impose terms of service that restrict how users can use the model's outputs

AND

2) Some US labs should take advantage of 1) to distill a, presumably smaller, model from the outputs of these frontier ones, and should release the resulting model in open weights form

It's really the same problem that we see with quantum physics -- the subject matter is esoteric enough that common sense can't be used to determine whether what you're listening to is really advanced theory or utter BS (or some combination of the two); they both sound the same, and the situation isn't helped by the presence of large numbers of people who think they know what they are talking about but don't. :/

Some place between sometimes and often - I'm not here to argue that - researchers are the first to discover.

But it's also almost meaningless to just go, "I'm such an amazing realist, the worst case happens every time."

That's not really being realist. That's just you not knowing actual things, and having an emotion.

Sounds like you're running up against the context window of whatever LLM you're using. Try converting your code to a minified abstract syntax tree first and pass that to your LLM, and ask it what full text it needs in addition to that. It should stay within the context window much better that way.

I prefer the Martian bear.

I don't think they lose money on everything - it seems their coding subscription plans are probably not proifitable when maxxed out, as many people do, but at current full API rates it is likely quite a profitable business. It's not so hard to figure out, since the cost to provide the service is basically the cost of the hardware amortized over it's useful lifetime.

It does seem that AI is basically a commodity, and when cheap is good enough then the cheapest providers win, just as happened to PCs. No need to pay for a Ferrari when all you need/want is a Ford.

Yes, the AI companies, especially OpenAI and mostly Anthropic, are lobbying for government regulation, partly to exclude Chinese and open weights competition.

It is complicated though.

The guy, Jacob Coxon, you are presumably referring to, may well be regarded as part of Anthropic's own fear-mongering campaign and push for at least partly, maybe largely, self-serving regulation, but at the same time there are many developers working for these companies who say that the internal fear of what they are developing is almost universal.... and yet they keep working on it for whatever reason(s), whether that is pending IPO-riches or some justifcation about needing to develop AI to keep up with an arms race (which they seem to be the ones accelerating).

It seems what we need to do, and perhaps actually will, hopefully before it's too late, is to distinguish AI that is useful for everyday use like coding, and AI that is ridiculously overpowered for things like this and anyways dangerous, that indeed should be government regulated.

It's not an either/or matter of regulate or not - there needs to be some intelligence put into what is regulated or not, just as we allow people to own guns, but not fully automatic assault weapons or rocket launchers, allow people to drive cars, but do not license top fuel dragsters to be used on public roads.

> Controlling superintelligent AI is a fantasy.

Perhaps, but what we have today is something much dumber, yet arguably equally dangerous, in the form of these RL-trained LLMs.

It's like trying to beat DeepBlue or Stockfish as chess - it may not be intelligent, but by automating something and applying brute force compute to it, it becomes highly capable.

Many of these RL-trained LLMs have been specifically trained to be good at hacking and exploit generation, and also just for finding software bugs, which is only one step-removed. Take a hacking-trained LLM and apply brute force compute to it and you get the hacking version of Stockfish - you can call it dumb if you like, but it'll be able hack into many/most of the systems that have a vulnerability to exploit.

Could we at least control these hacking experts? Perhaps, or minimally we need their release to be slow enough to give time for all the potentially vulnerable systems (esp. infrastructure - air control systems, power/water plant control systems, etc, etc) to be checked and hardened against attacks.

We're playing with machines that we don't fully understand, which can move faster than we can, and we're giving them more and more autonomy. AIs don't have to kill ALL humans for us to legitimately begin to worry. A black swan event where a bunch of automated systems all suddenly begin maximizing the wrong expectation value at the same time could cause a massive global catastrophe so fast the damage would be done before anyone even knew it was happening, let alone be able to halt it.

Forgetting he exists would be the best punishment.

Oh, you just mean every who isn't on the approved list.