I am familiar with fine tuning, but it seems you are not familiar with its limitations. There's a reason that even the frontier labs distinguish between pretraining and fine tuning.

There's not a great analogy for conventional software and fine tuning, but if one has the pretraining corpus then there are no limits (other than GPU time), so that's a good analogy for open source.

You seem to think that since fine tuning allows SOME customization then there's no need to be able to retrain from scratch.

The least bad analogy I can think of is a compiled binary with an SDK for extensibility. That lets you customize some aspects of the behavior. But it is not a substitute for having the full source code and being able to rebuild from it.

Yes and no. You're not wrong, but there's a limit to how far you can go with that approach. A web search for "catastrophic forgetting" will explain more.

It's a hard problem.

That's a bit like saying that compiled binaries are just as good as source code because you can modify the program's behavior by altering the machine code. Just reverse engineer it and patch the binary. Who needs source code anyway?

Also, finding the right weights to update is hard. And by hard I mean that an army of PhD students and postdocs is working on that problem.

As for continued training of an existing model... a web search for "catastrophic forgetting" will be instructive. You can mitigate that risk using the original training data, but then we're back to needing access to that original training data.

The idea of "open weights" AI models makes about as much sense as "open binary" software.

It means you can run it locally. It doesn't mean you can modify the source and recompile and enjoy your customized version.

The following analogies aren't perfect but they are useful:

Using backpropagation to make an LLM from training data is like using a compiler to turn source code into an LLM. Using evaluations to monitor the training process is like using tests to monitor the continuous integration pipeline.

An open source LLM would include the mountain of data that it was trained on, plus the software that trained it. Including evaluations.

There are of course a number of complicating factors, but to a first approximation a truly open model must include the data that was used for pre training and fine tuning. As well as the software that ran the training. And the evaluations.

Part of the challenge is that training systems are often highly customized datacenters full of GPUs, so making portable build environments is hard. But nobody promised that this would be easy.

A5k1ng f0r 4 fr13nd.

It was a few years ago, but it was fascinating.

https://www.preposterousuniver...

They focused entirely on the cost to the AI consumers - which is not unreasonable since that's who Gartner's customers are.

But it's still a noteworthy omission for anyone with the slightest interest in the trajectory of the LLM industry as a whole. If OpenAI / Anthropic / XAI are providing their services at a loss, and their customers are consuming those services at a loss, then the GPU and RAM vendors are the only winners.

It is standard practice for those private owners of the means of production to use some of the profits to bribe / lobby the government to revise policy in ways that support their businesses.

In other words, corruption is no less a feature of capitalism than it is of communism.

"We're losing money on every chat message, so we need to make up for it in volume!"

On the supply side: the largest allocation capital in human history is flowing into datacenter construction.

On the demand side: the two biggest providers are competing on price because they're just taking turns at the tops of the capability leaderboards.

And they're both losing money.

* recoup your investment

Imagine thinking that you can recoup your interesting in making third party hardware for Microsoft's latest idea.

Windows phones
Windows CE tablets/phones before that
Every windows CE device, in fact
Spot watches
Home server
Cortana smart speakers...

There was a series called Humans from the UK that was also great. Only 4 seasons.

...and not only routing around it but standing up new servers to deliver more and faster.

I wonder if John Gilmore truly knew just how right he was.

This article talks more about what Asocks does.

https://www.techtimes.com/arti...

And to answer another Slashdotter's question, yes, there is an Android component here.

Yes, and that's something that LLMs are, in my experience, extraordinarily bad at.

We fired a new hire last year who was clearly just delegating everything to an LLM. When I asked why he did something the way he did, he responded with a list of obvious LLM hallucinations. Bizarre "requirements" that nobody ever asked for.

If you ask it for reasons, it will give you reasons. Like when RFKjr asked for medical research to support his absurd conclusions and it hallucinated a bunch of bogus citations.

It'll be interesting to see whether the frontier labs are able to improve on the current state of the art. The information might exist in the company, spread across emails and chats and random docs, so it's not impossible. But deducing intent does sound harder than writing code.