Cool, and my grandma's goal could have very well been to win the Tour De France, yet it still didn't make her any more bicycle-like.
Your analogy is that your grandma wanting to achieve something unachievable for her is comparable to LLM pretraining specifically trying *not* to memorize token sequences?
Not very intelligent, are you?
Do you have any proof of that?
Do I have proof of that?
It's self-fucking-evident. Do you think a calculator decodes an unwrapped ALU output to answer your inputs?
But guess what- it could.
Because vast majority of people definitely see various forms of "1 + 1 = 2" repeat many times before they get to the level where they can derive that formula themselves, and I'd argue it's fairly easy to prove that vast majority of people actually remember that "1 + 1 = 2" and don't derive it in practice.
Strawman. Probably unintentional on your part given your demonstrated inability to hold a train of thought, but:
I asserted that I did not need to, you answered with "well I bet most people do."
If the simplified example was confusing, replace 1 + 1 = 2 with 44321 * 454531 = 20145268451. Yes, an LLM can solve that, and no, it has never seen it in its training data.
As far as I know, we have seen LLMs generally tending towards being more reliable at repeating things that are already in the data than they are at inventing novel things
LLMs are terrible repeaters, except for very contrived situations.
The fact that their knowledge base is centered around the exact data they have ingested is not a mystery. That being said, models generalize- and that is the goal of pretraining- for them to generalize. Repating is an error that you want to avoid, because a model that repeats is a dumb model.
We don't want the model to know that Paris comes after "What is the capital of France?".
We want it to have encoded Paris as a thing called a capital of a place that is called france that is a country in europe. And they do.
and as far as I know, there's been zero LLMs that have written operating systems "from scratch" that haven't had multiple complete operating systems in their training data.
It would indeed be very unlikely that an LLM that had not seen examples of an operating system could ever generate one. That doesn't mean it reproduced what it learned. It means it learned the shape of an operating system from its data.
Again, I do not need to encode 44321 * 454531 = 20145268451 to solve for 44321 * 454531.
I've been training neural nets since well before the initial Transformers paper.
Perhaps you have, but in the same way an LLM would- you had no fucking idea what you were doing ;)
You vomiting a couple [overfitting mitigation methods] isn't impressive to me.
I suspect nothing impresses a bullshit artist such as yourself- it's part of your persona to hide that.
We've known for a long time that some information is more useful when blurred, and often easier to memorize when less specific. Yes, of fucking course, we're aiming for generalization in AI models.
And your regurgitation of things you don't understand isn't impressive to me.
But I you haven't shown anything to prove that this particular LLM isn't just sticking together pieces of kernels it has seen. Nobody calls a collage "from scratch".
It's quite easy to prove that this doesn't contain a single line of linux within it. Nobody said it doesn't contain the semantic concepts that it learned from linux- that was, in fact, the entire point you dim-witted poser.
The models knows what goes into a scheduler. It can reproduce a scheduler with any kind of variation you need within the shapes of other concepts it has learned. It does not need the code it took to train its weights to understand that shape.
Nobody calls a collage "from scratch".
And nobody calls someone else's code a collage, though by your definition, it most certainly is.
