Comment Re:They can only self-improve if they are capable (Score 1) 156
You can actually plot improvement as measured by benchmark vs power consumption to train, and you see the power consumption curve upwards while the benchmarks seem to be asympote trending towards a specific limit. I've got a theory on it too. An LLMs basic fitness criteria is "Output text as close as possible to the training data". Well one thing that is NOT in the training data is text generated by anything smarter than a human. Because we've never seen an entity thats smarter than us. So the whole thing seems to be converging on a limit thats essentially 'one extremely well read human", and it has no data to tell it what something smarter than a human would look like.
All training further does is push that curve closer to the limit line, but it can never go above it. Theres a reason a lot of AI researchers are pretty adamant that the transformer LLM is probably not the final form of AI that can do the whole AGI superintelligence thing.