While LLM's aren't likely to be THE thing to finally crack AGI, they are a crucial part of future research into it. We will need an interface, and language is OUR interface. That being said, a lot of your thinking and reasons are flawed.
>they are at the same time deeply flawed in ways that humans of average intelligence are not
I think you are vastly over estimating "average" human intelligence.
>continuing to hallucinate, and not understanding when their regurgitated human reasoning patterns actually apply or not
Remind you of anyone? Like religious zealots that deny basic science, flat earthers, antivaxxers, and millions upon millions of humans who refuse to change their viewpoints even when there is overwhelming evidence that they are wrong...
>They give advice without thinking thought the consequences, then just say "my bad" and move on if you are knowledgeable enough to call them out.
See point 2 above. Except most of the humans doing this don't bother to admit they may have been wrong. They just double down on their ignorance and that it's "their truth" or some other horse shittery.
>There are also huge gaps in LLM capability that even the most stupid of humans don't have, such as on the job learning. If you show a stupid human how to do something simple, enough times, they will eventually get it, and memorize the skill
Evidence says otherwise. If you don't believe me, just go watch people try to pay with credit / debit at the store. Something they've been doing for well over a decade, and the majority look at the card reader like it's asking them to do brain surgery instead of swipe their card and MAYBE put in their pin that they haven't changed in 10 years yet still can't seem to remember.
Face it, "average" human intelligence isn't really that hard of a bar to beat.
>It may learn "in context" one day, but the next day, and the next, will be groundhog day when you have to teach it all over again.
Currently, yes, this is a limitation with static inference only models. I've read a few papers where some people are experimenting with giving non-volatile memory access to store newer data in. So the newer architecture models can have a reference list of already inferred / corrected data to not only "learn" things that either weren't in their original training data, but were learned wrong, or the model was corrected for a hallucination when running a certain inference path.
A side benefit of that would be drastically reducing compute time the longer the model is online. If it checks the memory file, see that it already ran those inference calculations and got an answer, it can skip that whole step. If the model then only has to do heavy calculations for 40% instead of 100%, that reduced power used, compute time, cooling needs in the datacenter, and probably myriad other things that aren't as obvious as those.