Supposed to be the obligatory joke thread? But no citation of Betteridge?
Me? I think it's a deep and complicated philosophic topic, but the relevant starting joke is "It's the poor craftsman who blames his tools" extended to "... but it's the bankrupt craftsman who doesn't use the tools that allow him to compete in the market in the real world." ChatGPT is such a tool for certain kinds of paid work, and just ignoring the tool isn't going to make it go away. (Though I also like the extension "...but it's the worst craftsman who doesn't know and want the best tool for each job at hand.")
There should also be a joke about "Know your enemy" around here. Citation of The Art of War called for? Or the more recent book Mastering AI by Jeremy Kahn, whose answer is very much "No because of my rose-colored glasses." Rather a shallow book overall and I'd still recommend A Thousand Brains by Jeff Hancock as the best AI book I've read over the last few years.
But back to the original story... I am inclined to agree with him, which is why I'm trying to limit my use of AI tools and also trying to consider how such tools affect my approach to solving problems. There is a strong temptation to just look at the result, test it sufficiently, and not think deeply enough about why it works. But even worse, when you do ask a GenAI to explain, the explanations it offers always seem quite plausible and believable, but they do not represent any "human understanding" of why.
The human-level epistemology of Mercier and Sperber are also relevant, which got me to look at all of the links to try to find any reference. Nothing I could find, but perhaps I should have been looking for some other authors who have done more recent work on the topic? Short version: Most of the time we do not think. You could say we act (and speak) stupidly, but it doesn't really rise to that level. Rather it is more like the way GenAI responds based on patterns that fit. For humans, it is the entire situation, not just a prompt, but we respond to the situation and most of our responses "work" well enough. If we are asked "Why?", then we are pretty good at explaining ourselves, but its basically confabulation and the explanation did not exist until after the fact. The Enigma of Reason says a lot about 'my-side' bias in the explanations, but that's predicated on normal intelligence. In some cases the broken brain just spews gibberish and obvious lies...