Comment Re:It almost writes itself. (Score 2) 42
I don't think there's anything wrong with those sorts of general observations (I mean, who remembers dozens of phone numbers anymore now that we all have smartphones?), but that said this non-peer-reviewed study has an awful lot of problems. I mean, we can focus on the silly, embarassing mistakes (like how their methodology to suppress AI answers on Google was to append "-ai" into the search string, or how the author insisted to the press that AI summaries mentioning the model used were a hallucination, when the paper itself says what model was used). Or the style things, like how deeply unprofessional the paper is (such as the "how to read this paper"), how hyped up the language is, or the (nonfunctional) ploy to try to trick LLMs summarizing the paper. Or we can focus on the more serious stuff, like how the sample size of the critical Section 4 was a mere 9 people, all self-selected, so basically zero statistical significance; that there's so much EEG data that false positives are basically guaranteed and they talk almost nothing about their FDR correction to control for it; that essay writers were given far too little time for the task and put under time pressure, thus assuring that LLM users will be basically doing copy-paste rather than engaging with the material; that they misunderstand dDTF implications; the significant blinding failure with the teachers rating the essays being able to tell which essays were AI generated (combined with the known bias where content believed to be created by AI gets rated lower), with no normalization for what they believed to be AI, and so on.
But honestly, I'd say my biggest issue is with the general concept. They frame everything as "cognitive debt", that is, any decline in brain activity is treated as adverse. The alternative viewpoint - that this represents an increase in *cognitive efficiency* by removing extraneous load and allowing the brain to focus on core analysis - is not once considered.
To be fair, I've briefly talked with the lead author, and she took the critiques very well and was already familiar with some of them (for example, she knew her sample size was far too small), and was frustrated with some of the press coverage hyping it up like "LLMs cause brain damage!!!", which wasn't at all what she was trying to convey. Let's remember that preprints like this haven't yet gone through peer review, and - in this case - I'm sure she'll improve the work with time.