A lot of the 'headline' announcements, pro and con, are basically useless; but this sort of thing does seem like a useful cautionary tale in the current environment where we've got hype-driven ramming of largely unspecialized LLMs as 'AI features' into basically everything with a sales team; along with a steady drumbeat of reports of things like legal filings with hallucinated references; despite a post-processing layer that just slams your references into a conventional legal search engine to see if they return a result seeming like a pretty trivial step to either automate or make the intern do.
Having a computer system that can do an at least mediocre job, a decent percentage of the time, when you throw whatever unhelpfully structured inputs at it is something of an interesting departure from what most classically designed systems can do; but for an actually useful implementation one of the vital elements is ensuring that the right tool is actually being used for the job(which, at least in principle, you can often do since you have full control of which system will process the inputs; and, if you are building the system for a specific purpose, often at least some control over the inputs).
Even if LLMs were good at chess they'd be stupid expensive compared to ordinary chess engines. I'm sure that someone is interested in making LLMs good at chess to vindicate some 'AGI' benchmark; but, from an actual system implementation perspective, this is the situation where the preferred behavior would be 'Oh, you're trying to play chess; would you like me to set "uci_elo" or just have Stockfish kick your ass?" followed by a handoff to the tool that's actually good at the job.