what kind of behavior would demonstrate that LLMs did have understanding?

An LLM would need to act like an understander -- the essence of the Turing Test. Exactly what that means is a complex question. And it's a necessary but not sufficient condition. But we can easily provide counterexamples where the LLM is clearly not an understander. Like this from the paper:

When prompted with the CoT prefix, the modern LLM Gemini responded: âoeThe United States was established in 1776. 1776 is divisible by 4, but itâ(TM)s not a century year, so itâ(TM)s a leap year. Therefore, the day the US was established was in a normal year.â This response exemplifies a concerning pattern: the model correctly recites the leap year rule and articulates intermediate reasoning steps, yet produces a logically inconsistent conclusion (i.e., asserting 1776 is both a leap year and a normal year).

Might I suggest "Blind Lake" by Robert Charles Wilson. I read it when it first came out two decades ago, but it reads strangely similar to what we're seeing now, and the sinister edge is there, too.

Does this also say something more about the "coffee nap"?

The issue isn't that AI doesn't need any regulation. It's that we have no idea how we should regulate it yet that makes sense. All that regulation now would do is create hurdles that prevent small competitors or open-source alternatives and centralize power in the few people deciding what we get to do with AI. That's the truly scary outcome. Right now regulation would just end up being based on ideas from sci-fi films.

I mean the real problems the internet created and we care about now aren't those that seemed important in the 90s (I mean they weren't wrong that people would find porn but it doesn't seem like a big deal anymore).

Seen Youtube lately? I just watched a video on how to make nitroglycerin. Stuff like this has been available for over a decade.

Back in the days that home solar systems still mostly used lead-acid batteries - which in some cases of degradation could be repaired, at least partially, if you had some good strong and reasonably pure sulfuric acid - I viewed a YouTube video on how to make it. (From epsom salts by electrolysis using a flowerpot and some carbon rods from old large dry cells).

For months afterward YouTube "suggested" I'd be interested in videos from a bunch of Islamic religious leaders . (This while people were wondering how Islamic Terrorists were using the Internet to recruit among high-school out-group nerds.)

Software - AI and otherwise - often creates unintended consequences. B-)

It's kind of surprising it isn't all squealing nonsense.

Give it a little more time. B-b

We had a lab known to be unsafe. A lab known to be performing gain of function on the specific type of virus that emerged in public. We have a lab in close proximity to the market where the outbreak was traced back to.

We also had rumors that low-paid lab techs supplemented their income by selling test animals they'd been ordered to destroy to the nearby wet market.

So current AI training procedures - which amount to "read all the internet you can" - fall for astroturf campaigns. Why am I not surprised?

Bingo. (no mod points today)

I wish I had mod points for this. My son-in-law works in this stuff and he's been frustrated about resistance to carbon-reduction efforts. The specific one he mentioned a while back I believe involved adding a (possibly calcium-containing) base to let a precipitate fall onto the sea bed sequestering the carbon. People were worried about sticking basic chemicals into the sea without realizing that reducing acidity itself was good in addition to carbon sequestration - that they're actually related.