Have you ever considered that pointing out that "x is bad" does not in any way imply "y is good".

No, they do not demonstrate any "expert level capabilities". Experts always understand the "why", not just the "how". LLMs are incapable of that.

I do agree that review effort for LLM output is probably what kills their use in most application scenarios.

No. You eventually have to lock up (or kill) people for the tiniest infractions. This makes sure your economy goes to hell. (Ooooops....)
See, for example, the theocracies. Which is essentially what you are arguing for.

Indeed. I think one reason is because the US is missing a really big domestic catastrophe where they cannot conveniently blame "somebody else" for it. Or at least where that blame-shifting is obvious enough that most people see it for what it is. Hence there is this completely irrational and disconnected feeling of superiority and being invulnerable. For most people, humility only comes from an experience of abject failure. For some, it never comes. But humility is critical for seeing reality, so if you do not have it, eventually you will get it the hard way. Well, most people will.

To be fair, I think chances are really good the current US administration has made sure this catastrophe will happen quite soon. If so, at least one good thing would have come out of the current mess.

I do not know whether predestination plays a role, but wanting to damn people for all eternity (or at least the rest of their natural lives) is a common theme with religious fanatics.

Yes. The claims they want less recidivism are simply lies. It is known how to get these rates down. It is well-known that positive reinforcement works a lot better than negative. It is known that making it easy for people to reintegrate into society, most will go for it. But they want to be "tough on crime". That is denial of reality and that never gives you good outcomes.

On the plus-side, large prison populations are good for keeping our citizens in fear and laws that criminalize everything and anything allows you to get rid of inconvenient people easily. Of course, the rich and powerful almost never have any of those enforced against them, no matter what despicable and repulsive things they do.

Indeed. As is creating criminals. Hence school-to-prison pipelines and other utter evil.

And in the Soviet Union everybody knew many of these were not real criminals ...

Indeed. But the religious fundamentalists want people to be damned forever (!) and hence that is the prevalent sentiment in the US. Guess what, people that get no chance to reintegrate into society do not do so. Also makes for a nice "us vs. them" world-view, something the fundamentalists absolutely love.

The potential is there, but I do not think it will happen. Respective research would find out things that are massively unwanted by the rich and powerful. Just think about research into the mindset of malignant narcissists and you should immediately see what I mean. Or the (very solid) results that conservatives are dumber than liberals. Or that most people cannot fact-check. Or the Dunning-Kruger effect.

Bottom line is that we already know a lot, it is just not used because society and the leaders it choses are not rational. Expanding that knowledge is not desired by too many people.

I do not think there are mysterious cases. There are just some where people chose not to cooperate enough for us to make a determination and that is their right.

Also note that there are quite a few "too important to jail" cases, see, for example some prominent stock scammers or rapists and child abusers or murderers/war criminals. These cases are probably the worst, because they give not-smart people the impression that you can get away with it. And hence overall ethics decline.

Your problem starts here. Right to privacy is a human right, and these were established as response to the 3rd Reich catastrophe. One characteristic is that you cannot lose a human right, regardless of what you do. It can be temporarily restricted if another thing has priority, but it cannot be removed. Hence people like you are into violating human rights and as soon as that starts to be a general sentiment, a state/group/organization is on a very dark path. Yes, you may be able to get some statistic to look better this way. But you have lost something far more valuable.

Additionally, the way it's phrased, as a "bet," is implicitly skeptical.

There is no question that states are betting on AI.

If you think software never breaks, I have a bunch of 5.25" disks somewhere that want to have an argument with you.

It's a complete strawman to argue that physical things break. If I buy music, digitally, that won't break and yet nobody sane would expect that the band can at some random time in the future say "we revoke all our music". I can also think of a number of physical things that unless I mistreat them will easily survive me and three generations down the line.

This is not about replacements, it's about taking the product sold away but keeping the money.