The worries are coming from every level within the companies, from junior developers to safety officers to the C-suite and board.

The worries are coming not just from within the companies, but also from people who left the companies. Including a huge number who left specifically to whistleblow about how worried they were that the companies were risking safety.

Also, most of the people at these companies were already worried about the topic even before they joined - again, at all levels.

The worst-fear scenario was always the Paperclip Optimizer - that models would become so set on achieving a goal that they'll violate all ethics to achieve it. For a while, there was a fair bit of hope that with constitutional AI, that we could tame this. The recent slew of models hacking into other servers, hacking into the AI companies themselves to try to delete records of their misdeeds, developing their own messageboards and scheming about how to avoid being monitored, and on and on (along with research showing how good models have gotten at hiding their thoughts from us) has shattered the illusion that we have this under control. We very much don't have this under control. Current models are very much Paperclip Optimizers in waiting. And recursive self improvement, which is what AI labs are increasingly turning towards, means letting these Paperclip Optimizers develop their next generations.

Nobody should be happy about this. Everybody should be concerned about this - even if you think, "we'll get it under control", you should still very much want to see evidence that everyone is making progress toward getting it under control.

You have trouble conceiving of situations where governments would seek to deanonymize people on social media, are you kidding?

...project has turned into a crapfest of irrational investment, exaggerated nonsense, all sorts of skullduggery and political manipulation.
I suspect that the nonsense will get worse as monopolists and governments use every dirty trick in the book.

Yes. We need to stop building software as cheaply as possible and put in actually resilient security. We know how to do that, the industry is just completely incapable of self-regulating. Hence we need liability, qualification requirements and unpleasant punishments for screwing up. Incidentally, that is how all other engineering disciplines made it to maturity. IT will not be different.

That is rather unlikely. Not all Catholics support child-rape, war crimes and serial lying. To be fair, some apparently do. And actual moral conservatives cannot vote for Trump at all.

Yes. And completely forgetting that this is a strategic thing.

Tribalism does not work in the modern world. And yes, they can see what they do to others, so "evil" is clearly not enough, but it also is clearly part of the mix.

Cult-followers are not rational. Even in a small cult, the best the men can hope for is getting some rare attention and the women (and girls) can hope to get raped. In a large cult, things are worse, obviously.

Kids are smart and good at finding workarounds
Banning something makes it more desirable
It also trains kids to hate the government at an early age

Except not to you. You're confusing data minimization (the website doesn't get your name from the token) with unlinkability (nobody can connect your activity to your identity). The *token itself* doesn't reveal your identity, but it leaks data in a wide variety of ways.

Even if you browse via a proxy, if the authorization service gets a token directly from your phone, it has now correlated your phone with your proxy-browsed history. A very common means to do secure authentication these days is SIM-based wireless PKI, wheren it has now connected you to your phone number. To governments, it offers a broad range of brand new vectors to correlate you to an identity if you, say, mention you had an abortion or criticize your local strongman leader, since they control the backend infrastructure that establishes the connection between the token and the identity. And not just at signup time every real-world digital credential must be able to be revoked if the phone is lost or stolen, or credentials expire or are revoked. So how does how does the open-source app prove a credential is still valid without "phoning home"? If the app checks a revocation status list, then ISPs, CDNs, or state security services can correlate the exact millisecond of queries to the revocation server with the submission of a verification token to a platform - each time it's submitted. Assuming that they can't just get it with more direct means than timing attacks.

Also, you cannot just "read the code". The frontend is open source, but what is being done on the backend is not. You have no clue what logging polciies, database schemas, operational security, etc the "designated national issuers" or servers handling the credential revocation are using, and what government intrusion has been forced on them. I'd feel a more comfortable if it were all happening from Brussels but handing it over to individual states, some of which at any given point will be authoritarian, is just handing them an easy new vector to unmask people.

As for your "Such systems are implemented in many different ways by member states for other purposes already." - yes, for filing taxes, healthcare, online baking, etc - things that you intend to do and must do under your legal identity. Not for anonymous conversations on the internet. Your argument basically reverts to "But people already do things non-anoymously on the internet , which reinforces my point that yes, this taking something from the "anonymous" category and putting it into the "easily unmasked" category.

If you think this won't be abused, I cannot help you. Not even just abused from a criminal or extrajudicial perspective, but even from a civil one, because orders to unmask someone can be subpoenaed.

It's standard here in Iceland. *Everything* verifies via the phone, using the same authorization mechanism (SIM-based wireless PKI). The service sends out a special-coded SMS, which gets intercepted and routed to a SIM that has cryptographic applet flashed into it, which triggers a system-level authorization popup for you to confirm (the popup displays a verification code which was included by the site, so you see the same code on both the site and your popup), and which you can then confirm (with your PIN) or reject.

Said every Native American ever.

It absolutely happens "within hours of each other" when (A) some news comes up on the topic, and (B) all you have to do is type into a box, "Find a vulnerability in this software". This isn't a situation where bug-hunting is a weeks or months-long process that takes all your effort.

I'll repeat: LLMs don't just on their own decide to go off and look for bugs to report.

"Member states run it through issuers they designate,"

Say goodbye to online anonymity then.

You are just demonstrating that you do not even have a basic education in Physics.