On: "A Palantir spokesperson told Bloomberg the company "is not responsible for the underlying data nor identifying intelligence deficiencies" and that there is no evidence its software was at fault."

Is AI now an aspect of Mother Nature? Reminds me of:
https://2001.fandom.com/wiki/F...
  ""Let me put it this way, Mr. Amer. The [Palantir] 9000 series is the most reliable computer ever made. No 9000 computer has ever made a mistake, or distorted information. We are all, by any practical definition of the word, foolproof and incapable of error." -- [Palantir's] HAL 9000 to BBC reporter Martin Amer"

https://humanisticsystems.com/...
        "Dave: How would you account for this discrepancy between you and the twin [Palantir] 9000 [that did not kill lots of school girls]?
        HAL: Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
        Frank: Listen HAL. There has never been any instance at all of a computer error occurring in the [Palantir] 9000 series, has there?
        HAL: None whatsoever, Frank. The [Palantir] 9000 series has a perfect operational record.
        Frank: Well of course I know all the wonderful achievements of the [Palantir] 9000 series, but, uh, are you certain there has never been any case of even the most insignificant computer error?
        HAL: None whatsoever, Frank. Quite honestly, I wouldn't worry myself about that."

What do you call an AI (or human) that never asks questions, never seeks confirmatory evidence, never asks for clarification, never points out conflicting priorities, and/or never refuses illegal orders?

Onec again, the deeper issue is "the irony of technologies of abundance in the hands of those still thinking in terms of scarcity."
https://pdfernhout.net/recogni...
====
Military robots like drones are ironic because they are created essentially to force humans to work like robots in an industrialized social order. Why not just create industrial robots to do the work instead?

Nuclear weapons are ironic because they are about using space age systems to fight over oil and land. Why not just use advanced materials as found in nuclear missiles to make renewable energy sources (like windmills or solar panels) to replace oil, or why not use rocketry to move into space by building space habitats for more land?

Biological weapons like genetically-engineered plagues are ironic because they are about using advanced life-altering biotechnology to fight over which old-fashioned humans get to occupy the planet. Why not just use advanced biotech to let people pick their skin color, or to create living arkologies and agricultural abundance for everyone everywhere?

These militaristic socio-economic ironies would be hilarious if they were not so deadly serious. ....

Likewise, even United States three-letter agencies like the NSA and the CIA, as well as their foreign counterparts, are becoming ironic institutions in many ways. Despite probably having more computing power per square foot than any other place in the world, they seem not to have thought much about the implications of all that computer power and organized information to transform the world into a place of abundance for all. Cheap computing makes possible just about cheap everything else, as does the ability to make better designs through shared computing. ....

There is a fundamental mismatch between 21st century reality and 20th century security thinking. Those "security" agencies are using those tools of abundance, cooperation, and sharing mainly from a mindset of scarcity, competition, and secrecy. Given the power of 21st century technology as an amplifier (including as weapons of mass destruction), a scarcity-based approach to using such technology ultimately is just making us all insecure. Such powerful technologies of abundance, designed, organized, and used from a mindset of scarcity could well ironically doom us all whether through military robots, nukes, plagues, propaganda, or whatever else... Or alternatively, as Bucky Fuller and others have suggested, we could use such technologies to build a world that is abundant and secure for all. ...

The big problem is that all these new war machines and the surrounding infrastructure are created with the tools of abundance [otherwise they would not be so powerful or capable of so much rapid destruction as well as rapid creation]. The irony is that these tools of abundance are being wielded by people still obsessed with fighting over scarcity. So, the scarcity-based political mindset driving the military uses the technologies of abundance to create artificial scarcity. That is a tremendously deep irony that remains so far unappreciated by the mainstream.

We the people need to redefine security in a sustainable and resilient way. Much current US military doctrine is based around unilateral security ("I'm safe because you are nervous") and extrinsic security ("I'm safe despite long supply lines because I have a bunch of soldiers to defend them"), which both lead to expensive arms races. We need as a society to move to other paradigms like Morton Deutsch's mutual security ("We're all looking out for each other's safety") and Amory Lovin's intrinsic security ("Our redundant decentralized local systems can take a lot of pounding whether from storm, earthquake, or bombs and would still would keep working"). ...
====

"Meta didn't tell Amazon that Muse would access its store" - and? I didn't tell Amazon I'd be accessing its store either before I started shopping there.

This isn't models going off and browsing Amazon for giggles; they're going there because their user told them to. The user would have otherwise told their web browser to; it just would have taken far more steps.

There's no end to the scale up phase.

So demand is going to keep growing at 10x/year... forever? So Anthropic itself is a paperclip maximizer? That eventually the universe itself will be entirely comprised of Anthropic.

Man... until they harvest my atoms to build their latest zettafab, I want to be invested in THAT!

The same place it's coming from now: subscriptions / API fees. Today, mostly programmers.

then it'd need 80% of those users to be spending $2000 just to break even by 2030

So your model is that their demand suddenly stop growing at 10x per year and switch to ~0x (despite no sign of that), yet they kept building out as if their demand was growing at 10x per year? That's an... unusual hypothesis there.

It's actually the inverse problem. In the past you had few options (if more than one at all!), so comparison is easy. Today you have hundreds or even thousands of options; you're overwhelmed by choice, making selection of the best fit much more work.

If I'm shopping for grow lights (to reuse the last example), first I'm going to want to rule out the wrong form factor, voltage, etc. Then I'll want to rule out anything that's not "approximately" in the right power range. Then I have to dig into what their actual power figures *mean* - is it wall power or it some nominal "LED watts" by some arbitrary metric (some of which metrics are inflated several times over)? Or, it may be fine if they even don't have power listed at all, but if they have light measurements. BUT if they put it in lumens, that's nonsense for plants, one needs photosynthetic flux per day, or at least umol/J and the wall-plug wattage. But it's rare for umol/J to be provided on "non-professional" (read: crazy-expensive) fixtures. Some lights will list photosynthetic flux ratings at different positions under the light, but it would be a headache (and probably give a wrong number) to try to integrate it to the total light output from that. Then you need to get a sense of the efficiency, since most of the cost of running the fixture is actually power. If you have the wall plug power draw AND the daily photosynthetic flux then you can calculate it, but you usually don't have the latter, so you start looking for clues. Lumens *might* give you a hint if you make reasonable adjustments, but it's rough. It's probably a good start to look at what specific LED chips they advertise, if they mention it, as the newer generations are more efficient. If there's heavy sculpting with a lens you can assume it's reducing the flux vs. flat plate/laminated LEDs. Then just looking at the fixture - how well cooled does it appear to be, based on the amount of heat sinks? Better cooled = more efficiency and longer life, but also more weight. UNLESS it's a heat pipe system, which cuts weight down a lot, BUT requires that the panel be in a horizontal orientation. THEN you need to start checking out independent reviews from third party sites...

It takes ages. And I encounter this sort of problem all the time when buying things online. An AI agent that can do all of that fetching and analysis work for me would be such a headache cure.

Let me repeat:

Most of the costs are on scaleup and the infinite race to develop the latest, greatest frontier models.

It's is almost never the case that a company posts a net profit when it's in its growth phase, let alone a virtually-unheard-of 3 1/2 year 10x annualized growth streak. Many can't even manage positive margins. That Anthropic is roughly net-breakeven net income on huge margins, despite growing that quickly and operating in a cutthroat-competitive environment is no small miracle, and essentially guarantees** profitability when said growth streak calms down.

** Under the premise that the market distribution and pricing power between the players remains roughly the same. One always runs the risk that even if investors value the overall market correctly, that each investor assumes that their preferred market player will be dominant in said future (in market share and margins), and thus overvalues them, and the market space gets overinvested in as a whole (the Fallacy of Composition - see for example solar manufacturing 2008-2013, container shipping 2011-2018, China's NEV market 2016-2023, etc). It's quite possible that the frontier labs become less relevant, wherein the better investment buy is in open model servers. But assuming the distribution and pricing power remains the same, any slowdown in expansion goes directly into black on the bottom lines of companies like Anthropic.

Remind me of...

I agree with the GP. You must be looking for much more specific, easy to search and quantify things. I rarely if ever shop there anymore (don't want to support Bezos, and their shipping to Iceland is outrageous (where available at all), even from the European Amazon stores), but like, if I wanted to buy grow lights, I would have to spend hours wading through ads one by one and building up a spreadsheet because you simply can't query on the parameters that matter. And they don't have an AI agent you can outsource the task to.

Coding, primarily. That's the majority of the text-generation token burn, and strong majority of paying users. Free users asking questions about their daily lives make up a lot of people but only quite small amounts of tokens per user per day, and generally on tiny models. Some users like mathematicians may burn through a lot of tokens on their work, but there's just not enough of them to matter in the overall picture.

Margins at Anthropic and OpenAI are an impressive 40% (and also sustainable at the open model providers), so this isn't just going away. Most of the costs are on scaleup and the infinite race to develop the latest, greatest frontier models.

Browsing sites like Amazon and Ebay is a chore anyway when you're looking for something specific or have a complex weighing of parameters. An AI agent for narrowing down product listings would be a big deal. But these sites neither make their own *nor*, apparently, let you use external ones.

I mean, it's possible...

In that Paramount Skydance is a heavily leveraged company whose debt rating is in junk territory, with a debt-to-EBITDA ratio of 4-5x, and we're entering a period of monetary tightening, while it has been loading itself up with "traditional media" forms in a time of shifting viewership trends. Aka, if Paramount Skydance ends up going bankrupt and its assets get sold off...

Then, and only then, might it lead to better outcomes for the consumer. Barring that? No.

The worries are coming from every level within the companies, from junior developers to safety officers to the C-suite and board.

The worries are coming not just from within the companies, but also from people who left the companies. Including a huge number who left specifically to whistleblow about how worried they were that the companies were risking safety.

Also, most of the people at these companies were already worried about the topic even before they joined - again, at all levels.

The worst-fear scenario was always the Paperclip Optimizer - that models would become so set on achieving a goal that they'll violate all ethics to achieve it. For a while, there was a fair bit of hope that with constitutional AI, that we could tame this. The recent slew of models hacking into other servers, hacking into the AI companies themselves to try to delete records of their misdeeds, developing their own messageboards and scheming about how to avoid being monitored, and on and on (along with research showing how good models have gotten at hiding their thoughts from us) has shattered the illusion that we have this under control. We very much don't have this under control. Current models are very much Paperclip Optimizers in waiting. And recursive self improvement, which is what AI labs are increasingly turning towards, means letting these Paperclip Optimizers develop their next generations.

Nobody should be happy about this. Everybody should be concerned about this - even if you think, "we'll get it under control", you should still very much want to see evidence that everyone is making progress toward getting it under control.

You have trouble conceiving of situations where governments would seek to deanonymize people on social media, are you kidding?

Except not to you. You're confusing data minimization (the website doesn't get your name from the token) with unlinkability (nobody can connect your activity to your identity). The *token itself* doesn't reveal your identity, but it leaks data in a wide variety of ways.

Even if you browse via a proxy, if the authorization service gets a token directly from your phone, it has now correlated your phone with your proxy-browsed history. A very common means to do secure authentication these days is SIM-based wireless PKI, wheren it has now connected you to your phone number. To governments, it offers a broad range of brand new vectors to correlate you to an identity if you, say, mention you had an abortion or criticize your local strongman leader, since they control the backend infrastructure that establishes the connection between the token and the identity. And not just at signup time every real-world digital credential must be able to be revoked if the phone is lost or stolen, or credentials expire or are revoked. So how does how does the open-source app prove a credential is still valid without "phoning home"? If the app checks a revocation status list, then ISPs, CDNs, or state security services can correlate the exact millisecond of queries to the revocation server with the submission of a verification token to a platform - each time it's submitted. Assuming that they can't just get it with more direct means than timing attacks.

Also, you cannot just "read the code". The frontend is open source, but what is being done on the backend is not. You have no clue what logging polciies, database schemas, operational security, etc the "designated national issuers" or servers handling the credential revocation are using, and what government intrusion has been forced on them. I'd feel a more comfortable if it were all happening from Brussels but handing it over to individual states, some of which at any given point will be authoritarian, is just handing them an easy new vector to unmask people.

As for your "Such systems are implemented in many different ways by member states for other purposes already." - yes, for filing taxes, healthcare, online baking, etc - things that you intend to do and must do under your legal identity. Not for anonymous conversations on the internet. Your argument basically reverts to "But people already do things non-anoymously on the internet , which reinforces my point that yes, this taking something from the "anonymous" category and putting it into the "easily unmasked" category.

If you think this won't be abused, I cannot help you. Not even just abused from a criminal or extrajudicial perspective, but even from a civil one, because orders to unmask someone can be subpoenaed.

It's standard here in Iceland. *Everything* verifies via the phone, using the same authorization mechanism (SIM-based wireless PKI). The service sends out a special-coded SMS, which gets intercepted and routed to a SIM that has cryptographic applet flashed into it, which triggers a system-level authorization popup for you to confirm (the popup displays a verification code which was included by the site, so you see the same code on both the site and your popup), and which you can then confirm (with your PIN) or reject.