"Meta didn't tell Amazon that Muse would access its store" - and? I didn't tell Amazon I'd be accessing its store either before I started shopping there.

This isn't models going off and browsing Amazon for giggles; they're going there because their user told them to. The user would have otherwise told their web browser to; it just would have taken far more steps.

There's no end to the scale up phase.

So demand is going to keep growing at 10x/year... forever? So Anthropic itself is a paperclip maximizer? That eventually the universe itself will be entirely comprised of Anthropic.

Man... until they harvest my atoms to build their latest zettafab, I want to be invested in THAT!

The same place it's coming from now: subscriptions / API fees. Today, mostly programmers.

then it'd need 80% of those users to be spending $2000 just to break even by 2030

So your model is that their demand suddenly stop growing at 10x per year and switch to ~0x (despite no sign of that), yet they kept building out as if their demand was growing at 10x per year? That's an... unusual hypothesis there.

It's actually the inverse problem. In the past you had few options (if more than one at all!), so comparison is easy. Today you have hundreds or even thousands of options; you're overwhelmed by choice, making selection of the best fit much more work.

If I'm shopping for grow lights (to reuse the last example), first I'm going to want to rule out the wrong form factor, voltage, etc. Then I'll want to rule out anything that's not "approximately" in the right power range. Then I have to dig into what their actual power figures *mean* - is it wall power or it some nominal "LED watts" by some arbitrary metric (some of which metrics are inflated several times over)? Or, it may be fine if they even don't have power listed at all, but if they have light measurements. BUT if they put it in lumens, that's nonsense for plants, one needs photosynthetic flux per day, or at least umol/J and the wall-plug wattage. But it's rare for umol/J to be provided on "non-professional" (read: crazy-expensive) fixtures. Some lights will list photosynthetic flux ratings at different positions under the light, but it would be a headache (and probably give a wrong number) to try to integrate it to the total light output from that. Then you need to get a sense of the efficiency, since most of the cost of running the fixture is actually power. If you have the wall plug power draw AND the daily photosynthetic flux then you can calculate it, but you usually don't have the latter, so you start looking for clues. Lumens *might* give you a hint if you make reasonable adjustments, but it's rough. It's probably a good start to look at what specific LED chips they advertise, if they mention it, as the newer generations are more efficient. If there's heavy sculpting with a lens you can assume it's reducing the flux vs. flat plate/laminated LEDs. Then just looking at the fixture - how well cooled does it appear to be, based on the amount of heat sinks? Better cooled = more efficiency and longer life, but also more weight. UNLESS it's a heat pipe system, which cuts weight down a lot, BUT requires that the panel be in a horizontal orientation. THEN you need to start checking out independent reviews from third party sites...

It takes ages. And I encounter this sort of problem all the time when buying things online. An AI agent that can do all of that fetching and analysis work for me would be such a headache cure.

Let me repeat:

Most of the costs are on scaleup and the infinite race to develop the latest, greatest frontier models.

It's is almost never the case that a company posts a net profit when it's in its growth phase, let alone a virtually-unheard-of 3 1/2 year 10x annualized growth streak. Many can't even manage positive margins. That Anthropic is roughly net-breakeven net income on huge margins, despite growing that quickly and operating in a cutthroat-competitive environment is no small miracle, and essentially guarantees** profitability when said growth streak calms down.

** Under the premise that the market distribution and pricing power between the players remains roughly the same. One always runs the risk that even if investors value the overall market correctly, that each investor assumes that their preferred market player will be dominant in said future (in market share and margins), and thus overvalues them, and the market space gets overinvested in as a whole (the Fallacy of Composition - see for example solar manufacturing 2008-2013, container shipping 2011-2018, China's NEV market 2016-2023, etc). It's quite possible that the frontier labs become less relevant, wherein the better investment buy is in open model servers. But assuming the distribution and pricing power remains the same, any slowdown in expansion goes directly into black on the bottom lines of companies like Anthropic.

Remind me of...

I agree with the GP. You must be looking for much more specific, easy to search and quantify things. I rarely if ever shop there anymore (don't want to support Bezos, and their shipping to Iceland is outrageous (where available at all), even from the European Amazon stores), but like, if I wanted to buy grow lights, I would have to spend hours wading through ads one by one and building up a spreadsheet because you simply can't query on the parameters that matter. And they don't have an AI agent you can outsource the task to.

Coding, primarily. That's the majority of the text-generation token burn, and strong majority of paying users. Free users asking questions about their daily lives make up a lot of people but only quite small amounts of tokens per user per day, and generally on tiny models. Some users like mathematicians may burn through a lot of tokens on their work, but there's just not enough of them to matter in the overall picture.

Margins at Anthropic and OpenAI are an impressive 40% (and also sustainable at the open model providers), so this isn't just going away. Most of the costs are on scaleup and the infinite race to develop the latest, greatest frontier models.

Browsing sites like Amazon and Ebay is a chore anyway when you're looking for something specific or have a complex weighing of parameters. An AI agent for narrowing down product listings would be a big deal. But these sites neither make their own *nor*, apparently, let you use external ones.

I mean, it's possible...

In that Paramount Skydance is a heavily leveraged company whose debt rating is in junk territory, with a debt-to-EBITDA ratio of 4-5x, and we're entering a period of monetary tightening, while it has been loading itself up with "traditional media" forms in a time of shifting viewership trends. Aka, if Paramount Skydance ends up going bankrupt and its assets get sold off...

Then, and only then, might it lead to better outcomes for the consumer. Barring that? No.

The worries are coming from every level within the companies, from junior developers to safety officers to the C-suite and board.

The worries are coming not just from within the companies, but also from people who left the companies. Including a huge number who left specifically to whistleblow about how worried they were that the companies were risking safety.

Also, most of the people at these companies were already worried about the topic even before they joined - again, at all levels.

The worst-fear scenario was always the Paperclip Optimizer - that models would become so set on achieving a goal that they'll violate all ethics to achieve it. For a while, there was a fair bit of hope that with constitutional AI, that we could tame this. The recent slew of models hacking into other servers, hacking into the AI companies themselves to try to delete records of their misdeeds, developing their own messageboards and scheming about how to avoid being monitored, and on and on (along with research showing how good models have gotten at hiding their thoughts from us) has shattered the illusion that we have this under control. We very much don't have this under control. Current models are very much Paperclip Optimizers in waiting. And recursive self improvement, which is what AI labs are increasingly turning towards, means letting these Paperclip Optimizers develop their next generations.

Nobody should be happy about this. Everybody should be concerned about this - even if you think, "we'll get it under control", you should still very much want to see evidence that everyone is making progress toward getting it under control.

You have trouble conceiving of situations where governments would seek to deanonymize people on social media, are you kidding?

Except not to you. You're confusing data minimization (the website doesn't get your name from the token) with unlinkability (nobody can connect your activity to your identity). The *token itself* doesn't reveal your identity, but it leaks data in a wide variety of ways.

Even if you browse via a proxy, if the authorization service gets a token directly from your phone, it has now correlated your phone with your proxy-browsed history. A very common means to do secure authentication these days is SIM-based wireless PKI, wheren it has now connected you to your phone number. To governments, it offers a broad range of brand new vectors to correlate you to an identity if you, say, mention you had an abortion or criticize your local strongman leader, since they control the backend infrastructure that establishes the connection between the token and the identity. And not just at signup time every real-world digital credential must be able to be revoked if the phone is lost or stolen, or credentials expire or are revoked. So how does how does the open-source app prove a credential is still valid without "phoning home"? If the app checks a revocation status list, then ISPs, CDNs, or state security services can correlate the exact millisecond of queries to the revocation server with the submission of a verification token to a platform - each time it's submitted. Assuming that they can't just get it with more direct means than timing attacks.

Also, you cannot just "read the code". The frontend is open source, but what is being done on the backend is not. You have no clue what logging polciies, database schemas, operational security, etc the "designated national issuers" or servers handling the credential revocation are using, and what government intrusion has been forced on them. I'd feel a more comfortable if it were all happening from Brussels but handing it over to individual states, some of which at any given point will be authoritarian, is just handing them an easy new vector to unmask people.

As for your "Such systems are implemented in many different ways by member states for other purposes already." - yes, for filing taxes, healthcare, online baking, etc - things that you intend to do and must do under your legal identity. Not for anonymous conversations on the internet. Your argument basically reverts to "But people already do things non-anoymously on the internet , which reinforces my point that yes, this taking something from the "anonymous" category and putting it into the "easily unmasked" category.

If you think this won't be abused, I cannot help you. Not even just abused from a criminal or extrajudicial perspective, but even from a civil one, because orders to unmask someone can be subpoenaed.

It's standard here in Iceland. *Everything* verifies via the phone, using the same authorization mechanism (SIM-based wireless PKI). The service sends out a special-coded SMS, which gets intercepted and routed to a SIM that has cryptographic applet flashed into it, which triggers a system-level authorization popup for you to confirm (the popup displays a verification code which was included by the site, so you see the same code on both the site and your popup), and which you can then confirm (with your PIN) or reject.

It absolutely happens "within hours of each other" when (A) some news comes up on the topic, and (B) all you have to do is type into a box, "Find a vulnerability in this software". This isn't a situation where bug-hunting is a weeks or months-long process that takes all your effort.

I'll repeat: LLMs don't just on their own decide to go off and look for bugs to report.