That Mac Studio with a 2tb ssd is $7900 not $10K. The old 512GB was a little over $10K but they dropped that option. As for price as far as I know it hasn't gone up. The new M5 Max 128 didn't get a price increase over the M4 Max (with the same size SSD configured) so hopefully the next studio will follow the same pattern.

But yea if you want to run large models for a reasonable price it's the only game in town right now.

If you want to use it async that's fine as long as async means 10's of minutes or more between turns for development work. And it's not just tokens per second. Prefill is compute bound and is going to be very slow even compared to a low end GPU. Larger contexts are going to pressure the KV cache reads which will also impact tps, and coding generally uses lots of context each turn. It all adds up.

AFAIK there are no unified RAM systems that are good to run AI on that don't use soldered RAM. AMD tried it but couldn't get it to work.

I hope you looked into benchmarks for running models in system RAM vs VRAM.

You joke but following up with "Are you sure?" will trigger a lot of models to go verify their output with external sources like web searches.

Funny enough, this is one place where the fix is easy but probably not cheap. They just need to build guardrails that automatically checks any case law references against something like LexisNexis and feeds back to the AI if it makes something up. Case law is extremely well documented and fairly structured in how it's indexed. You wouldn't even need to use AI for the lookup, a competent traditional search algo would work. Of course that's going to be expensive since it will require access to the case law data electronically and SOMEONE is going to make bank on that. It's not something you will get with a $20 ChatGPT subscription.

It baffles the mind that Microsoftware - known for decades for being unreliable shit - is allowed on space missions at all, no matter how uncritical the role. The potential for malware alone is ludicrous. "Hey, pay us 2500 bitcoins if you want your space capsule back".

Then again, I figure the days when NASA did the right stuff are long past.

No. The entire model is loaded into memory but the feed-forward layers are split into subnetworks. A router picks a few of the best "experts" on a per-token basis to activate and those are the 3.8B that are activated. It's a way to increase inference performance so you get the speed of a dense 3.8B model with the output quality close to (but not equal to) a 26B dense model.

This is so true, so true.

And it's not even US specific. In the wake of the Ukraine war, German parliament voted to give itself 100 billion of additional taxpayer money (i.e. debt) to spend on defense. Recently a report came out of all the money spent so far, 90% did not go towards the intended purpose.

Why any of the jokers in charge of our governments are still not in jail baffles me more and more every year. Oh yes, it's because they make the rules, sorry, my bad.

That's what campaign donations get you, if they are large enough.

This is why congress occasionally bullies the big tech companies. We all think they might want to have some regulation or to punish them. Oh sweetie... they're saying "nice company you have there... would be a shame if something happened to it..."

The best time to leave github was when the evil empire bought it. The second best time is now.

Seriously. Anyone who thought MS wouldn't fuck it up in the same way they fuck up everything they touch can't be helped. It's Microsoft for crying out loud.

Don't care. If your shit shows me ads, it's not getting into my kitchen. Note to self: Don't buy appliances from Samsung anymore.

Yes, I am vocal in how much I hate ads. I believe the CEOs of advertising companies should get one hit with a stick for every time their ad bothered someone even in the slightest.

Exactly what I'm saying.

The fact that users and enterprise customers are not demanding better software from Microsoft with the same fervor their ancestors demanded that the witch be burnt speaks volumes.

And I'm specifically talking about operating systems here. Software can crash for all I care. I'm fine software quality being all over the place, the market can sort that out. But operating systems are natural monopolies and the foundation for everything else. We should not accept shoddy quality there.

Users should never be able to do things that cause crashes in the same way that drivers should not ever be able to press any button or press any pedal that causes the engine to spontaneously burst into flames.

I'm also a Mac user, but let's not boast here, shall we?

My personal guess would have been at least 10x. Did Microsoft bribe the study authors?