Not talking about VRAM or unified memory. Regular DRAM.

There's been a lot of advancement recently on how to best manage MoE expert swapping (FreeToken is just one example).

How are you connecting to Slashdot from the year 2024?

If you have ~128 GB of RAM, consider Qwen 3.8 Flash + Freetoken. You'll only get half the tok/s as 27B but the quality is significantly higher than 27B - it beats it solidly in all compared benchmarks (for example, DeepSWE 1.1: 42,2% vs 58.7%).

Grok is a "discount model". It's cheaper than Claude Sonnet yet better than it. Doesn't compare to Fable or Opus, but then again, they're way more expensive models.

Note: I avoid using Grok - not just for the Musk factor, but because there are better options, e.g. GLM-5.3 and Qwen-3.8. Also, if you really want to save money, GLM-5.3 Flash and Qwen-3.8 Flash are amazingly good for their price range.

[ConfusedMathLady.gif]

It's more a case of the fox being in charge of the chicken coop; or bank robbers gaining control of the bank.

The job of the press in a democracy is to hold officials up to scrutiny.

Like for example Bill Atkinson's famous productive -2000 lines: https://www.folklore.org/Negat...

For decades I've been suggesting on-and-off that a big problem is too many commercial programmers writing too much code and making too much work for each other with an endless proliferation adhoc interfaces, each a new "standard".

As an egregious example, years ago I read somewhere that Microsoft Office embeds like six entirely different virtual machines. Maybe it has improved since with the CLR?

Except for proprietary competitive forces, how many spreadsheets, compilers, VMs, email clients, word processors, and databases do we really need? Even in the realm of games, how many game engines do we need? Granted, game content may be a different story, but even there how many low-to-medium-quality games do we need?

The same is true to an extent with the entire internet. How many low-to-medium-quality comments do we need to wade through on various topics?

Imagine if everyone was using, say, (optionally-typed) Scheme with a great set of libraries to write everything (including with "org mode" in a Scheme-powered emacs-like multi-tool with a calendar etc), and there was just no "JSON" or "JavaScript" or "XML" or "Office-compatible adhoc file formats" or even "C" or "Rust". Not saying that an optionally-typed Scheme is the best choice, but what a different computing world to imagine. Guess my Symbolics 3600 Lisp machine experience in the 1980s spoiled me. :-) And I have to admit that TypeScript on widespread Scheme-inspired JavaScript VMs with WebAssembly has moved the computing world a lot in that direction with pros and cons.

I know an obvious objection (as with Linux's endless front-end and distributions) is about choice, but at some point there is a tyranny of excessive choice, like "twisty little passage ways all alike".
https://en.wikipedia.org/wiki/...
        "The Paradox of Choice - Why More Is Less is a book about overchoice written by American psychologist Barry Schwartz and first published in 2004 by Harper Perennial. In the book, Schwartz argues that eliminating consumer choices can greatly reduce anxiety for shoppers. The book analyzes the behavior of different types of people (in particular, maximizers and satisficers). This book argues that the dramatic explosion in choice--from the mundane to the profound challenges of balancing career, family, and individual needs--has paradoxically become a problem instead of a solution and how our obsession with choice encourages us to seek that which makes us feel worse."

I have no problem with people writing personal code for fun or learning or personal growth (which I highly recommend) -- the same way a person may play the piano as a hobby without expecting every time they play a tune the result needs to become a packaged commercial musical product.

The main thing I think about these days is how to have a much simpler tech ecosystem. Including for this reason outlined by Rich Hickey (author of Clojure) in "Simple Made Easy":
https://www.infoq.com/presenta...
====
Rich Hickey emphasizes simplicity's virtues over easiness', showing that while many choose easiness they may end up with complexity, and the better way is to choose easiness along the simplicity path.

We should aim for simplicity because simplicity is a prerequisite for reliability.

Simple is often erroneously mistaken for easy. "Easy" means "to be at hand", "to be approachable".

"Simple" is the opposite of "complex" which means "being intertwined", "being tied together". Simple != easy.

What matters in software is: does the software do what is supposed to do? Is it of high quality? Can we rely on it? Can problems be fixed along the way? Can requirements change over time? The answers to these questions is what matters in writing software not the look and feel of the experience writing the code or the cultural implications of it.

The benefits of simplicity are: ease of understanding, ease of change, ease of debugging, flexibility.

Complex constructs: State, Object, Methods, Syntax, Inheritance, Switch/matching, Vars, Imperative loops, Actors, ORM, Conditionals.

Simple constructs: Values, Functions, Namespaces, Data, Polymorphism, Managed refs, Set functions, Queues, Declarative data manipulation, Rules, Consistency.

Build simple systems by:
* Abstracting - design by answering questions related to what, who, when, where, why, and how.
* Choosing constructs that generate simple artifacts.
* Simplify by encapsulation.
====

Or, as a joke:
"Weinberg's Law: If builders built buildings the way (current-AI-enhanced-HR-staffer) programmers wrote programs, then the first woodpecker that came along would destroy civilization."
https://quoteinvestigator.com/...

Ed Zitron is a moron who has predicted the past nine out of zero AI crashes, who simultaneously gets paid to hate on AI while also getting paid to promote AI (DoNotPay).

The funny thing is that he openly acknowledges that inference is very profitable (~40% margins) for all parties - he just argues that you should count training. But that argument makes no sense, because if capital tightens across the AI industry, then everyone will cut back on training and scaleup, and... then what? Where's the implosion? Everyone is profitable then. You may lose certain players in competition against others (their as-mentioned profitable assets being gobbled up by others), but the industry remains healthy.

There are more dramatic types of possible contagion risk, but they all end up with *someone* holding onto the high-profit-generating inference assets, even if it's high-seniority investors, leaving the the low seniority investoors high and dry.

The enshittification of Cursor continues.

Maybe this will force them to support more open models.

Inference is neural network limited. On systems with entirely separated data and compute, this is a memory bus constraint. In quantized systems, ALUs and FPUs are illogical. A simple 16 cell LUT is suitable for noise free multiplication. A neural network circuit with a dedicated LUT and addressable buffers more or less eliminates bandwidth constraints. But this requires neural networks more similar to FPGA cells rather than classic compute.

That said, separating inference from data drops network depth considerably. And in transformers depth increases memory access exponentially which is why nearly identical MoE and dense models perform so differently. We of course need data centers for models with massive numbers of active parameters. But massive numbers of active parameters weaken models as that means depending on training rather than looking up facts. Huge active parameter sets will never be a good design. A real world analogy is that huge models are like Jeopardy champions who have crap loads of partial trivia facts in their heads. Smaller models are like librarians who don't have a trillions useless facts memorized but knows how to find the data for the researchers who will use the librarian repetitively to follow through to the answer. The model only needs enough training to use the data.

What really matters is, how fast it can find the data.

Inference suffers greatly when you treat weights as data. And bigger models have more trivia like facts memorized from their training. This means more active parameters and greater network depths with higher memory bandwidth needs. Even now, you see the chatbots are improving drastically because they rely substantially more on data and way less on inference.

I am not sure which part you consider gibberish, but it makes perfect sense to me. But it might sound better in my head.

P.S. I actually have considerable data to backup many of my claims. At work I'm sitting on about 50MW of compute. We build in shipping crates in a repurposed mine. There is a group of us where our goal is to avoid spinning up more compute. We need to deliver inference to 200,000 users eventually. (not customers, employees) and we could throw millions at NVidia and end up hosting some crap model, but we focus instead on cutting that memory bandwidth need. And yes, I'm far from being the smart guy on the team.

Meanwhile, what those evil liberals think should happen to YOU is... that you be given healthcare and a general social safety net.

They've been launching dummysats from Starship for a number of launches now, so I'm not sure why you think they're "forever away". The only reasons they've been using dummysats and not actual satellites is 1) Not wanting to risk more valuable payloads thusfar, and 2) wanting to keep Starship suborbital (e.g. trajectory that reintersects the atmosphere) at just under orbital velocity until they can verify deorbit reliability (failure to do so would leave that massive ship falling onto a random point on Earth).

If they were to choose to, they could start launching payloads on all subsequent Starship missions, even if they're not yet ready for reliable recovery or quick reuse. In fact, given the killing off of Falcon 9 in Florida, one can presume that's the plan (if not immediately, then soon).

Um... no they're not?

Further reading: Musk's other unstated reasons for shutting down Falcon 9 in Florida (Behind The Black)

TL/DR, the author thinks that SpaceX shutting down Falcon 9 launches from Florida was secretly a magnanimous gesture by a charitable Elon Musk to boost the competition:

I therefore honestly believe that SpaceX’s decision was not entirely based on its own internal needs. Those needs certainly were the main motive, but based on Elon Musk’s passionate dream of making the human race multi-planetary, as well as his repeated willingness to help his competitors when they needed SpaceX to launch their satellites, I believe he recognized that SpaceX’s dominance could be hurting the rest of the rocket industry, and that the instant it made business sense to clear the field for these competitors, he was quite willing to do so.

LOL.

Yes, I remember way back when Mike Pence had nearly absolute unchecked authoritarian power.