Comment Re:What is this bullshit? (Score 1) 93
Should also be modded funny and "Yes, Virginia, there is a coverup." Even Santa Claus knows that much.
Should also be modded funny and "Yes, Virginia, there is a coverup." Even Santa Claus knows that much.
Also, there's a number of literally free models you can use. They're not as good and generally rate limited, but they're better than nothing.
It doesn't matter how cheap they are when their use is effectively unbounded. Slop purveyors open 1,000 bug reports or PRs?
These models typically cost on the order of 10-20 cents per MILLION tokens in (assuming cache misses!) and ~50 cents per MILLION tokens out. 1/100th the cost of Fable or Astra, for a task that's far less token-hungry than coding. 1000 bug reports or PRs is perfectly handleable. It simply is not an issue.
Secondly, if there's evidence that a bad actor is involved, a model is perfectly capable of asking you if you want to ban them or banning on its own if you give it permission. And if they're behaving in a way that makes them hard to ban (sockpuppeting, etc), likewise asking you / acting on its own to report the attack.
Where do we send our funding requests?
Right here. Note I'm not going to write your harness for you, but if you come back to me with a harness and a serious proposal, I'll totally look at it.
What's worse than trusting an AI oracle too much?
Trusting an AI oracle that knows nothing but has a high priority to flatter you and tell you whatever it "thinks" you want to hear.
Worst: The AI oracle has hidden "motives" coming from (1) a sociopath with infinite greed or (2) a fascist government or (3) both.
Scare-quoted words may not mean what you think they mean. But you can ask your favorite AI to interpret them for you.
I can only find a single example in all of history of a person who was fooled by ELIZA, and only in a "he thought it was a remote-working employee taking the piss" way, not in a "he thought a computer was alive" way.
This notion that ELIZA mass-fooled people is simply not in line with the actual reality we live in. Weizenbaum pushed it heavily in his later life because he was an anti-AI zealot (driven in part due to anger about the use of computers for targeting during the Vietnam War, but also because some people started seriously considering the use of a tool like ELIZA/DOCTOR for talk therapy), with his main example being claims that because his secretary wanted to continue using it and asked him to leave the room, she had been stupidly duped into thinking it was alive. Of which Weizenbaum's own daughter later commented:
I always heard that just really kind of soaked in arrogance. Like, how could she be so stupid to think that this is actually a meaningful communication? Here you have somebody who so needs a landing pad for her feelings that she’s willing to embrace it, right? And he totally misses that, he totally misses the human need and instead speaks of just evidence of this person’s stupidity rather than evidence of their humanity.
Literally, before you ever mention ELIZA again, actually try the thing. For more than 30 seconds. The illusion falls apart really, really quickly. It's not fooling anyone. What actually happened is that people tend to break into two groups. One deliberately tries to get the bot to say ridiculous things to prove what it's doing and show it off as the cheap artifice that it is. The other does just the opposite, "role-playing" and deliberately trying to avoid saying things that would cause the bot to break the roleplay, to try to get the conversation to last coherently as long as they can. Neither group is "tricked".
I'm disappointed by how little my toaster talks, and I need to remedy that situation.
If you think you can just type "Make a LLM that can run on a floppy disk", leave your session sitting there for a while with zero interaction, and come back and it's done and distributed, I have some terrible news for you...
"Tell Me You've Never Used ELIZA Without Telling Me You've Never Used ELIZA"
Also, can we talk for a second about how cool computers used to look?
Basically, the Cray X-MP/2 had 16-32 MB of fast RAM and a then-absurd ~5 GB/s memory bandwidth and a solid state "disk" (actually MOS RAM cache) with 256MB-1GB of storage and a 100-1000 MB/s bandwidth / sub-25-microsecond latency. The DD-39 / DD-49 disk arrays could hold 1,2GB per drive, up to 38GB, 6-10MB/s unstriped or ~50MB/s striped. Honestly, you could readily go with a far larger number of total params, so long as you kept the active params small enough to fit in RAM. An architecture like Freetoken could significantly improve performance as well. Performance would be limited by the 400 MFLOPS of compute, not the bandwidth (really, apart from the fact that all math was 64 bit float, it wasn't a bad architecture for inference!). You'd want to do token batching and maybe add a speculative decoding model to boost performance further.
Of course, you could never train a model like that on the Cray!
The funny thing is, if we could send the weights and inference back in time, a 1983 Cray X-MP/2 running a 300M–400M total / ~35M–40M active param MoE - probably similar to the reasoning capability of the computer in Wargames - could probably output 1-3 tok/s (or make an equivalent number of reasoned decisions per second with a Jev-like decision head). In Wargames, the computer seems to output at about 3 tok/s.
Who knew that that movie was technologically plausible?
ED: Looking into it, the state of the art for actually-useful RAG is still about 60-100M params, so still way off from what you can fit on a floppy disk. Quantized, you're looking at maybe 25 floppy disks, 30-50MB at 4-bit compressed.
For running it comfortably, a '96-98 era Pentium MMX / Pentium II fully-in-ram with 64-128MB ram and a 1-4GB hard drive should be easy and run at a workable (go brew a cup of coffee
For at least being able to fit it in RAM, you're looking at the 1987-1990 era - a Mac II with the FDHD could support up to 64 MB of RAM, and a 386DX could handle 32-64MB on some motherboards. It would have cost tens of thousands of dollars and taken like half an hour per answer.
If you're really damned patient and willing to stream from disk, we can go back to 1984-1986 (IBM PC AT / XT + 40MB HDD). It could just keep the current hidden state and a tiny 640k weight buffer in RAM and stream the model layer by layer. Prefilling a tiny 300 token passage would take like 2 days, and a 40 token answer would take like 5 hours. On a 4.77 MHz 8088 PC, you're talking maybe 1-2 weeks per query.
And if you're a masochist... technically you could stream straight off of manually-swapped floppy disks! Then you could run it on a 1977-1982 Apple IIe or Commodore 64
A "tiny" LLM is generally considered to be about 1B params or less. This one is 366k params.
Of course, it seems to be immensely narrowly trained: "Trained on about 3,500 facts, each asked in many wordings Outside those it makes things up confidently"
What would be more interesting, IMHO, would be to try to train a RAG model. Something that is trained on *no* facts, merely on converting your statements into search terms, and turning search results back into answers. I wonder how small you could get that? I've often wondered how far back we technically could have run an actually-useful LLM if we were to have had the weights and inference engine.
I don't think you understand how cheap these models are.
Combined with the low token usage compared to normal coding, it's a total non-issue. I'll pay for any major OSS project's token usage for project management myself if they ask, that's how little it is.
I think superintelligence has a well established meaning. It's the ENEMY of mankind. But why would the YOB care? His YUGE plan is to take the money, pardon himself (and all witnesses against him), run away, and then to boot it all he is probably sure that he'll be dead before any of his chickens try to come home. But jokes on the YOB if the malevolent superintelligence gets here before he can finish imploding/dying.
Time is the most valuable thing a man can spend. -- Theophrastus