output is about 6 tokens/s with 16k context window i'm not having any issues since it went live this afternoon. it's not sparkling like opus 4.5/6 but gets the job done

i generally send it a voice note via telegram while driving and then check back in like 1-2 min, or it is sending me a reminder about something on our shared calendar. it's still faster than texting my buddy about making plans for this weekend or whatever.

I'm using a $200 used ~5 year old (from the ebay listing) HP EliteDesk 805 G6 DM Desktop Ryzen 5 PRO 4650GE 3.3GHz 32GB RAM 512GB SSD WiFi in cpu mode... you don't need a gpu to run single user local LLM... just a bunch of ram. This isn't 2022 anymore

It's about 5 tokens/second which is totally fine for an async assistant. 20 tokens/second is about the lower limit for usable in realtime. You can also set it up to use a smaller model for quick questions (what are the next 6 items on my calendar/to-do list?) and drop through to the bigger slower model for harder questions (can you add this feature to my internal ticketing system and redeploy?)

I ordered 64gb of ram about an hour ago and i'm planning on running either qwen 35B-A3B 8 bit or 122B-A10B 3 bit in fully offline mode.
 
  >the actual cost of 'running the AI.'
 
is a fixed $200 cost (ram upgrade) + electricity

I cancelled my subscription overnight, and I'm using the free credits they gave me to wrap up some things and transition away. I am not going to be locked into someone's walled garden again.

twosat confessed:

Perhaps nobody posted that link because it's not funny. Oh it wants to be funny, and it tries to be funny - but it fails to be even mildly humorous for three long, boring minutes.

You're welcome ...

I believe "Y'all aren't ready for that" was her response when asked if she would run for president.

They make household filters specifically designed for microplastics now. I don't know how well they actually work.

Sure, but can you go "Berlin to Warsaw in one tank" using renewables?

TDS stands for "Tiny Dick Syndrome"... right?

Obviously, to go to 100% renewables we need a grid upgrade and far more energy storage. Sodium batteries look promising for that, but they need to ramp up production enough to drive the per-unit cost down.

I wonder if anybody regrets buying those big diesel bubba trucks to coal roll the Teslas with now...

Trump wants all the wind turbines and solar panels torn down, so we can use more "beautiful clean coal"!

People used to complain that my Setter was too skinny, even though she had unlimited kibble supply from a 2 gallon dispenser. I figured out years later when I found one of her teeth on the floor that she had abscesses on her teeth, which likely made it hurt to eat kibble. But she never complained about it, so I never knew. Turns out I should have been spending the money for wet food. I miss that dog, she only lived for 12 years.