Comment Re:American Open Weight Models (Score 1) 108
Any quantization beyond Q4_0 goes mad (losing context, making category errors, malforming markup tags, mangling and inventing words) long before the context reaches a size I would consider "useful". I've tried running 31B at IQ3_XS and it looks alright from a distance, for a few prompts at a time. Unfortunately the errors compound and semantic drift goes into runaway. I have no interest in playing with quantizations any smaller than that. The only thing "massively wrong" is the suitability of a 2-bit quantization for my actual use. I've investigated it, and it can be an amusing toy, but it is far from usable for any meaningful work.
Even Q4_K_M was too error-prone for meaningful use, and I was using Q5_K_M as my baseline of "usable for a while". Fortunately, most of the heavily quantized models are based on QAT releases now, and a 4-bit QAT model is pretty much on par with a 5-bit non-QAT model.