"What you said earlier" was a lot of hot air about looking at spectrum plots instead of actually listening to the encodes in a controlled environment.

Did you ABX the 128kbps encode? If not, it's unsubstantiated. Plausible, sure, but not substantiated.

First and foremost, there's a practicality issue here. Listening tests are very hard to conduct - but they become considerably easier at lower bitrates. Everybody would love to see a 192kbps test or a 256kbps test or whatnot, but actually ABXing differences at those rates becomes so rare that it largely becomes a contest of which encoder outwits a set of brutally difficult problem samples that hardly ever occur in reality (or which encoders favor the artifacts heard by a very selective set of golden ears). The meaning of the results is thus compromised. At 128k the differences are significant enough to accurately compare encoders amongst themselves with more samples.

Second, low bitrates still matter for cell phones, flash players, iPhone/iPod Touches, etc... Applications shift as storage space increases. There will likely always be devices out there with under 30GB of storage space, and there will always be people who want to put 1,000 albums on said devices at high quality. So there will always be a use for low bitrates. (Heck, I don't even think my music collection would fit on my 60GB iPod at 256k!)

Third, bandwidth still matters for online music distribution. MySpace does most of its streaming with 96kbps CBR (blech!). While this test specifically isn't that useful for MySpace encoding, the general question - of how to eke out maximum quality for nominal bitrate - will be important for as long as bandwidth costs money.

Spectral music might make for great samples for this kind of testing, but your assertion is ultimately unsubstantiated unless you can provide real listening test results that show it makes for a more sensitive test. There are all kinds of subtle things going on that might seem to make for great encoder testing, but largely turn out to make an imperceptible difference. Just because so many overtones exist (99% of which do not exist in msot acoustic music, btw!) doesn't mean they are necessarily audible if they are perturbed. More specifically, I'd anticipate that most FFT-based imaging techniques would hammer encoder lowpasses very hard, but would not be nearly as hard on preecho performance or stereo imaging artifacts. In a lot of situations, the preecho is a lot more important than the lowpass.

That's great if you're trying to use a codec for a purpose it was never designed for and nobody actually uses. Would you choose a JPEG codec based on its ability to encode/decode raw audio? Would you choose a car based on its ability to traverse the English Channel?

A lot of very high quality encoder tuning has been done with $30 headphones on laptops. Concentration and patience is more important than equipment here.

The ABC/HR zip, and one sample zip. Each sample zip is a separate test that can be run completely separately from the others. Testing each sample may take quite some time (it took 1-2 hours for a single sample last night for me) - so splitting this up actually does make a bit of sense. That said, even on Windows this test has been plagued with problems. I've had to downgrade to Java 1.5 to avoid a crash.