Vorbis And Musepack Win 128kbps Multiformat Test

technology is sexy writes "After 11 days of collecting results Roberto Amorim today announced the results of his 2nd Multi-Format listening test: Vorbis fork AoTuV scored the highest and ranks as the winner together with open source contender Musepack closely followed by Apple's AAC implementation and LAME MP3, which improved markably since last year thanks to further tunings of its VBR model done by Gabriel Bouvigne. Sony's ATRAC3 format ranks last after WMA on the third place. The suprising success of AoTuV (compared to last year's performance of Xiph.org's reference implementation) shows the potential of Vorbis and possible room for further tuning and improvments. Take a look at the detailed results and their discussion at Hydrogenaudio.org."

Striving for innovation

So given Microsofts stated goal to bring us innovative technology, they should throw in the towel and ship OggVorbis and derivatives with Windows, right?

No

No, they won't. Their definition of innovation is making the same thing in an incompatible way.

Re:Striving for innovation

its important to point out two things especially:
WMA brought clearly worse quality than (good old) MP3 at 128kbps

itunes AAC brought clearly better quality than WMA at 128kbps

so why should anyone even a minute consider buying crap quality wma encodes at napster, coca-cola, walmart or however the wma-based music stores are called?

on the legal way -> itunes is better
on the illegal way -> even old mp3 (next to vorbis or aac) is better

Re:Striving for innovation

The Microsoft-published PC version of Halo uses Ogg Vorbis for all of its audio - so it's not as if there hasn't been a precendent. :-)

But does it matter?

When everyone gets an iPod, dood, or the WinFooTunes player that you get with your Dell only works with WMA, or your in-dash CD player only groks 128kbps MP3s, whats the practical application of the other codecs? It's nice that we propeller-heads on Slashdot can smirk while we rip everything to FLAC and write custom Perl apps to transcode-on-the-fly to our wireless enabled MythTV box, but for John Q. Drone^H^H^H^H^HConsumer, none of this matters.

So how do we get the word out? How do we start the revolution? Open-Source hardware?

Re:But does it matter?

I don't think we have to "get the word out." Most cool tech innovations make it into the mainstream if they are really good enough. I remember buying a CD-RW (4x2x2) for $400. Everybody thought I was stupid for spending so much on a piece of hardware. Later I spend $300 on a 64MB MP3 player. The guy at the desk told me that I shouldn't get a MP3 player because changing the media was really hard. Yet most people now have both of these gadgets. If Vorbis is license free and simple enough to put on a hardware chip then it will slowly gain support and slowly people will begin to see it.

Re:But does it matter?

There is a difference to this. At the time there was no alternative to the CD-RW. There are many compressed sound types that are being marketed.

Re:But does it matter?

You have a point. There are devices however such as from iRiver [iriver.com] which can play Ogg Vorbis and the winning encoder/codec in the Slashdot story AoTuV seems to be just an encoder fork which is bitstream compatible with Ogg Vorbis.

Re:But does it matter?

There are devices however such as from iRiver which can play Ogg Vorbis

I was one of those nincompoops who rushed out and bought one the moment I read the words "iRiver" and "Ogg" in the same sentence, but when I updated its firmware the latest version with Ogg Vorbis support, I found that many of my files wouldn't play.

It turns out that most of the iRiver players with Ogg support added have a half-baked implementation and support only a limited range of bit-rates and frequencies. The iFP-300 series, to which my player belongs, only supports 96Kbps - 360Kbps (if it's a VBR file and the bitrate drops above or below that, distortion occurs), and also has trouble with files encoded in less commonly used frequences (i.e. lower than 44.1KHz).

In case anyone like me thought iRiver was committed to improving their Ogg support, their latest iFP series players are even more limited, supporting only 96Kbps - 225Kbps at 44.1KHz. Their new iDP series doesn't support Ogg at all. And owners of the iMP have been waiting months for Ogg support which still has not materialised.

Only the H series supports a decent range of Ogg Vorbis bitrates, but even it only officially supports one frequency (44.1KHz).

Blind testing? And "Best sound" or "Accuracy"?

Of course it matters! I took a quick look, but I didn't seem to find anything describing how the tests were performed.

IMHO, the best way to test is to provide an uncompressed source and a variety of compressed files, and ask "which most closely matches the uncompressed source" -- and NOT "which sounds best."

Years ago, I did an a/b switch test with a high-end audio engineer between a CD and a 128kbit/s MP3. Though we could both clearly hear a difference, he actually guessed wrong.

My point is: the test needs to be blind, and the test should be looking for compressed files that most closely sound like the uncompressed original -- and not the ones that "sound best."

Why would you care about accuracy?

If a sound was perfectly accurate except for an instantaneous annoying pop every few seconds, it would probably average as the best codec, but it would be useless as a consumer standard. I remember a codec shootout years ago where Mp3Pro sounded "tinny," WMA sounded "flat," and MP3 sounded "fuzzy." Was being objectively closer to the source material more important than the type of distortion introduced? Not at all.

When dealing with sound equipment, from pre-amps to encoders, the tone of the introduced distortion is very important. Everything introduces distortion, in some way or another. You just want it to make the sound better, not worse.

mp3 still defacto standard

No matter what researchers find the best format, the best format for users is what they can doubleclick to play, use on their el-cheapo portable mp3 player or whatever music device they own.

This might be of interest to musicians but the proverbial "jane doe" will keep using mp3 for quite a while

Re:mp3 still defacto standard

This might be of interest to musicians but the proverbial "jane doe" will keep using mp3 for quite a while

Actually it's not that simple. Jane and Joe Doe will start using Ogg, AoTuV or other TLA and ETLA compression schemes when their favorite music players feature them. In the case of Ogg, it's not going to happen anytime soom because:

1 - There's an entrenched MP3 market, as you said

2 - It's an open-source format, i.e. it reeks of piracy and hackers in the minds of music player manufacturers and of the public

3 - It doesn't have the backing of major industry players, being seen as a "maverick" effort to undermine other potentially money-making closed-source formats

4 - It certainly doesn't have the backing of the RIAA, because it doesn't have DRM and other in-the-customer's-face copyright protection schemes

In short, people using Ogg will be opensource-aware and advocates for a long time to come. As for other Apple customer-unfriendly sort of schemes, I'm not convinced the general populace has bought into the idea of paying for music tracks that can become unplayable at the next Apple format-change-du-jour, because they're copyright-protected and therefore impossible to convert to another standard (in theory).

So yes, you're right, MP3 will stay around for a long time. I certainly won't convert my collection anytime soon...

Re:mp3 still defacto standard

2 - It's an open-source format, i.e. it reeks of piracy and hackers in the minds of music player manufacturers and of the public

Yeah - because piracy and hackers didn't have a hand in making MP3 popular.

Zilch

Reason 2 is bogus.

2 - It's an open-source format, i.e. it reeks of piracy and hackers in the minds of music player manufacturers and of the public

I think you are way off here.

Firstly, a number of portable players support Ogg Vorbis. There is a list of four here [wikipedia.org], I'm sure the number will increase.

Secondly, I'd doubt that many of the public know about Ogg Vorbis, let alone consider it to "reek[s] of piracy and hackers".

Furthermore, the "success" of P2P music sharing indicates that the public are the last group of people to have morals about the source or the format of the music they listen to.

Ogg isn't as widely used by the public, because it is not known by the public, it is as simple as that. That will change, as more and more players support it, and the public find out that it is a DRM free alternative to the flexibility restricted formats such as AAC.

The Big Marketing Push

The open source ones don't have the big push amungst the general population. So, number 3 on the list Apple (ACC) can say in independent tests ACC scored higher than WMA or MP3. The top 2 don't have the marketing push to get out and be popular in the general population.

This does give more fuel to Apple. Although I'm not complaining about them having fuel over Microsoft.

Good.

Good to know MP3 is still improving. Yes vorbis and others are great, but i know every software and hardware player out there plays MP3. I'll be ripping all my cds to high quality MP3 befor i go to college, not because its the absolute best, but because its a standard. Standards aren't always the most efficient, but their strength lies that you cant change them on a week to week basis. Whatever hologrphic storage based finger sized half terabyte 24th generation iPod i buy ten years from now will probably still play my 128 and 256 MP3s.

Re:Good.

I think he was getting at the fact that MP3 is pretty future-proof. Sure, maybe a couple portable players support OGG and FLAC right now, but people who look for that feature are RARE. If the product doesn't do well and the company sees that that feature isn't something people want, they won't use it in their products in the future. MP3, however, is pretty much guaranteed to be around for a loooooooooong time.

Would you want to be stuck using a 10-year-old OGG player in the future when the awesome 300GB new-tech MP3 players built into your watch are out?

Re:Good.

OK, you have a point. But check VorbisHardware [xiph.org] for hardware with Ogg Vorbis support. Also, though Lame did well, MP3 is known to have some limitations. But if you have to use MP3, experiment --alt-preset standard in Lame. It was made to offer very good sound quality in bitrates that average below 200 kbps in most cases.

Regarding the results... It's a bit surprising that this third party tuning/tweak of Vorbis did so well. Which is great and I think Xiph should think about incorporating this work on their official encoder as soon as possible, in order to take advantage of its potential. You may be surprised about the relative low performance of AAC. This is partially due to the fact that the chosen AAC encoder was a CBR only encoder (because it was the best AAC encoder at this bitrate on a previous test - Nero encoder is also a good one and offers VBR encoder). With a good implementation of VBR AAC, it should be possible to get a better performance.

While most of the tested codecs/formats showed good performance at 128 kbps, this test alone shows that none can give transparency ( transparency == unability to distinct from the original source for most people and under good conditions) at this bitrate, contrary to what many think. People who think this is important should demand higher quality files from famous online music services (like iTunes Music Store).

People interested in lossy audio encoding should also try Musepack (file extension .mpc). It is considered by many of the hydrogenaudio enthusiasts as the best format at medium/high bitrates, offering transparency with bitrates normally lower (with standard preset ~170 kbps, typical 142 ... 184 kbps) than what is possible with other formats/codecs. It's now open source (LGPL, iirc). Its biggest disadvantage is the lack of support in portable players (though decoding musepack is faster than decoding the other formats in this test). There are plugins for almost every software player and foobar2000 [foobar2000.org] (which I consider the best one) has native support for it

best vs popular

i realize the geeks of the populace want the highest quality encoding to win. naturally. and it helps when something such as vorbis is rated so highly; it gives it even more geek cred.

however: as someone who studied music and audio, i am constantly surprised at what people will listen to. my friends (well some of them) have no problem cranking low quality mp3s of 50 cent, while i drop my jaw at the poor audio quality as a result of lost information. one time i even remarked to my dad "oh its an mp3" when he was playing something i had given to him which had been apparently later encoded. he wasnt sure (he didnt do the encoding) but doublechecked and yes it was mp3 (probably 160 kbps). he was impressed, when to me the timbral change in the cymbals was a dead giveaway. another time i asked a friend of mine if he was using aac to import all his cds in to itunes when he had been recently doing so. he looked at me blankly and said "whats aac?". which meant, yes he was.

i apologize for rambling, this is what im arriving at:
despite early adoption influence etc that geeks hold, how much does all of this really matter. most people dont care what format its in as long as they can listen to it. and often they cant discern loss of quality unless its extreme. so while i applaud these efforts, im simply wondering if -- aside from research -- they arent futile.

Expensive earbuds and MP3 players

It could also be marketing.

MP3 players got *heavily* marketed after Napster and friends got press and serious college use. "MP3" became associated with "free music". They took off.

The iPod, a decent but not earth-shattering MP3 player, sold *much* better than other MP3 players out there. Why? Marketing. Lots of ads -- the only significant difference to cause such a change.

Vorbis doesn't have a lot of ad money behind it pushing it.

I'd also like to point out that:

* People still use CBR MP3s. CBR was designed for exactly one reason -- allowing constant-rate streaming. It's *stupid* to use CBR for locally stored files -- it gets significantly worse quality for the size -- I've generally found that on the music I listen to, using VBR is equivalent to at least a 30% increase in bitrate in terms of my ability to distinguish between a master an an MP3. If people cared about quality, CBR MP3s would not exist. They wouldn't even have to switch their hardware/software around, since it's the same format, but they won't even go that far.

I *really* get a kick out of it when people buy an MP3 player and a pair of high-end earbuds. It's just plain inane. They just purchased a low-quality audio playback device and then spent a huge amount of money on an expensive pair of earbuds that don't let them hear the now missing nuances of the audio. It's the ultimate in trendiness -- like buying Nike or Banana Republic clothing. iPod + expensive earbuds is not "the ultimate in sound reproduction" even if you really, honestly gave a lot of retailers a whole lot of money for the combo.

Re:Expensive earbuds and MP3 players

The DAC in the iPod is fairly high quality. It is not unreasonable for someone to simply encode their CDs using Apple's lossless codec and put them on the iPod. With a 40G model around 60 albums (assuming an average size of 650M) could be stored losslessy in WAV; a few more using Apple's lossless encoder. It would be like turning your 40G iPod into a 5G iPod and swapping music around but such is life.

It becomes more realistic when you have 80G and 100G drives in your player; in a few months the Neuros [neurosaudio.com] is supposed to have 80G backpacks available (right now up to 40G are available and a few online stores are advertising the availability of the 80G model early) and you can order an 80G backpack right now from Cool4u2View [cool4u2view.com]. The Neuros doesn't support any lossless codecs except for WAV right now (although there is support for WMA I have never used it and do not know if it supports WMA lossless or even if WMA lossless is anything more than tagged WAV). 80G is still around 110 albums. The Neuros IIRC uses the same DAC as the iPod so the quality of the sound would be excellent.

For me -b 160kbps Vorbis files are good enough; I plan to re-encode my collection to FLAC when I get a larger HD for music (right now it is a poor little 20G that only has 4G free) as well as Vorbis (abcde makes it easy to encode to more than one format and put them in different directories) -q5 (for my Neuros).

So your last comment still applies to most people. Not everyone though.

iTunes AAC encoding problems

It is interesting that the note that they used the AAC encoder in iTunes 4.2 instead of the newer 4.5 because of "quality" concerns.

Apparently there's some "high frequency ringing" going on [hydrogenaudio.org].

Better stick to something else for now, if planning to rip to AAC.

How much of this is just OGG fans voting?

I read some of the results, and I'm not a Vorbis hater or anything, but how much of this is open source fans voting for their favorite codec? I looked at the test just now, but can't tell if it was blind or not.

Re:How much of this is just OGG fans voting?

It was a double blind test (ABC/HR) adhering to ITU-R BS.1116-1 [itu.int]. Read more about the methodology in the initial announcement [slashdot.org].
In addition to being double blind results were also encrypted so manipulation is very unlikely.

Re:How much of this is just OGG fans voting?

I was always fond of LAME encodings in a high quality VBR mode. It was always my favorite method of storing my music on my hard drive, since the quality was quite good. Over time, I decided that I would really start comparing it to some other formats for long-term archiving. I wanted to settle with one format, once and for all. I had originally been a BladeEnc user, but LAME seemed far superior to me.

When I first enconded some of my music in the Vorbis format, I was a bit underwhelmed when comparing it to LAME. It didn't really sound the same. Then, I compared the Vorbis files to the raw WAV rips. Surprisingly, the Vorbis files sounded more true to the original WAV rips. I was very surprised. All this time, my ears had tuned to the LAME acoustic model, which wasn't as accurate as I had once thought. After comparing a large portion of my CD collection in both LAME and Vorbis encodings, I made a decision...

I decided to start using FLAC. That way, I could listen to al of my music without any concern for quality. Sure, each CD takes up about 300 MB of space (50%-60% average compression), but it sounds so sweet.

If quality is a concern, maybe LAME MP3/AAC/Ogg Vorbis aren't the the right choices. Hard drive limitations aren't so much of an issue anymore. I guess that I cna see a point in having lower quality files for easy web transmission and low storage capacity, but the quality difference is just too noticable for me to ignore, when comparing any of these formats to a lossless format like FLAC. That's also one of the reasons that I like Magnatune so much, since I can buy music online that is already compressed in lossless FLAC format.

Yes, but how did Ogg do?

*ducks*

Open source wins: really cool, but ...

it isn't everything. Microsoft still has enough cash to fight this. Our local radio (Switzerland) still broadcasts in .asx. I sent them an e-mail asking them why. They said because their server is sponsored by Microsoft. Now I listen to virgin radio, they broadcast in broadband ogg [virginradio.co.uk]

Inaccurate test, big bitrate differences

Quoting post on second page of discussion:

This particular test should be called, "The 128 kbps test for iTunes/WMA, and the low-130 test for AC3 and LAME, and the close-to-160 test for MPC/Vorbious.

Leahy iTunes MPC Vorbis Lame WMA Atrac3
bitrate 128 155 149 133 128 132
Score 4.34 4.41 4.68 4.11 4.37 3.76

That really doesn't look very fair to me! MPC and Vorbis using about 20% more bits than Lame and iTunes AAC.

Re:Inaccurate test, big bitrate differences

Those numbers are wrong. The real average bitrates are:

iTunes MPC Vorbis Lame WMA Atrac3
128 136 135 134 128 132

Take your time to look at the detailed results [rjamorim.com] yourself next time.

Re:Inaccurate test, big bitrate differences

128 / 149 = 86%

Vorbis is not a CBR codec like WMA. It's almost impossible to get it directly on the nose. The encoder doesn't easily allow that kind of control without seriously damaging the quality of the finished file. I'm not sure that the 14% difference really matters as much as you insist.

To be fair though, WMA does perform reasonably well for a CBR format. However, that's not what the test is about. It's about getting the best sound out of a similar amount of space.

I don't doubt that Vorbis would still beat WMA if the bitrates were 100% even, to be honest with you. It's just not that simple to get it directly on the nose. It would have been interesting to see the results of Vorbis on a quality level that is a notch lower, so that we could see how much variance there is between each level.

Compared with radio

Compare this with radio. There are a lot of popular AM and LW radio stations here in the UK even though FM is a superior format. MP3 will be around for almost ever due to the popularity and level of takeup.

Aotuv vorbis enconder for Debian

There is a Debian repository at rarewares [rarewares.org] where you can get the aotuv version of vorbis enconder. Grab yours today!

More vorbis content is needed

One way I see Vorbis making it into the mainstream is if there were high availability of Vorbis content on the net. This includes P2P channels as well. If music releasers in the underworld start adopting vorbis, then Joe 'I own the original CD' Downloader will get a far wider familiarity with the codec, same as to what happened IMHO with xvid. More content will eventually lead consumers to start demanding vorbis compatibility in their hardware.

Control

It would have been nice to have an original unencoded piece and rate it against the masses. That way we'd be sure the listeners weren't picking up on a mastering problem that is muffled by an encoder.

Vorbis and MP3 formats from a techie view point

Having designed and written a mp3 decoder and now working on a vorbis decoder I can't say I'm that suprised by vorbis coming out on top.

From a technological standpoint the Vorbis codec has 10 years of audio compression R&D in it since MP3 was invented.

MP3 is a subband DCT based codec using fixed window length. Vorbis is also DCT based but encodes an approximation to the orginal frame's spectral curve and also uses variable length window length.

In using the source from the vorbis library and the decoder specification to help guide its development I have to say it is a real joy to code. The people at xiph.org have really done a first class job and have approached some of the problems of audio codec design with some of the best lateral thinking that I have ever seen.

Believe me! Coming from me that is very rare praise.

What it means for Vorbis

Well, I can only comment from a technical point of view, but firstly it is very good news that we are progressing in the right direction in terms of quality. Secondly, compared with the other codecs (esp. the proprietary ones), Vorbis is quite simple and minimalistic and lacks a lot of advanced tools and profiles, yet we've been able to extract quite competitive performance from some adjustments here and there. There is more to do in Vorbis and Monty has some new ideas that he wants to implement in the next major version like a better stereo model, noise normalization (which in its current form is mostly experimental), and support for 5.1 stereo. Given the success of aoTuV and the fact that Monty is fully aware of these third-party tunings, I think Vorbis development is looking ever-more exciting. :) (Note I don't work for Xiph.Org but just one of those third party Vorbis tuners)

Lame CBR better than VBR?

Recently I decided to use lame with cbr 192kbps after comparing to the preset vbr settings (including extreme). I use the settings: --cbr -b 192 -h -q0

Using vbr I can hear the noise floor being modulated e.g. by a large amplitude low pass filtered bass sound. I contribute this to vbr changing bitrate. Maybe the psychoacoustic model just doesn't fit my ears:-)

The vbr files average around 200kbps anyway, so they're not smaller than 192kbps cbr.

It would have been nice if the test included cbr as well.

I use good headphones: Sennheiser HD-25, and my mp3 player: archos jukebox recorder running the open source firmware Rockbox [rockbox.haxx.se]

For Debian Users

libvorbis-aotuv-dev_0.b2-1_i386.deb [soniccompression.com]
libvorbis-aotuv-0_0.b2-1_i386.deb [soniccompression.com]
libvorbisenc-aotuv-2_0.b2-1_i386.deb [soniccompression.com]
libvorbis-aotuv-0_0.b2-1_i386.deb [soniccompression.com]
libvorbisfile-aotuv-3_0.b2-1_i386.deb [soniccompression.com]
oggenc-aotuv_1.0.1+aotuv-2_i386.deb [soniccompression.com]

You don't even need to uninstall your existing vorbis packages.

We are banwidth wasters :)

Quoting a couple of posts in Hydrogenaudio:

a post [hydrogenaudio.org]:

What about all the /.ers?
Seems they were just interested in wasting bandwidth after all

the reply [hydrogenaudio.org]:

More than 500 people downloaded the samples through bittorrent only - not counting HTTP downloads! :B

I won't ever understand these people.

Disclaimer: I am NOT new here :)

why 16 bits /44100 Hz ?

I'd wonder why songs are encoded from 16 bits / 44100 hz, while our ear can hear a difference with 96 khz/32bits and consumer sound cards can often do 48khz ? (if you can do better than rip from a CD, that is)

I mean, would 160 kbps sound better from 44100 or 48 / 96 khz ?

Interesting MPC outlier

The MPC codec was neck-and-neck with Vorbis most of the time, except for one song by Debussy. What is interesting though is that it only encoded at 91kbps for that song---suggesting that perhaps if it were forced to use more bits it might have scored higher. It seems the heuristics it uses to determine how many bits it needs didn't quite work for that song.

ABR vs VBR isn't exactly fair

Apple's AAC implementation uses ABR rather than a purely VBR implementation. Look at the bitrate distributions. Ogg's bitrate was significantly higher than iTunes' in the majority of listening tests.

don't fear the ripper

All these codecs work on audio data that has to come from somewhere, typically CDs. Xiph offers the standard CDDA->WAV converter that feeds LAME and the other codecs. Sure, CDParanoia III v9.8 works very well, but it's slow: typically the best error recovery cuts performance by at least 65%. The Paranoia news page [xiph.org] says

"March 27, 2001

Things on hold for now: No, that doesn't mean the project is dead, just that active development is on hold while we throw all the time we have available to get OggVorbis [xiph.org] to 1.0 in a reasonable amount of time. Once Vorbis hits 1.0, we'll get back to Paranoia."

Vorbis hit 1.0 [xiph.org] 22 months ago:

"README 1.14 22 months xiphmont That's it. Full 1.0 libVorbis code handoff to release engineering."

Of course, this is OSS, and free, to boot (pun(s) intended ;). So it's hard to demand a new version, even after 2 years. But a lot has happened, especially in performance opportunities, since that release. So where's Xiph's committment to the expectations they created with that announcement? Where's the community effort to advance this essential tech?

Hacking the paranoia lib for device performance tweaks isn't my bag, baby. I would help another, lead, developer, by testing, or higher level hacking, or project organization support, etc. This library is the bottleneck for media format freedom - everyone on unix/linux has to use it, in one form or another. That widespread use usually drives progress in OSS. Where's the love?

JACK, Jack, jack & the ripper

The speed of CDparanoia is not limited by the cdparanoia code. Regardless of future improvements, speed gain would never be one of them. If anything, additional error correction would only ever make it slower. Your limit is Linux forcing programmed I/O because the IDE subsystem doesn't know how to use DMA on non-multiple-of-512-byte sector sizes. CDDA is 2532 bytes per sector. Linux 2.6 partially fixes this.

Also, cdparanoia (III) was finished long ago. It has not bitrotted. As new kernels came out, we+others kept it up to date. The distribution maintainers have added whatever fixes have been necessary for their distros. Nothing that worked in 1999 is broken today.

In summary... paranoia does 100% of what *I* need it to. I write software that I need. I don't have to keep releasing 'improved' versions of software that already works as an ego-trip or to placate a marketing department desperate to sell you the same thing in a new box every six months.
Others have expressed interest in doing new things with paranoia, but no one has followed through... at least not yet. Paranoia isn't all that complicated to use or hack. That speaks to a pretty damned low demand for new versions.

The website: yah, OK, I'm lousy at writing HTML updates. My diary hasn't been updated in three years. There is certainly a website attention span problem ;-)

Theora: I'm not one of the primary coders today, I only did the initial code import. Also, the Helix project has required relatively little time; Real has done nearly all the heavy lifting on integration there.But, if 'Theora is dead', why does CIA show 500 commits in the past two months?

DirectShow issues can be summed up as 'ugh, what an awful system'. But we'll make it work. The discussion about mux was proposed changes to spec. Voluminous discussion reveals what we have now is still the best option, as designed five years ago.

Monty

Bitrates!!!

Did anyone actually read the whole results section? Vorbis, MPC and LAME were using VBR that resulted in average bitrates higher (and in some cases much higher) than 128kbps. Go back to the results page [rjamorim.com] and scroll to the third from bottom table and have a look for yourself. On the samples where there was a clear or runaway winner, the bitrate for that codec is generally higher than the average. Only AAC and WMP actually had 128kbps samples.

For example, the final sample "Waiting" the winning codec MPC was actually 153kbps, followed by LAME and Vorbis at 144 and 148kbps. Unless I don't understand his algorithm, it doesn't seem as though he is taking these bitrate differences into account

The report claims to be a 128kbps Multiformat Test, and it's really a "128kbps standard preset on our encoder" Multiformat Test which may result in higher bitrates than 128kbps. I am sure that if you gave the static bitrate codecs the same bitrate as the resulting VBR average bitrates, they would be much more competitive.

No HE AAC?

I'm a little surprised that they didn't including HE AAC, a more recent MPEG-4 audio codec. It's best known in its AAC+ implementation from Coding Technologies. It's definitely entertainment quality at 48 Kbps at 44.1 stereo.

Anyone know why it got left off?

We also have parametric stereo AAC coming as well, which should be able to do entertainment quality at 24 Kbps.

160 kbps

So now at at the 128K level, the difficulty in perceiving differences is showing up on the graphs. Since portable players are coming out with more and more space these days (40 Gig iPod anyone), why not test at 160 kbps or 192kbps and then see if there is even a perceived difference. The file size from 128kbps to 160kbps is only going to increase by around 25-30% but you'll get a nearly indistinguishable file from the original.

I'd wager that at 192 kbps on any codec (except ATRAC by the looks of the numbers), only the real Golden Ears (TM) can hear the difference.

unfair comparison

The two winners had the highest average bitrates, 135 and 136 kbit, giving them an advantage over the encoders that outputted the correct bitrate.

Meaningless?

Though I think that this is a great idea for a study, aren't the conclusions found by this research ultimately meaningless? In his overall ratings graph, there are a total of eighteen samples and the other track-based analysis have fewer samples than that (N=17, N=15, etc.). This isn't even close to what is required for a good sampling of the US population! (I believe you need about 1,500 to get even close for most statistical comparison of US population.)

Add onto this the fact that no random sampling was done and this test is even more meaningless because of the self-selectivity of the participants. People who were sampled were most likely those with active Internet connections and those of the audiophile and/or Slashdot-type "Nerd" communities online. Naturally, we are more accustomed to hearing vorbis as a file format. Perhaps "Joe Blow" down the street hears a very different tune, but this method of sampling doesn't account for that, because Joe Blow never even heard about participating in this study.

Granted, that might be okay since I think perhaps the self-selectivity of the participants yield results that will match up with those who actually care, but I still can't see how the insignificant sample sizes used for this research can have any real meaning.

Why keep calling AoTuV a fork?

It's odd to keep hearing this code referred to as a 'fork'. Yes, it's based on our reference code while doing further tuning just like all the free MP3 encoders are based off of the original dist8 or dist10.

Fork seems to imply that they're trying to make something incompatible or doing it without our blessing. Neither is true! We never wanted to have *the* only encoder. Nor did we want to be the only people trying to improve Vorbis's encoding.

AoTuV is a 100% real Vorbis encoder and the results of the test speak for themselves. Aoyumi and crew deserve kudos, and I'm glad to see them working on improving Vorbis encoding.

Monty

Re:FLAC?

WTF? Why would you even need a listening test if codecs are lossless? By definition, lossless compression produces final sound identical to the source. If it didn't, it would be lossy.

It's called a "control"...

What if a lossless codec were included in the test - and it came in dead last?

That would provide useful information: either the listeners weren't up to the job or the lossy codecs at ~128 kbps were truly indistinguishable from the source material.

Re:FLAC?

Because this was designed to test various codecs at 128Kbps. You can't make flac do 128Kbps. Besides, flac, being lossless, sounds exactly the same as the source media, so what's the point of testing how it sounds?

Re:FLAC?

Assuming FLAC is truly LOSSLESS:

1) Is it really a codec? Seems to me it is a compression method for media, like .zip .tar etc., not an encoder... technically.
2) It should sound exactly like the original. LOSSLESS = no loss. No point in comparing it to lossy codecs, unless it's not truly lossless.
3) The stored file sizes although smaller than the raw music are still way to big to be portable IMO.

Re:FLAC?

1) Is it really a codec? Seems to me it is a compression method for media, like .zip .tar etc., not an encoder... technically.

"Codec" means "coder-decoder". FLAC sounds encoded to me, if you need a FLAC library to enable a piece of music-playing software to read it, then I'd say the FLAC library is a codec.

2) It should sound exactly like the original. LOSSLESS = no loss. No point in comparing it to lossy codecs, unless it's not truly lossless.

Actually, it's interesting to compare lossless and lossy compressions because, these days, there's a fair chance that very good lossy compression sound so good it's almost impossible to tell the difference with the lossless compression.

3) The stored file sizes although smaller than the raw music are still way to big to be portable IMO.

Depends how much smaller. I'd say anything that doesn't produce at least 5x compression is worthless in any music player. You can zip a wav file and despite being much smaller than the original, it will still feel worthless to you in a compactflash card in terms of size.

Re:FLAC?

1) Is it really a codec? Seems to me it is a compression method for media, like .zip .tar etc., not an encoder... technically.

Compressers are encoders of a particular variety. They just choose a different data representation as an encoder does, but make an effort to take advantage of specific known characteristics of the data they are compressing to get a smaller, reasonable representation..

ZIP and gzip (tar does not do compression, just file joining) do very poorly at compressing audio. They do things like look for patterns of repeating (or at least commonly seen) sequences of data, and simply say something like "every time you see "z1", I really mean ";lt&a href="". This approach often works very well in computer-generated files.

However, it's very unlikely that you will get exactly the same sequence of bits in an audio recording, so .zip/.gzip are very poor at compressing audio recordings. FLAC and similar lossless audio compression look for things like (I would imagine) relatively small deltas from each sample point to the next, since this is a common characteristic of audio data.

FLAC is indeed lossless.

Umm because it's a 128k test?

FLAC compresses to what, 5-600k? The only way you could make it fit the competition would be to mutilate the original WAV, and trust me, you wouldn't want to do that.

Certainly, it's a good reference if you want to compare across bitrates (what's 128k vs 256k vs lossless like in quality/size) but it has no place here.

Kjella

Re:FLAC?

Testing FLAC is useless, as most of you say.
But I think it was convenient testing any kind of lossless (FLAC or WAV).
The audience may have giving slanted scores because they know the clips suffered lossy codecs, they were hearing artifacts the way people hear "voices".
It would have been interesting to know if lossless scores 5.
And a codec wich really improves music would help many artists today XD

Re:FLAC?

The way I understand the testing methodology (and I could be wrong):

--- The files that the testers received were all uncompressed (or compressed with a lossless codec). In other words, they each had been run through their respective lossy encoders and then decoded back to .wav format. There are several reasons for this: 1) it prevents the testers from deducing anything from the file sizes; 2) it eliminates the need for the testers to have access to all the codecs; and 3) it simplifies the design of the ABC/HR+ testing tool.

--- The tests weren't pitting the codecs against each other per se, but rather each against the uncompressed original. So there is no sense in including a lossless codec in the test because, well, it's already there in the form of the control file.

Again I could be wrong about this, so please correct me if I am wrong. In any case, there is no sense in testing FLAC here.