[Q] What is ABX testing and how do I do it?

[seadzz]

How do I determine the right codec and bit rate for me or what is AXB testing?

Answer:

To determine the best codec that gives you the right balance between file size and audio playback quality is the source of great debate. There is no right codec or right bit rate for universal use since everyone has different needs (flash players vs. HD owners) and everyone has slightly different hearing. Men by nature have a slightly lower response to high frequencies (thanks to mother nature) and exposure to loud sounds over a prolonged period (jet blast, jackhammers, subway noise) also tends to degrade your hearing.

ABX testing is a method where you compare a reference sample against a test subject. If you know the name of the test subject that can bias things since it is easy to cheat. You need this test to be blind and using an ABX test tool makes sure you can’t cheat.

I have found ABX testing the best way to determine which codec sounds best to me and what bit rate to use. I thought I’d share how to do this test and what tools are required. It is not difficult and without some real tools your test can be biased since you are not doing it blind.

Tools:

There are many out there but the best (easy for me to use) is foobar 2000. If you don’t have that player yet go here and get it.

http://www.foobar2000.org/foobar2000_0.8.2.exe

You will also need the ABX tool plugin (is a dll file that you download and add to the foobar directory). It is located here.

http://www.foobar2000.org/foo_abx.zip


Procedure:

Take a CD you know well and identify one track that is not complex but has much detail, piano, violin, female singers, harpsichord or organ music is what you are looking for. You do not want hard rock, garage metal, screaming RAP music for this test….even if that is your favorite music please do not use it for ABX testing since all that screaming and banging will mask shortcomings and artifacts in the codec\bitrate. Again, soft passages with large differences from silence to a given detailed sound are what you are looking for.

Next rip the test track into a WAV file…..save it to a place where you can find it….desktop is a good idea. You then encode this track into a few differing codecs and differing bitrates. To keep from going nuts I recommend only using one codec at a time and use large bitrate spreads. Use 128 as a baseline and then jump to 160 and another at perhaps 192 or 224. You can fine-tune the bitrate later but your first pass should be at eliminating bitrates that sound like POS.

Once you have encoded this wav file into a few differing bitrates (make sure you name the files differently….use 196.mps or 128.org to keep things organized) you can move on to the setup process.

Click on foobar and open the player. Click preferences (upper left) and look for ABX Test. Open that up and unclick the normalize option. Save and shut foobar down.

Restart foobar and locate the tracks you want to test (only two at a time) and import them into the player. Make sure you hit the stop button since foobar will automatically start playing the imported files. If you downloaded and installed the abx.dll file correctly you should be able to highlight the two files to test by first clicking on the top file then hold the shift and arrow down key. With both files highlighted RIGHT CLICK the files. A dialog box will open, scroll down to the ABX test option and click it.

Files will automatically be loaded into a new window and ABX tools will popup. Now you can move on to finding a good part of the test track to use. You really only to use a small segment of the subject file…..no more than ten seconds. If you try to use the entire file you will go nuts. Listen to the track by pressing the A or B button and set a start and stop position on the track….again you only want a small file segment.

Once you have this segment selected listen to both A and B. then listen to X and Y. You will be trying to identify if A=Y or X and is B = X or Y. Once you think you have ID’s the track as being X or Y select the next test. This will take your impute and bring the result down 1/1 would mean you ID’s it right 0/1 would mean you blew it. Do this a number of times and the tester will tell you if you are correct or if you are guessing. If you correctly ID a file a number of times your probability of guessing % will approach zero. If on the other hand if you are wrong a lot you will see a high % chance you are guessing. The foobar tools will randomly mix up the X and Y track to prevent cheating thus making it hard to beat. and giving you results that have value.

I have used this approach to test enhanced versions of ogg and LAME against the original WAV with surprising results. On testing OGG files encoded at Q8 with two differing versions I got a 9/10 result, which tells me that I can ID, the two.

When I try to tell the difference between the improved ogg version from the WAV things to !@#$ fast. Test result look like 5/10, which is a 50-50 chance I am guessing.

I hope this little ditty helps you get where you want to go in the wonderful world of encoding.

Enjoy the music,

sdz