Difference between revisions of "Hydrogenaudio Listening Tests"

From Hydrogenaudio Knowledgebase
Jump to: navigation, search
m (→‎External Tests)
(Undo revision 35480 by 208.127.232.205 (talk) That's a private listening test)
 
(58 intermediate revisions by 16 users not shown)
Line 1: Line 1:
−
From time to time, HydrogenAudio members conducted listening tests, either public ones or private ones. These listening test should be valuable in gauging the performance of various digital audio techniques, e.g. compression, transcoding, etc., '''on a certain point in time'''.
+
One of the most visible ways that HydrogenAudio members have shown their enthusiasm for audio quality and their respect for [[TOS 8|scientific principles]] over the years is by organizing and participating in public blind listening tests of lossy codecs and presenting statistical analyses of the results. These results show the advantages and disadvantages of different formats and encoders and help identify which are "state of the art."
  
−
It should be noted that these tests '''may become obsoleted''' by newer enhancements in the techniques. So please ensure that there's no newer tests before drawing conclusions.
+
[[Private Listening Tests|Private listening tests]] also provide useful data, but the public HydrogenAudio tests are a rarity in having enough participants to both average out variations among individual listeners and still have statistically significant results.
  
−
This page contains links to Listening Tests carried out by various members of HydrogenAudio.
+
Note that a listening test cannot really compare formats or encoder "brands." Instead, it compares particular versions of particular encoders at particular bitrates. Results will differ with different encoders, newer versions of the same encoders, or different bitrates. Use caution when extrapolating, and use the most recent results when extrapolating to the newest encoder versions.
  
−
'''Note:''' There are no such things as listening tests for [[Lossless]] compression. There are only performance tests (e.g. compression ration, encode/decode speed, etc). So, check out the [[Lossless comparison|Lossless comparison page]] instead.
+
'''Note:''' There would be no point in listening tests of [[lossless]] encoders; the decoded results are bit-identical to the original uncompressed recordings. [[Lossless comparison|Lossless encoder comparisons]] instead consider technical aspects such as compression ratio and encode/decode speed.
  
 +
All tests are sorted by date, newest first. All dates are in [http://www.cl.cam.ac.uk/~mgk25/iso-time.html ISO8601 format].
  
−
=Format-specific Tests=
+
==Tests and Results==
−
 
+
{|class="wikitable" align="right"
−
==[[AAC]]==
+
!colspan=2| Result Symbol Key
−
 
+
|-
−
{| border="1" cellpadding="2" cellspacing="0" style="text-align:center;"
+
| > || Better than
−
|- style="background:#ffaaaa"
+
−
! Date !! Title<br>Samples !! Performed by !! Type !! Codecs !! Links
+
 
|-
 
|-
−
! align="left" | 2004-12-09 || Ahead vs Apple 128-kbps<br>Classical || Guruboolez || Private
+
| ~ || About the same as
−
| iTunes 4.1.052, 4.7.042;<br>Nero 2.5.9.7, 2.9.998 CBR, 2.9.998 VBR || [http://www.hydrogenaudio.org/forums/index.php?showtopic=29924&view=findpost&p=258644 HA Thread]
+
 
|-
 
|-
−
! align="left" | 2005-09-05 || 48- & 64-kbps HE-AAC<br>Classical || Guruboolez || Private
+
| ~> || About the same, but slightly better than
−
| CT aacPlus v.2 48- & 64-kbps<br>Helix 11.0.0.1897, Nero 3.2.0.20 VBR || [http://www.hydrogenaudio.org/forums/index.php?showtopic=36868&view=findpost&p=325091 HA Thread]
+
 
|}
 
|}
  
−
 
+
===Multiformat Tests===
−
==[[MP3]]==
+
  
 
{| border="1" cellpadding="2" cellspacing="0" style="text-align:center;"
 
{| border="1" cellpadding="2" cellspacing="0" style="text-align:center;"
−
|- style="background:#aaffaa"
+
|- style="background:#aaffff"
−
! Date !! Title<br>Samples !! Performed by !! Type !! Codecs !! Links
+
! Date !! Title<br>Samples !! Performed by !! Codecs !! Result !! Links
 
|-
 
|-
−
! align="left" | 2004-01-18 || Dithering Test (1)<br>Special || Guruboolez || Private
+
| 2014-07
−
| [[MP3]] Decoders: <br>MAD, [[LAME]], Fraunhofer, foobar2000 || [http://www.hydrogenaudio.org/forums/index.php?showtopic=17728&view=findpost&p=174740 HA Thread]
+
| 96-kbps<br>General
 +
| Kamedo2
 +
| [[Apple AAC]] (iTunes 11.2.2), [[Opus]] (1.1),<br>[[Vorbis]] ([[aoTuV]] Beta6.03), [[MP3]] ([[LAME]] 3.99.5)
 +
| Opus > Apple ><br />MP3(V5) ~ Vorbis
 +
| [http://www.hydrogenaud.io/forums/index.php?showtopic=106354 HA Thread]<br>[http://listening-test.coresv.net/results.htm Complete Result]
 
|-
 
|-
−
! align="left" | 2004-11-15 || 130-kbps LAME<br>rjamorim || Guruboolez || Private
+
| 2011-04-10
−
| [[LAME]] 3.90.3 ABR, 3.96.1 ABR, 3.96.1 VBR || [http://www.hydrogenaudio.org/forums/index.php?showtopic=29422&view=findpost&p=253968 HA Thread]
+
| 64-kbps<br>General
 +
| IgorC
 +
| [[Vorbis]] [[aoTuV]], [[QuickTime AAC|Quicktime]] HE-[[AAC]], [[Opus]]
 +
| Opus > Apple ><br />Nero ~ Vorbis
 +
| [http://www.hydrogenaud.io/forums/index.php?showtopic=87785 HA Thread]<br>[http://listening-tests.hydrogenaud.io/igorc/results.html Complete Result]
 
|-
 
|-
−
! align="left" | 2005-01-11 || 128-kbps LAME<br>ff123 || Guruboolez || Private
+
| 2007-08-16
−
| [[LAME]] 3.90.3, 3.96.1, 3.97a5 || [http://www.hydrogenaudio.org/forums/index.php?showtopic=30547&view=findpost&p=264411 HA Thread]
+
| 64-kbps<br>General
 +
| Sebastian Mares
 +
| [[Vorbis]] [[aoTuV]], [[Nero AAC]],<br>[[WMA]] Pro
 +
| Nero > WMA Pro ~ Vorbis
 +
| [http://www.hydrogenaud.io/forums/index.php?showtopic=56397 HA Thread]<br>[http://listening-tests.hydrogenaud.io/sebastian/mf-64-1/results.htm Complete Result]
 +
|-
 +
| 2006-11-22
 +
| 48-kbps<br>General
 +
| Sebastian Mares
 +
| [[Vorbis]] [[aoTuV]], [[Nero AAC]],<br>[[WMA]] Pro, [[WMA]] Standard
 +
| Nero > WMA Pro ~ Vorbis > <br /> WMA Standard
 +
| [http://www.hydrogenaud.io/forums/index.php?showtopic=50408 HA Thread]<br>[http://listening-tests.hydrogenaud.io/sebastian/mf-48-1/results.htm Complete Result]
 
|-
 
|-
−
! align="left" | 2005-10-17 || [[LAME]] 3.98 -V5<br>Classical || Guruboolez || Private
+
| 2006-01-13
−
| [[LAME]] 3.98: -V5, with --vbr-new,<br>add --athaa-sensitivity 1 || [http://www.hydrogenaudio.org/forums/index.php?showtopic=37973&view=findpost&p=335086 HA Thread]
+
| 128-kbps<br>General
 +
| Sebastian Mares
 +
| [[QuickTime AAC]], [[Nero AAC]], [[LAME]] [[MP3]],<br>[[Vorbis]] [[aoTuV]], [[WMA]] Pro
 +
| 5-way tie
 +
| [http://www.hydrogenaud.io/forums/index.php?showtopic=40607&view=findpost&p=357117 HA Thread]<br>[http://listening-tests.hydrogenaud.io/sebastian/mf-128-1/results.htm Complete Result]
 
|}
 
|}
  
−
'''''Notes:'''
+
===[[AAC]] Tests===
−
(1) Performed under extreme listening conditions not reproducible on daily listening.''
+
−
 
+
−
 
+
−
==[[MPC]]==
+
  
 
{| border="1" cellpadding="2" cellspacing="0" style="text-align:center;"
 
{| border="1" cellpadding="2" cellspacing="0" style="text-align:center;"
−
|- style="background:#aaaaff"
+
|- style="background:#ffaaaa"
−
! Date !! Title<br>Samples !! Performed by !! Type !! Codecs !! Links
+
! Date !! Title<br>Samples !! Performed by !! Codecs !! Result !! Links
 
|-
 
|-
−
! align="left" | 2005-06-18 || --standard<br>Classical || Guruboolez || Private
+
| 2011-08-20
−
| 1.01j, 1.15u || [http://www.hydrogenaudio.org/forums/index.php?showtopic=34911&view=findpost&p=306910 HA Thread]
+
| 96-kbps <br>General
−
|}
+
| IgorC
−
 
+
| CT, [[Fraunhofer|FhG AAC]] via Winamp (not [[Fraunhofer FDK AAC|FDK]]), [[Nero AAC|Nero]], [[QuickTime AAC|QuickTime]] TVBR & CVBR
−
 
+
| Quicktime CVBR ~> TVBR ~><br /> Fraunhofer > CT > Nero
−
==[[Vorbis]]==
+
| [http://www.hydrogenaud.io/forums/index.php?showtopic=89765 HA Thread]<br>[http://listening-tests.hydrogenaud.io/igorc/aac-96-a/results.html Complete result]
−
 
+
−
{| border="1" cellpadding="2" cellspacing="0" style="text-align:center;"
+
−
|- style="background:#ffffaa"
+
−
! Date !! Title<br>Samples !! Performed by !! Type !! Codecs !! Links
+
−
|-
+
−
! align="left" | 2003-06-18 || 128-kbps<br>ff123/rjamorim || Guruboolez || Private
+
−
| GT2 & PCVS || [http://www.hydrogenaudio.org/forums/index.php?showtopic=10555&view=findpost&p=106189 HA Thread]
+
 
|-
 
|-
−
! align="left" | 2004-06-17 || 160-kbps<br>Special || Guruboolez || Private
+
| 2006-02-20
−
| CVS, GT3b2, [[aoTuV]]b2, [[aoTuV]]b2+QK || [http://www.hydrogenaudio.org/forums/index.php?showtopic=22495&view=findpost&p=218681 HA Thread]
+
| 48-kbps HE-AAC<br>General
 +
| Gabriel
 +
| 3gpp (v1), CT (v1 & v2), [[Nero AAC|Nero]] (v1 & v2)
 +
| All tie for most sample types;<br /> otherwise any v1 > any v2
 +
| [http://www.hydrogenaud.io/forums/index.php?showtopic=42679&hl= HA Thread]<br>[http://www.mp3-tech.org/tests/aac_48/results.html Complete Result]
 
|-
 
|-
−
! align="left" | 2004-02-05 || -q4<br>Transients || Guruboolez || Private
+
| 2006-02-05
−
| RC3 2002-03, RC3 2002-04, 1.01, 1.01 QK2 || [http://www.hydrogenaudio.org/forums/index.php?showtopic=18359&view=findpost&p=180697 HA Thread]
+
| 48-kbps HE-AAC<br>Classical
 +
| Ivan Dimkovic
 +
| [[Nero AAC|Nero]] HE-AAC v1 & v2
 +
|
 +
| [http://www.hydrogenaud.io/forums/index.php?showtopic=41191&hl= HA Thread]
 
|}
 
|}
  
−
 
+
===[[MP3]] Tests===
−
==[[WMA]]==
+
  
 
{| border="1" cellpadding="2" cellspacing="0" style="text-align:center;"
 
{| border="1" cellpadding="2" cellspacing="0" style="text-align:center;"
−
|- style="background:#ffaaff"
+
|- style="background:#aaffaa"
−
! Date !! Title<br>Samples !! Performed by !! Type !! Codecs !! Links
+
! Date !! Title<br>Samples !! Performed by !! Codecs !! Result !! Links
 
|-
 
|-
−
! align="left" | 2003-06-18 || 128-kbps<br>ff123/rjamorim || Guruboolez || Private
+
| 2008-10-17
−
| [[WMA]]9 Standard & Pro || [http://www.hydrogenaudio.org/forums/index.php?showtopic=10551&view=findpost&p=106158 HA Thread]
+
| 130-kbps<br>General
 +
| Sebastian Mares
 +
| [[LAME]] 3.97, [[LAME]] 3.98.2, [[iTunes]], [[Fraunhofer|FhG MP3]], [[Xing|Helix]]
 +
|
 +
| {{ha|https://hydrogenaud.io/index.php/topic,66564.0.html|Thread}}<br>[http://listening-tests.hydrogenaud.io/sebastian/mp3-128-1/results.htm Complete Result]
 
|}
 
|}
  
 +
==External Tests==
 +
Other people have held listening tests in venues outside of Hydrogenaudio (although the organizers of the tests tend to be HA members :).  The results of these tests are found below:
  
−
=Multiformat Tests=
+
* [https://web.archive.org/web/20070720024213/http://www.ff123.net/ ff123's site on audio testing and listening tests] (most recent test 2002-07-27)
−
 
+
* [https://web.archive.org/web/20071014001946/http://www.rjamorim.com/test/ Roberto's Public Listening Tests] (most recent test 2004-07-11)
−
{| border="1" cellpadding="2" cellspacing="0" style="text-align:center;"
+
* [http://listening-tests.hydrogenaud.io/sebastian/ Sebastian Mares' Listening Tests] (most recent test updated 2008-11-22)
−
|- style="background:#aaffff"
+
* [http://www.soundexpert.info/ SoundExpert Listening Tests] (continually updated)
−
! Date !! Title<br>Samples !! Performed by !! Type !! Codecs !! Links
+
−
|-
+
−
! align="left" | 2003-10-11 || 130-kbps<br>Classical (Baroque) || Guruboolez || Private
+
−
| [[AAC]] (FAAC, PsyTel, Nero, QuickTime)<br>[[MP3]] [[LAME]] 3.90.3, [[MPC]] 1.14, [[Vorbis]] GT3b1, [[WMA]]9 Pro || [http://www.hydrogenaudio.org/forums/index.php?showtopic=14091&view=findpost&p=142684 HA Thread]
+
−
|-
+
−
! align="left" | 2003-12-13 || 128-kbps<br>Classical || Guruboolez || Private
+
−
| [[AAC]] (iTunes, Nero, FAAC), <br>[[Vorbis]] 1.01, [[WMA]]9, [[MP3]] [[LAME]] 3.93.1 || [http://www.hydrogenaudio.org/forums/index.php?showtopic=16395&view=findpost&p=163116 HA Thread]
+
−
|-
+
−
! align="left" | 2004-07-12 || 175-kbps<br>Classical || Guruboolez || Private
+
−
| [[Vorbis]] Megamix -q5,50, -q5,99 -q6,00<br>[[MP3]] [[LAME]] 3.97a3 -V3, -V2, [[MPC]] -q5 ||
+
−
[http://www.hydrogenaudio.org/forums/index.php?showtopic=23355&view=findpost&p=225046 HA Thread]
+
−
|-
+
−
! align="left" | 2004-12-09 || 130-kbps<br>Classical || Guruboolez || Private
+
−
| [[AAC]] Nero 2.9.9.998<br>[[Vorbis]] aoTuVb3 ||
+
−
[http://www.hydrogenaudio.org/forums/index.php?showtopic=29925&view=findpost&p=258645 HA Thread]
+
−
|-
+
−
! align="left" | 2005-03-05 || 140-kbps VBR<br>Classical || Guruboolez || Private
+
−
| [[AAC]] Nero 3.0.0.4, [[Vorbis]] 1.1, QK28 ||
+
−
[http://www.hydrogenaudio.org/forums/index.php?showtopic=32080&view=findpost&p=279240 HA Thread]
+
−
|-
+
−
! align="left" | 2005-03-17 || 256-kbps Transcode from MP3<br>subset of ff123 || Guruboolez || Private
+
−
| [[AAC]] iTunes 4.7, [[MP3]] [[LAME]] 3.97a8<br>[[MPC]] 1.15u, [[Vorbis]] aoTuVb3, [[WavPack]] 4.2b3 Lossy ||
+
−
[http://www.hydrogenaudio.org/forums/index.php?showtopic=32440&view=findpost&p=282909 HA Thread]
+
−
|-
+
−
! align="left" | 2005-07-11 || 80-kbps<br>Various || Guruboolez || Private
+
−
| [[LAME]] [[MP3]], [[AAC]]-HE and -LC,<br>[[Vorbis]] [[aoTuV]], [[WMA]] Std || [http://www.hydrogenaudio.org/forums/index.php?showtopic=35438&view=findpost&p=312296 HA Thread]
+
−
|-
+
−
! align="left" | 2005-08-22 || 180-kbps<br>Classical || Guruboolez || Private
+
−
| [[AAC]], [[MP3]], [[MPC]], [[Vorbis]] [[aoTuV]]b4 || [http://www.hydrogenaudio.org/forums/index.php?showtopic=36465 HA Thread]
+
−
|-
+
−
! align="left" | 2006-01-13 || 128-kbps<br>General || Sebastian Mares || Public
+
−
| iTunes & Nero [[AAC]], [[LAME]] [[MP3]],<br>[[Vorbis]] [[aoTuV]], [[WMA]] Pro || [http://www.hydrogenaudio.org/forums/index.php?showtopic=40607&view=findpost&p=357117 HA Thread]<br>[http://www.maresweb.de/listening-tests/mf-128-1/results.htm Complete Results]
+
−
|}
+
−
 
+
  
−
=External Tests=
 
−
Other people also held listening tests outside of Hydrogenaudio (although the organizer of the tests are usually HA members :)
 
  
−
So, check out the following sites:
+
[[Category:Listening Tests]]
−
* [http://ff123.net/ ff123's site on audio testing and listening tests]
+
−
* [http://www.rjamorim.com/test/index.html Roberto's Public Listening Tests]
+
−
* [http://www.soundexpert.info/ SoundExpert Listening Tests]
+

Latest revision as of 05:18, 5 March 2024

One of the most visible ways that HydrogenAudio members have shown their enthusiasm for audio quality and their respect for scientific principles over the years is by organizing and participating in public blind listening tests of lossy codecs and presenting statistical analyses of the results. These results show the advantages and disadvantages of different formats and encoders and help identify which are "state of the art."

Private listening tests also provide useful data, but the public HydrogenAudio tests are a rarity in having enough participants to both average out variations among individual listeners and still have statistically significant results.

Note that a listening test cannot really compare formats or encoder "brands." Instead, it compares particular versions of particular encoders at particular bitrates. Results will differ with different encoders, newer versions of the same encoders, or different bitrates. Use caution when extrapolating, and use the most recent results when extrapolating to the newest encoder versions.

Note: There would be no point in listening tests of lossless encoders; the decoded results are bit-identical to the original uncompressed recordings. Lossless encoder comparisons instead consider technical aspects such as compression ratio and encode/decode speed.

All tests are sorted by date, newest first. All dates are in ISO8601 format.

Tests and Results

Result Symbol Key
> Better than
~ About the same as
~> About the same, but slightly better than

Multiformat Tests

Date Title
Samples 		Performed by Codecs Result Links
2014-07 96-kbps
General 		Kamedo2 Apple AAC (iTunes 11.2.2), Opus (1.1),
Vorbis (aoTuV Beta6.03), MP3 (LAME 3.99.5) 		Opus > Apple >
MP3(V5) ~ Vorbis 		HA Thread
Complete Result
2011-04-10 64-kbps
General 		IgorC Vorbis aoTuV, Quicktime HE-AAC, Opus Opus > Apple >
Nero ~ Vorbis 		HA Thread
Complete Result
2007-08-16 64-kbps
General 		Sebastian Mares Vorbis aoTuV, Nero AAC,
WMA Pro 		Nero > WMA Pro ~ Vorbis HA Thread
Complete Result
2006-11-22 48-kbps
General 		Sebastian Mares Vorbis aoTuV, Nero AAC,
WMA Pro, WMA Standard 		Nero > WMA Pro ~ Vorbis >
WMA Standard 		HA Thread
Complete Result
2006-01-13 128-kbps
General 		Sebastian Mares QuickTime AAC, Nero AAC, LAME MP3,
Vorbis aoTuV, WMA Pro 		5-way tie HA Thread
Complete Result

AAC Tests

Date Title
Samples 		Performed by Codecs Result Links
2011-08-20 96-kbps
General 		IgorC CT, FhG AAC via Winamp (not FDK), Nero, QuickTime TVBR & CVBR Quicktime CVBR ~> TVBR ~>
Fraunhofer > CT > Nero 		HA Thread
Complete result
2006-02-20 48-kbps HE-AAC
General 		Gabriel 3gpp (v1), CT (v1 & v2), Nero (v1 & v2) All tie for most sample types;
otherwise any v1 > any v2 		HA Thread
Complete Result
2006-02-05 48-kbps HE-AAC
Classical 		Ivan Dimkovic Nero HE-AAC v1 & v2 HA Thread

MP3 Tests

Date Title
Samples 		Performed by Codecs Result Links
2008-10-17 130-kbps
General 		Sebastian Mares LAME 3.97, LAME 3.98.2, iTunes, FhG MP3, Helix Thread on hydrogenaudio
Complete Result

External Tests

Other people have held listening tests in venues outside of Hydrogenaudio (although the organizers of the tests tend to be HA members :). The results of these tests are found below:

Retrieved from "https://wiki.hydrogenaud.io/index.php?title=Hydrogenaudio_Listening_Tests&oldid=37727"