BS.1387 - Revision history

Artoria2e5: /* Other objective metrics */

2023-08-25T00:07:08Z

Other objective metrics

← Older revision		Revision as of 00:07, 25 August 2023
Line 56:		Line 56:
	* CDPAM is allegedly the model that comes closest to human datasets. Unfortunately the only pre-trained model works on 22050 Hz. And it's academic neural-network code -- not something you can expect to run on first try.		* CDPAM is allegedly the model that comes closest to human datasets. Unfortunately the only pre-trained model works on 22050 Hz. And it's academic neural-network code -- not something you can expect to run on first try.

	There are also much more primitive methods that don't attempt anything perceptual, preferred by peddlers of Bluetooth codecs:		There are also much more primitive methods that don't attempt anything perceptual, preferred by peddlers of Bluetooth codecs (they make more sense for analogue systems like DACs):
	* SNR		* SNR
	* THD+N		* THD+N

Artoria2e5: /* GstPEAQ */

2023-08-10T15:07:15Z

GstPEAQ

← Older revision		Revision as of 15:07, 10 August 2023
Line 35:		Line 35:
	[https://github.com/HSU-ANT/gstpeaq GstPEAQ] is an implementation of PEAQ, ''both'' basic and advanced, in GStreamer. In addition to the ODG, it also outputs the distortion index (DI), which is not clipped at extremes and not fitted to score anchors. On the HA multiformat dataset:		[https://github.com/HSU-ANT/gstpeaq GstPEAQ] is an implementation of PEAQ, ''both'' basic and advanced, in GStreamer. In addition to the ODG, it also outputs the distortion index (DI), which is not clipped at extremes and not fitted to score anchors. On the HA multiformat dataset:
	* The advanced model gives a correlation improvement of ~0.2 over basic;		* The advanced model gives a correlation improvement of ~0.2 over basic;
	* DI is slightly better at predicting subjective scores than ODG, with a correlation improvement of ~0.04.		* DI is slightly better at predicting subjective scores than ODG, with a correlation improvement of ~0.03.

	== Comparison with subjective listening tests ==		== Comparison with subjective listening tests ==

Artoria2e5 at 15:06, 10 August 2023

2023-08-10T15:06:38Z

← Older revision		Revision as of 15:06, 10 August 2023
Line 31:		Line 31:
	* [http://www.rarewares.org/others.html EAQUAL Tools] zip compression archive of the utility used to perform EAQUAL tests provided by Rarewares.		* [http://www.rarewares.org/others.html EAQUAL Tools] zip compression archive of the utility used to perform EAQUAL tests provided by Rarewares.
	* [https://github.com/ivan-codelegs/eaqual ivan-codelegs] github fork, adds macOS support		* [https://github.com/ivan-codelegs/eaqual ivan-codelegs] github fork, adds macOS support

			== GstPEAQ ==
			[https://github.com/HSU-ANT/gstpeaq GstPEAQ] is an implementation of PEAQ, ''both'' basic and advanced, in GStreamer. In addition to the ODG, it also outputs the distortion index (DI), which is not clipped at extremes and not fitted to score anchors. On the HA multiformat dataset:
			* The advanced model gives a correlation improvement of ~0.2 over basic;
			* DI is slightly better at predicting subjective scores than ODG, with a correlation improvement of ~0.04.

	== Comparison with subjective listening tests ==		== Comparison with subjective listening tests ==
Line 43:		Line 48:
	* [https://sourceforge.net/projects/peaqb/ peaqb] is another implementation of PEAQ. Last updated 2003.		* [https://sourceforge.net/projects/peaqb/ peaqb] is another implementation of PEAQ. Last updated 2003.
	* There a good number of Matlab implementations for researchers. But it's Matlab, so there's gonna be academic code smell.		* There a good number of Matlab implementations for researchers. But it's Matlab, so there's gonna be academic code smell.

	There is one advanced-version implementation: [https://github.com/HSU-ANT/gstpeaq GstPEAQ], which builds on GStreamer. It deviates from the standard less than the previous ones do, and most importantly has the advanced stuff. If any future experimentation on PEAQ is to be done, GstPEAQ should be used.

	== Other objective metrics ==		== Other objective metrics ==

Artoria2e5: /* Comparison with subjective listening tests */

2023-08-10T15:03:25Z

Comparison with subjective listening tests

← Older revision		Revision as of 15:03, 10 August 2023
Line 36:		Line 36:
	* [https://hydrogenaud.io/index.php/topic,124607.msg1031323.htmlGstPEAQ: PEAQ done right, allegedly \|\| Multiformat correlation] - great Pearson correlation (0.924, DI Adv): samples of three quality groups		* [https://hydrogenaud.io/index.php/topic,124607.msg1031323.htmlGstPEAQ: PEAQ done right, allegedly \|\| Multiformat correlation] - great Pearson correlation (0.924, DI Adv): samples of three quality groups

	~~It remains inconclusive whether EAQUAL can replace~~ human raters ~~at all~~. ~~Codec developers and researchers do find~~ PEAQ useful.		HA comparisons between PEAQ and human raters remain inconclusive. PEAQ is considered useful for an approximation of human senses in codec development and research, but concrete results still need human participation.

	== Other implementations ==		== Other implementations ==

Artoria2e5 at 15:01, 10 August 2023

2023-08-10T15:01:45Z

← Older revision		Revision as of 15:01, 10 August 2023
Line 12:		Line 12:

	== EAQUAL ==		== EAQUAL ==
	'''EAQUAL''' (''Evaluation Of Audio Quality'') is an open-source software that implements PEAQ's basic model ''only''. Several tests have been performed using EAQUAL most notably using numerous [http://www.hydrogenaudio.org/forums/index.php?showtopic=20264 AAC encoders] to determine via a [http://en.wikipedia.org/wiki/Pearson_correlation Pearson Correlation] the linear relationship between human ratings and EAQUAL ratings on a given set of test samples. The results, however when using objective testing methodologies are still inconclusive and mostly only used by codec developers and researchers.		'''EAQUAL''' (''Evaluation Of Audio Quality'') is an open-source software that implements PEAQ's basic model ''only''.

	=== Invoking EAQUAL ===		=== Invoking EAQUAL ===
Line 31:		Line 31:
	* [http://www.rarewares.org/others.html EAQUAL Tools] zip compression archive of the utility used to perform EAQUAL tests provided by Rarewares.		* [http://www.rarewares.org/others.html EAQUAL Tools] zip compression archive of the utility used to perform EAQUAL tests provided by Rarewares.
	* [https://github.com/ivan-codelegs/eaqual ivan-codelegs] github fork, adds macOS support		* [https://github.com/ivan-codelegs/eaqual ivan-codelegs] github fork, adds macOS support

			== Comparison with subjective listening tests ==
			* [http://www.hydrogenaudio.org/forums/index.php?showtopic=20264 EAqual results for the AAC@128v2 listening test] - fair Pearson correlation (0.699) among higher-quality samples: all AAC
			* [https://hydrogenaud.io/index.php/topic,124607.msg1031323.htmlGstPEAQ: PEAQ done right, allegedly \|\| Multiformat correlation] - great Pearson correlation (0.924, DI Adv): samples of three quality groups

			It remains inconclusive whether EAQUAL can replace human raters at all. Codec developers and researchers do find PEAQ useful.

	== Other implementations ==		== Other implementations ==

Artoria2e5: /* Interpreting EAQUAL output */

2023-08-10T06:08:50Z

Interpreting EAQUAL output

← Older revision		Revision as of 06:08, 10 August 2023
Line 20:		Line 20:

	=== Interpreting EAQUAL output ===		=== Interpreting EAQUAL output ===
	EAQUAL outputs one score, the PEAQ "ODG" rating. This ODG (Objective Difference Grade) rating is designed by ITU to match an SDG (Subjective Difference Grade) rating, which is the difference between the subjective (1–5) scores between the two input samples.		EAQUAL outputs one score, the PEAQ "ODG" rating. This ODG (Objective Difference Grade) rating is designed by ITU to match an SDG (Subjective Difference Grade) rating, which is the difference between the subjective (1–5) scores between the two input samples. Assuming the HydrogenAudio subjective scoring system, where the reference sample is always scored as a perfect 5, adding 5 points to ODG should produce an approximation of the subjective score.

	=== Status of the project ===		=== Status of the project ===

Artoria2e5: /* Other objective metrics */

2023-08-10T06:03:12Z

Other objective metrics

← Older revision		Revision as of 06:03, 10 August 2023
Line 46:		Line 46:
	* [https://github.com/google/visqol VISQOL] is Google's open-source metric. It works for both speech and music, but the neural network (don't worry, it runs fast enough on a CPU) is trained for short clips only. Maybe someone can write a tool to one file into many clips and see individual segment scores.		* [https://github.com/google/visqol VISQOL] is Google's open-source metric. It works for both speech and music, but the neural network (don't worry, it runs fast enough on a CPU) is trained for short clips only. Maybe someone can write a tool to one file into many clips and see individual segment scores.
	* CDPAM is allegedly the model that comes closest to human datasets. Unfortunately the only pre-trained model works on 22050 Hz. And it's academic neural-network code -- not something you can expect to run on first try.		* CDPAM is allegedly the model that comes closest to human datasets. Unfortunately the only pre-trained model works on 22050 Hz. And it's academic neural-network code -- not something you can expect to run on first try.

			There are also much more primitive methods that don't attempt anything perceptual, preferred by peddlers of Bluetooth codecs:
			* SNR
			* THD+N

	== External links==		== External links==

Artoria2e5: /* Other implementations */

2023-08-10T05:56:00Z

Other implementations

← Older revision		Revision as of 05:56, 10 August 2023
Line 38:		Line 38:
	* There a good number of Matlab implementations for researchers. But it's Matlab, so there's gonna be academic code smell.		* There a good number of Matlab implementations for researchers. But it's Matlab, so there's gonna be academic code smell.

	There is one advanced-version implementation: [https://github.com/HSU-ANT/gstpeaq GstPEAQ], which builds on GStreamer. It deviates from the standard less than the previous ones do, and most importantly has the advanced stuff.		There is one advanced-version implementation: [https://github.com/HSU-ANT/gstpeaq GstPEAQ], which builds on GStreamer. It deviates from the standard less than the previous ones do, and most importantly has the advanced stuff. If any future experimentation on PEAQ is to be done, GstPEAQ should be used.

	== Other objective metrics ==		== Other objective metrics ==

Artoria2e5: /* Other implementations */

2023-08-10T05:55:11Z

Other implementations

← Older revision		Revision as of 05:55, 10 August 2023
Line 38:		Line 38:
	* There a good number of Matlab implementations for researchers. But it's Matlab, so there's gonna be academic code smell.		* There a good number of Matlab implementations for researchers. But it's Matlab, so there's gonna be academic code smell.

	There is one advanced-version implementation: [https://github.com/HSU-ANT/gstpeaq GstPEAQ], which builds on GStreamer. It ~~very slightly~~ deviates from the standard.		There is one advanced-version implementation: [https://github.com/HSU-ANT/gstpeaq GstPEAQ], which builds on GStreamer. It deviates from the standard less than the previous ones do, and most importantly has the advanced stuff.

	== Other objective metrics ==		== Other objective metrics ==

Artoria2e5: /* Other objective metrics */

2023-08-10T05:42:25Z

Other objective metrics

← Older revision		Revision as of 05:42, 10 August 2023
Line 44:		Line 44:

	* ITU also has PESQ and POLQA, both designed for speech.		* ITU also has PESQ and POLQA, both designed for speech.
	* [https://github.com/google/visqol VISQOL] is Google's open-source metric. It works for both speech and music, but is trained for short clips only. Maybe someone can write a tool to one file into many clips and see individual segment scores.		* [https://github.com/google/visqol VISQOL] is Google's open-source metric. It works for both speech and music, but the neural network (don't worry, it runs fast enough on a CPU) is trained for short clips only. Maybe someone can write a tool to one file into many clips and see individual segment scores.
			* CDPAM is allegedly the model that comes closest to human datasets. Unfortunately the only pre-trained model works on 22050 Hz. And it's academic neural-network code -- not something you can expect to run on first try.

	== External links==		== External links==