FLAC decoder testbench: Difference between revisions

Revision as of 20:15, 22 November 2021

To further the use and development of FLAC, a testbench containing files with different parameters was constructed. The testbench consists of 59 FLAC files, each one having a different combination of characteristics (like samplerate, bitdepth and number of channels) and FLAC features (like blocksize, maximum partition order, maximum predictor order, maximum predictor precision etc.) However, all combinations fall within the so called subset: a limited set of FLAC parameters that fit in the frame header and are supposed to be 'streamable'. Please be aware that certain tested features (such as blocksize 16 or a samplerate of 38457Hz) are pretty much useless in day-to-day usage, and that failure to properly decode such a file is mostly of interest to developers than to users.

For more information on which file tests what characteristics and features, please refer to the readme.txt file that is included in the testbench. This testbench is licensed under CC0 1.0.

The testbench can be downloaded here

	CDDA (44.1kHz/16-bit and less)					High-res (96kHz/24-bit and more)					Multichannel		Metadata
Decoder	Blocksize	Features	Samplerates	Bitdepths	Variable blocksize	General	Blocksize	Features	Samplerates	Bitdepths	Channel lay-outs	Features	Missing	Extreme	MIME	Remarks
Files in test set	1-10	11-18	19-21	22-23	24-27	28	29-30	31-32, 34	33-36	37	38-43	44	45-47	48-55	56-59
Android	Fails	Plays	Plays	Skips	Plays	Plays	Plays	Plays	Plays	Fails	Fails	Skips	Plays	Plays	Android 11 with Google Play updates of 1st of August 2021 on Samsung XCover 5
Citroën DS5	Plays	Plays	Skips	Fails	Plays	Skips							Plays	Hangs	Plays
Chromecast	Fails	Plays	Fails	Plays	Fails	Plays	Fails	Fails	Fails	Plays	Rejects		Plays	Reboots	Rejects	Heavy stutter with blocksize 16. Mangles non-standard samplerates. Doesn't start playing on old variable blocksize format, blocksizes 16384 and 13456, predictor order 32, 384kHz. Skips multichannel and image/avif. Crashes on file with 1000x repeating VORBISCOMMENT
Cowon iAudio 7	Fails	Plays	Fails	Rejects	Rejects	Rejects	Rejects	Rejects	Rejects	Rejects	Freezes		-	-	-	Freezes on blocksizes 254, 725; skips file with blocksize 1937. 38457Hz and 41kHz are speeded up; 22050 Plays. All files with bitdepth other than 16 are skipped. Files with variable block size are skipped. 3.0 plays as silence, others 16/44 MCH files make freeze.
JVC KD-R871BT	Fails	Plays	Fails	Plays	Rejects	Plays	Freezes	Reboots	Fails	Rejects	Rejects		Rejects	Reboots	Rejects	Heavy stutter with blocksize 16. Stretches non-standard samplerates. Samplerates above 96kHz, variable blocksize files and multichannel files 'not supported'. Freezes on blocksize 16384 and 13456. 192kHz is played back very slow. Skips all multichannel, files with cover image, extreme metadata. Reboot loop on extreme large VORBISCOMMENT
Kenwood KMM-BT304	Fails	Plays	Rejects	Plays	Rejects	Plays	Rejects	Rejects	Rejects	Plays	Rejects	Rejects	Plays	Reboots	Plays	File 03 plays extremely noisily, barely recognizable. File 51 produces a "NO DEVICE" error and a reboot. File 54 produces a reboot with no message. These files produce a "NA FILE" error and the player skips to the next: 19 21 24 25 26 27 29 30 31 34 35 36 38 39 40 41 42 43 44 45 46 48 55. (firmware: te1706k.krm from 2020-03-05)
Onkyo TX-NR686	Fails	Plays	Rejects	Plays	Rejects	Plays	Plays	Plays	Fails	Plays	Rejects		Plays	Reboots	Plays	Tested with revision 5, firmware version 1071-2000-1070-0011.
Panasonic TX-60CX750E	Plays	Plays	Fails	Plays	Plays	Plays	Plays	Plays	Fails	Plays	Plays	Fails	Plays	Fails	Plays	Based on libFLAC 1.2.1, according to copyright notice. Firmware version 3.228, testbench revision 5
Pioneer BDP-150	Freezes	Fails	Fails	Plays	Freezes	Plays	Freezes	Fails	Fails	Plays	Freezes		Freezes	Freezes	Plays	Firmware v1.08, testbench revision 5
Rockbox	Plays	Plays	Plays	Plays	Plays	Plays	Skips	Skips	Plays	Plays	Freezes		Freezes	Fails	Plays	Tested with Rockbox 3.15 (latest stable) on Sansa Fuze with testbench revision 5
Software players
Foobar 2000	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Skips	Plays	Foobar2000 version 1.6.7, testbench revision 5
VLC	Plays	Plays	Plays	Plays	Fails	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Crashes	Plays	VLC 3.0.16 on testbench revision 5
Winamp 5.8	Plays	Plays	Plays	Freezes	Plays	Plays	Plays	Plays	Freezes	Plays	Freezes		Plays	Plays	Plays	Winamp 5.8 build 3660, testbench revision 5
Windows Media Player	Plays	Plays	Plays	Rejects	Plays	Plays	Plays	Plays	Plays	Rejects	Plays	Plays	Plays	Rejects	Plays	Tested version 12.0.19041.1288 on testbench revision 5
mpv	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Plays	Hangs	Plays	mpv-x86_64-20211031-git-4a80de9 on Windows, testbench revision 5
Mediamonkey 5	Plays	Plays	Plays	Crashes	Plays	Plays	Plays	Plays	Plays	Silence	Breaks	Plays	Plays	Unresp.	Plays	Version 5.0.1.2433 with testbench revision 5

Table legend
Plays	all files in this category are played as intended, without glitches, changes etc.
Rejects	One or more files are rejected by the player, either by skipping the file or mentioning a message like: "file not supported"
Fails	One or more files are not played back as intended, for example by stuttering, garbling or playing at incorrect speed, or by never starting playing (though not skipping, freezing or rebooting)
Freezes	On one or more files, the player crashes on decoding, leaving the unit inoperable until reboot, or the unit reboots by itself
The order of precedence is from bottom to top, so if one file freezes the player, one file is rejected and the other play fine, the table reads 'Freezes' for that category.
For the metadata tests, the (absence of) rendering of the metadata is not a criterium, simply whether its presence is an obstacle to playing

@@ Line 47: / Line 47: @@
 | 48-55
 | 56-59
+|-
+! Android
+| style="background: orange;" | {{tooltip|Fails|Heavy stuttering on blocksize 16}}
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: yellow;" | {{tooltip|Skips|Skips all variable blocksize files}}
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| style="background: orange;" | {{tooltip|Fails|No sound on 20 bit file}}
+| style="background: orange;" | {{tooltip|Fails|Skips 3.0, 4.0, 5.0 and 6.1, mutilated audio on 7.1}}
+| style="background: yellow;" | {{tooltip|Skips|File skipped}}
+| style="background: lime;"   | Plays
+| style="background: lime;"   | Plays
+| class="mw-collapsible mw-collapsed" | Android 11 with Google Play updates of 1st of August 2021 on Samsung XCover 5
 |-
 ! Citroën DS5
@@ Line 165: / Line 182: @@
 |-
 ! Pioneer BDP-150
-| style="background: red;"    | Freezes
+| style="background: red;"    | {{tooltip|Freezes|Freezes on non-standard blocksizes (16, 254, 725, 1000 and 1937)}}
-| style="background: orange;" | Fails
+| style="background: orange;" | {{tooltip|Fails|Garbled audio when playing file with LPC with qlp precision of 2}}
-| style="background: orange;" | Fails
+| style="background: orange;" | {{tooltip|Fails|Stretches non-standard samplerates to 44.1kHz}}
 | style="background: lime;"   | Plays
-| style="background: red;"    | Freezes
+| style="background: red;"    | {{tooltip|Freezes|Freezes on variable blocksize files with non-standard blocksizes, stutters on variable blocksize files with standard blocksizes}}
 | style="background: lime;"   | Plays
-| style="background: red;"    | Freezes
+| style="background: red;"    | {{tooltip|Freezes|Freezes on non-standard blocksize (13456)}}
-| style="background: orange;" | Fails
+| style="background: orange;" | {{tooltip|Fails| Incorrect decoding (clicks) of RICE2 partitions with escape codes}}
-| style="background: orange;" | Fails
+| style="background: orange;" | {{tooltip|Fails|Stretches non-standard samplerates to 44.1kHz. Stutters on 192kHz with 32th order predictors}}
 | style="background: lime;"   | Plays
-| style="background: red;" colspan="2" | Freezes
+| style="background: red;" colspan="2" | {{tooltip|Freezes|Freezes on files with more than 6 channels}}
-| style="background: red;"    | Freezes
+| style="background: red;"    | {{tooltip|Freezes|Freezes on files with an unknown number of total samples}}
-| style="background: red;"    | Freezes
+| style="background: red;"    | {{tooltip|Freezes|Freezes on files with 1000x repeated vorbiscomments, rejects files with too many seekpoints}}
 | style="background: lime;"   | Plays
-| class="mw-collapsible mw-collapsed" | Stretches non-standard samplerates to 44.1kHz. Freezes on non-standard blocksizes (16, 254, 725, 1000, 1937 and 13456), on variable blocksize files with non-standard blocksizes, files more than 6 channels, files with unknown number of total samples, files with 1000x repeated vorbiscomments. Stutters on variable blocksize files with standard blocksizes. Rejects files with too many seekpoints. Incorrect decoding (clicks) of RICE2 partitions with escape codes. Incorrect decoding (garbled audio) of LPC with qlp precision of 2. Stutters on 192kHz with 32th order predictors. Firmware v1.08, testbench revision 5
+| class="mw-collapsible mw-collapsed" | Firmware v1.08, testbench revision 5
 |-
 ! Rockbox
@@ Line 196: / Line 213: @@
 | style="background: orange;"| {{tooltip|Fails|Plays extremely slowly on file with APPLICATION metadata block}}
 | style="background: lime;"  | Plays
-| class="mw-collapsible mw-collapsed" |  Tested with Rockbox 3.13 on Sansa Fuze
+| class="mw-collapsible mw-collapsed" |  Tested with Rockbox 3.15 (latest stable) on Sansa Fuze with testbench revision 5
 |-
 ! colspan="16" | Software players