Difference between revisions of "LossyWAV"
(→Frequently asked questions: Added FAQ: "Q: Will it low-pass filter my audio? A: No.")
(→Codec compatibility: Added --merge-blocks switch to recommended Wavpack parameters)
|Line 120:||Line 120:|
Revision as of 15:57, 12 May 2013
|Use||Digital signal processing|
|Website|| 1.3.0 release thread|
1.3.0 development thread
lossyWAV is a free, lossy pre-processor for PCM audio contained in the WAV file format. Proposed by David Robinson, it reduces bit depth of the input signal, which, when used in conjunction with certain lossless codecs, reduces the bitrate of the encoded file significantly compared to unpreprocessed compression. lossyWAV's primary goal is to maintain transparency with a high degree of confidence when processing any audio data.
- 1 History
- 2 Indicative bitrate reduction
- 3 File identification
- 4 Quality presets
- 5 Supported input formats
- 6 Codec compatibility
- 7 Using lossyWAV
- 8 Frequently asked questions
- 9 External links
lossyWAV is based on the lossyFLAC idea proposed by David Robinson at Hydrogenaudio, which is a method of carefully reducing the bitdepth of (blocks of) samples which will then allow the FLAC lossless encoder to make use of its wasted bits feature. The aim is to transparently reduce audio bit depth (by making some lower significant bits (lsb's) zero), consequently taking advantage of FLAC's detection of consistently-zeroed lower significant bits within each single frame and significantly increasing coding efficiency. In this way the user can enjoy audio encoded using the same codec (which may be all important from a hardware compatibility perspective) at a reduced bitrate compared to the lossless version.
Subsequently, lossyFLAC proved itself to work with other lossless codecs, so the application name was changed to lossyWAV.
Since then, Nick has heavily developed and built upon lossyWAV, with valuable tuning performed by Horst Albrecht at Hydrogenaudio. Although the current lossyWAV implementation has built on David's original method, the method itself still very much belongs to its author.
Indicative bitrate reduction
It must be stressed that lossyWAV is a pure variable bit-depth pre-processor in that the overall sample size remains the same after processing but the number of significant bits used for the samples in a codec-block can change on a block-by-block basis. Bits-to-remove from the audio data are calculated on a block-by-block basis (codec-block length = 512 samples, 11.6msec @ 44.1kHz) using overlapping fast Fourier Transform (FFT) analyses of at least two lengths (default quality preset (-q 5) = 32, 64 & 1024 samples). After some manipulation, the results of each FFT analysis for a specific codec-block are then grouped and the minimum value used to determine bits-to-remove for the whole codec-block. Bit removal adds noise to the output, however the level of the added noise associated with the removal of a number of bits has been pre-calculated and the number of bits to remove will depend on the level of the noise floor of the codec-block in question. The added noise is adaptively shaped by default, however the user can select parameters to make the added noise fixed shaped or simply white noise. Each sample in the codec-block is then rounded such that the first <bits-to-remove> lsb's are zero. In this way the wasted bits feature of FLAC et al. is exploited.
|lossyWAV Test Set (16 bit / 44.1kHz)||Codec||lossless||--insane||--extreme||--high||--standard||--economic||--portable||--extraportable|
|10 Album Test Set||FLAC||854 kbit/s||627 kbit/s||548 kbit/s||477 kbit/s||442 kbit/s||407 kbit/s||353 kbit/s||311 kbit/s|
|Nick.C's Full Collection||FLAC||882 kbit/s||-||-||-||-||-||-||307 kbit/s|
lossyWAV-processed WAV files are named with a double filename extension, .lossy.wav, to make them instantly identifiable. e.g. ".lossy.flac" would indicate an audio file which was processed using lossyWAV, and subsequently encoded using FLAC.
The --correction parameter is used when processing to create a correction file which is named with the .lwcdf.wav double filename extension. When "added" to the corresponding .lossy.wav, using the --merge parameter, the original file will be reconstituted.
Combinations of lossyWAV with each specific encoder are referred to as lossyX, where X is an abbreviation of the lossless codec name. Combination names are listed in the "known supported codecs" section below.lossyWAV inserts a variable-length 'fact' chunk into the WAV file immediately after the 'fmt ' chunk. This takes the form:
fact/<size>/lossyWAV x.y.z @ dd/mm/yyyy hh:mm:ss, -q 5Where the version, date & time and user settings are copied. Additionally, if a lossyWAV 'fact' chunk is found in a file, the processing will be halted (exit code = 16) to prevent re-processing of an already processed file.
The --check parameter can be used to determine whether a file has previously been processed without trying to process it, exit code = 16 if already processed; exit code = 0 if not.
- --quality insane: (-q I or -q 10) Highest quality preset, generally considered to be excessive;
- --quality extreme: (-q E or -q 7.5) Higher quality preset, disc space-saving alternative to lossless archiving for large audio collections, considered to be suitable for transcoding to other lossy codecs;
- --quality high: (-q H or -q 5.0) High quality preset, midway between extreme and standard;
- --quality standard: (-q S or -q 2.5) Default preset, generally accepted to be transparent;
- --quality economic: (-q C or -q 0.0) Intermediate preset midway between standard and portable;
- --quality portable: (-q P or -q -2.5) DAP quality preset for use on a compatible DAP.
- --quality extraportable: (-q X or -q -5.0) Lowest quality preset for use on a compatible DAP.
All tuning for version 1.0.0 was performed on quality preset --standard with higher presets being more conservative. For versions 1.1.0, 1.2.0 and 1.3.0, tuning effort has been focused on the lowest quality preset in an effort to achieve an effective compromise between resultant bitrate and perceived quality. Quality preset --standard is generally accepted to be (and from testing so far is) transparent. If you find a track which --standard fails to achieve transparency after processing, please post a sample (no more than 30 seconds) in the development thread.
The upper frequency limit used in the calculation of minimum signal power varies, dependent on quality preset, in the range 15.159kHz to 16.682kHz
Supported input formats
- WAV: 9-bit to 32-bit integer; 1 to 8 channels; sample rate ≥ 32kHz PCM. Very high sample rates (>48kHz) have not been extensively tested. Tunings have been focussed on 16-bit, 44.1kHz samples (i.e. CD PCM).
|Codec||Supported||Encoder parameters||Combination name|
|FLAC||Yes||-5 -b 512 --keep-foreign-metadata||lossyFLAC|
|MPEG-4 ALS||Yes||-l -n512||lossyALS|
- Combinations of lossyWAV with each specific encoder are referred to as lossyX, where X is an abbreviation of the lossless codec name.
There is also evidence — so-called "Bit Shifting" — to suggest that lossyWAV may work with MLP, but this remains untested due to prohibitive prices of encoders. At least one commercial DVD-A uses constant bit-depth reduction with lower bit-depth on rear channels.
A comparison of portable media players is here, which shows FLAC and WMA Lossless compatibility among listed players. Any player supported by Rockbox can use FLAC or WavPack files after installing Rockbox.
NB: when encoding using a lossless codec, please ensure that the block size of the lossless codec matches that of lossyWAV (default = 512 samples). If this is not done then the lossless encoding of the processed WAV file will (almost certainly) be larger than it would otherwise have been. This is achieved by adding the "Encoder Parameters" in the table above to the command line of the lossless codec in question.
Another, possibly not obvious, feature of lossyWAV is that the processed output can be "transcoded" from one lossless codec to another lossless codec with absolutely no loss of quality whatsoever. This is solely due to the fact that lossyWAV output is designed to be losslessly encoded - something that lossless codecs do very well indeed.
lossyWAV 1.3.0, Copyright (C) 2007-2011 Nick Currie. Copyleft. This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful,but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>. Process Description: lossyWAV is a near lossless audio processor which dynamically reduces the bitdepth of the signal on a block-by-block basis. Bitdepth reduction adds noise to the processed output. The amount of permissible added noise is based on analysis of the signal levels in the default frequency range 20Hz to 16kHz. If signals above the upper limiting frequency are at an even lower level, they can be swamped by the added noise. This is usually inaudible, but the behaviour can be changed by specifying a different --limit (in the range 10kHz to 20kHz). For many audio signals there is little content at very high frequencies and forcing lossyWAV to keep the added noise level lower than the content at these frequencies can increase the bitrate dramatically for no perceptible benefit. The noise added by the process is shaped using an adaptive method provided by Sebastian Gesemann. This method, as implemented in lossyWAV, aims to use the signal itself as the basis of the filter used for noise shaping. Adaptive noise shaping is enabled by default. Usage : lossyWAV <input wav file> <options> Example : lossyWAV musicfile.wav Quality Options: -q, --quality <t> where t is one of the following (default = standard): I, insane highest quality output, suitable for transcoding; E, extreme higher quality output, suitable for transcoding; H, high high quality output, suitable for transcoding; S, standard default quality output, considered to be transparent; C, economic intermediate quality output, likely to be transparent; P, portable good quality output for DAP use, may not be transparent; X, extraportable lowest quality output, not fully transparent. Standard Options: -C, --correction write correction file for processed WAV file; default=off. -f, --force forcibly over-write output file if it exists; default=off. -h, --help display help. -L, --longhelp display extended help. -M, --merge merge existing lossy.wav and lwcdf.wav files. -o, --outdir <t> destination directory for the output file(s). -v, --version display the lossyWAV version number. -w, --writetolog create (or add to) lossyWAV.log in the output directory. Advanced Options: - take WAV input from STDIN. -c, --check check if WAV file has already been processed; default=off. errorlevel=16 if already processed, 0 if not. -q, --quality <n> quality preset (-5.0<=n<=10.0); (-5=lowest, 10=highest; default=2.5; I=10; E=7.5; H=5; S=2.5; C=0; P=-2.5; X=-5). --, --stdout write WAV output to STDOUT. --stdinname <t> pseudo filename to use when input from STDIN. Advanced Quality Options: -A, --adaptive <n/t> modify settings for Sebastian Gesemann's adaptive noise shaping method. takes a parameter to set the order of the FIR filter, (32<=n<=96; default=64; multiple of 8 only); "OFF" to disable adaptive shaping; "NOWARP" to disable default frequency warping; -a, --analyses <n> set number of FFT analysis lengths, (2<=n<=6; default=3, i.e. 32, 64 & 1024 samples. n=2, remove 32 sample FFT; n>3 add 512; n>4, add 256; n>6, add 128) nb. FFT lengths. stated are for 44.1/48kHz audio, higher sample rates will automatically increase all FFT lengths as required. -l, --limit <n> set upper frequency limit to be used in analyses to n Hz; (10000<=n<=20000; default=16000). --linkchannels revert to original single bits-to-remove value for all channels rather than channel dependent bits-to-remove. --maxclips <n> set max. number of acceptable clips per channel per block; (0<=n<=16; default=3,3,3,3,3,2,2,2,2,2,1,1,1,0,0,0). -m, --midside analyse 2 channel audio for mid/side content. --nodccorrect disable DC correction of audio data prior to FFT analysis, default=on; (DC offset calculated per FFT data set). --scale <n> factor to scale audio by; (0.0625<n<=8.0; default=1). -s, --shaping [n] enable fixed noise shaping, takes optional parameter [n] to allow user defined shaping proportion (0.0<=n<=1.0), otherwise default to quality setting dependent value. Disables adaptive noise shaping. --static <n> set minimum-bits-to-keep-static to n bits (default=6; 7<=n<=28, limited to bits-per-sample - 4). -U, --underlap <n> enable underlap mode to increase number of FFT analyses performed at each FFT length, (n = 2, 4 or 8, default=2). Output Options: --bitdist show distrubution of bits to remove. --blockdist show distribution of lowest / highest significant bit of input codec-blocks and bit-removed codec-blocks. -d, --detail enable per block per channel bits-to-remove data display. -F, --freqdist enable frequency analysis display of input data. -H, --histogram show sample value histogram (input, lossy and correction). --longdist show long frequency distribution data (input/lossy/lwcdf). --perchannel show selected distribution data per channel. -p, --postanalyse enable frequency analysis display of output and correction data in addition to input data. --sampledist show distribution of lowest / highest significant bit of input samples and bit-removed samples. --spread [full] show detailed [more detailed] results from the spreading/ averaging algorithm. -W, --width <n> select width of output options (79<=n<=255). System Options: -B, --below set process priority to below normal. --low set process priority to low. -N, --nowarnings suppress lossyWAV warnings. -Q, --quiet significantly reduce screen output. -S, --silent no screen output. Special thanks go to: David Robinson for the publication of his lossyFLAC method, guidance, and the motivation to implement his method as lossyWAV. Horst Albrecht for ABX testing, valuable support in tuning the internal presets, constructive criticism and all the feedback. Sebastian Gesemann for the adaptive noise shaping method and the amount of help received in implementing it and also for the basis of the fixed noise shaping method. Matteo Frigo and for libfftw3-3.dll contained in the FFTW distribution Steven G Johnson (v3.2.1 or v3.2.2). Mark G Beckett for the Delphi unit that provides an interface to the (Univ. of Edinburgh) relevant fftw routines in libfftw3-3.dll. Don Cross for the Complex-FFT algorithm originally used.
Example drag 'n' drop batch file
Simply drag the FLAC files onto this batch file and it will process, recode in FLAC and copy ALL of the tags from the input FLAC file, placing the output lossyFLAC file in the same directory as the input FLAC file. Requires flac.exe and tag.exe to be somewhere on the path.
@echo off :repeat if %1.==. goto end if exist "%1" flac -d "%1" --stdout --silent|lossywav - --stdout --standard --stdinname "%1"|flac - -b 512 -o "%~dpn1.lossy.flac" --silent && tag --fromfile "%1" "%~dpn1.lossy.flac" shift goto repeat :end
lossyWAV and FFTW
Since version 1.2.0, lossyWAV has been compatible with FFTW although not dependent on it. Should the user wish to take advantage of the increased processing speed available when using FFTW (from superior FFT implementations), libfftw3-3.dll should be placed in a directory on the host computer which features on the path.
lossyWAV and WINE
The cause of lossyWAV's WINE incompatibility was found and removed during the development of 1.2.0 and retrospectively amended for 1.1.0b in a maintenance release (1.1.0c).
lossyWAV and foobar2000
Example foobar2000 converter settings:lossyFLAC settings:
Encoder: C:\Windows\System32\cmd.exe Extension : lossy.flac Parameters: /d /c C:\"Program Files"\bin\lossywav - --quality standard --silent --stdout| C:\"Program Files"\bin\flac - -b 512 -5 -f -o%d --ignore-chunk-sizes Format is : lossless or hybrid Highest BPS mode supported: 24lossyTAK settings:
Encoder: C:\Windows\System32\cmd.exe Extension : lossy.tak Parameters : /d /c C:\"Program Files"\bin\lossywav - --quality standard --silent --stdout| C:\"Program Files"\bin\takc -e -p2m -fsl512 -ihs - %d Format is: lossless or hybrid Highest BPS mode supported: 24lossyWV settings:
Encoder: C:\Windows\System32\cmd.exe Extension : lossy.wv Parameters: /d /c C:\"Program Files"\bin\lossywav - --quality standard --silent --stdout| C:\"Program Files"\bin\wavpack -hm --blocksize=512 --merge-blocks -i - %d Format is : lossless or hybrid Highest BPS mode supported: 24lossyWMALSL* settings:
Encoder: C:\Windows\System32\cmd.exe Extension : lossy.wma Parameters : /d /c c:\"program files"\bin\lossywav - --quality standard --silent --stdout| c:\"program files"\bin\wmaencode - %d --codec lsl --ignorelength Format is : lossless or hybrid Highest BPS mode supported: 24
Enclose the element of the path containing spaces within double quotation marks ("), e.g. C:\"Program Files"\directory_where_executable_is\executable_name. This is a Windows limitation.
lossyWMALSL conversion uses WMAEncode.exe by lvqcl found here.
lossyWAV and EAC
- For example settings, see EAC and LossyWAV.
Frequently asked questions
- Question: Why is the ".wav" file extension used?
- Answer: The ".wav" file extension is used because lossyWAV is a digital signal processor and not a codec. No decoding is required for any program to play a WAV file which has been processed with lossyWAV as it remains compliant with the RIFF WAVE format.
- Question: Why create a processor which means that I cannot be sure that a lossless file is truly lossless?
- Answer: Unless one creates the lossless file personally, one can never be completely sure that the file is indeed lossless. E.g. a lossless file you receive could be transcoded from MP3 without your knowledge. To distinguish a lossyWAV file from lossless files it is recommended to use the extension .lossy.EXT where EXT is the original extension e.g. .lossy.flac
- Question: Is it VBR?
- Short answer: Yes.
- Question: Do I need to re-process to change lossless codecs?
- Short answer: No.
- Question: Is it transparent?
- Short answer: At preset --standard, almost certainly.
- Question: Is it lossless?
- Short answer: No.
- Question: Will it ever have a CBR mode?
- Short answer: No.
- Question: Will it low-pass filter my audio?
- Short answer: No. The frequency limit is for the analysis only. LossyWAV cannot low-pass filter your audio.
- Question: Why should I use this?
- Original lossyFLAC thread - Introduction of the concept by David Robinson (Replay Gain developer) and initial development
- lossyWAV 1.3.0 development thread
- lossyWAV 1.3.0 release thread - Release of version 1.3.0 on 06 August 2011
- lossyWAV 1.2.0 development thread
- lossyWAV 1.2.0 release thread - Release of version 1.2.0 on 16 December 2009
- lossyWAV 1.1.0 development thread
- lossyWAV 1.1.0 release thread - Release of version 1.1.0 on 12 July 2008