UTF-8 - Revision history

Pepoluan: Reformat, categorized, kill that funky-wannabe spelling

2006-09-11T21:36:49Z

Reformat, categorized, kill that funky-wannabe spelling

← Older revision		Revision as of 21:36, 11 September 2006
Line 1:		Line 1:
	==UTF-8==		'''UTF-8''' stands for '''UCS Transformation Format 8 bit'''. It is a upward compatible way to portably encode all languages on this planet.

	~~UTF-8 stands for UCS Transformation Format 8 bit. It is a upward compatible way to portable encode all languages on this planet.~~		The following ASCII control characters (Range: 0x00...0x1F, 0x7F) are allowed:

	~~Another remark:~~
	The following control characters (Range: 0x00...0x1F, 0x7F) are allowed:
	*0x0A: Line feed (Unix Way)		*0x0A: Line feed (Unix Way)
	*0x0C: Form feed (with intrinsic line feed)		*0x0C: Form feed (with intrinsic line feed)


	===UTF-8 ~~has the following properties:=~~==		==UTF-8 Properties==

	* UCS characters U+0000 to U+007F (ASCII) are encoded simply as bytes 0x00 to 0x7F (ASCII compatibility).
	*This means that files and strings which contain only 7-bit ASCII characters have the same encoding under both ASCII and UTF-8.
	*All UCS characters >U+007F are encoded as a sequence of several bytes, each of which has the most significant bit set. Therefore, no ASCII byte (0x00 to 0x7F) can appear as part of any other character.
	*The first byte of a multibyte sequence that represents a non-ASCII character is always in the range 0xC0 to 0xFD and it indicates how many bytes follow for this character. All further bytes in a multibyte sequence are in the range 0x80 to 0xBF. This allows easy resynchronization and makes the encoding stateless and robust against missing bytes.
	*UTF-8 encoded characters may theoretically be up to six bytes long, however 16-bit BMP characters are only up to three bytes long.
	*The sorting order of Bigendian UCS-4 byte strings is preserved.
	*The bytes 0xFE and 0xFF are never used in the UTF-8 encoding.

	~~Used in [[APE Tag Item]]s~~


	~~===Weblinks:===~~		* UCS characters U+0000 to U+007F (ASCII) are encoded simply as bytes 0x00 to 0x7F (ASCII compatibility). <br />This means that files and strings which contain only 7-bit ASCII characters have the same encoding under both ASCII and UTF-8.
			* All UCS characters >U+007F are encoded as a sequence of several bytes, each of which has the most significant bit set.<br />Therefore, no ASCII byte (0x00 to 0x7F) can appear as part of any other character.
	*~~[http://www~~.~~unicode.org~~/ ~~Unicode~~.~~org]~~		* The first byte of a multibyte sequence that represents a non-ASCII character is always in the range 0xC0 to 0xFD and it indicates how many bytes follow for this character. All further bytes in a multibyte sequence are in the range 0x80 to 0xBF.<br />This allows easy resynchronization and makes the encoding stateless and robust against missing bytes.
	*~~[http:~~/~~/www.unicode~~.~~org/charts/ Glyph tables]~~		* UTF-8 encoded characters may theoretically be up to six bytes long, however 16-bit BMP characters are only up to three bytes long.
	*~~[http://www~~.cl.~~cam.ac.uk~~/~~~mgk25/unicode.html Markus G~~. ~~Kuhn's Unicode Page] (University of Cambridge, UK)~~		* The sorting order of Bigendian UCS-4 byte strings is preserved.
	*~~[http://www.columbia.edu/kermit/utf8.html~~ UTF-8 ~~sampler] (Web browser test)~~		* The bytes 0xFE and 0xFF are never used in the UTF-8 encoding. <br />Instead, they have the important function to indicate endian-ness of the UTF-8 encoded file.
	*~~[http://www~~.~~microsoft~~.~~com~~/~~typography/unicode/cscp~~.~~htm Codepages used by OS/2 and Windows]~~

			==API Calls==

	===Windows API===		===Windows API===
Line 59:		Line 45:


	Interface is very similar to Windows API, but ~~mr crptc t b mr dffclt t ndrstnd~~.		Interface is very similar to Windows API, but more cryptic thus more diffcult to understand.

	Convert current locale (multibyte) to Unicode (wide character) and then encode to UTF-8 using the simple generic scheme below. Behaviour of function depends on locale settings of the enviroment variable $LC_CTYPE.		Convert current locale (multibyte) to Unicode (wide character) and then encode to UTF-8 using the simple generic scheme below. Behaviour of function depends on locale settings of the enviroment variable $LC_CTYPE.


	===Conversion scheme===		==Conversion scheme==


	{\|border="1" cellspacing="1"		{\|border="1" cellspacing="1"
Line 130:		Line 115:
	Examples: The Unicode character U+00A9 = 00010 101001 (copyright sign) is encoded in UTF-8 as		Examples: The Unicode character U+00A9 = 00010 101001 (copyright sign) is encoded in UTF-8 as

	11000010 10101001 = 0xC2 0xA9		11000010 10101001 = 0xC2 0xA9

	and character U+2260 = 0010 001001 100000 (not equal to) is encoded as:		and character U+2260 = 0010 001001 100000 (not equal to) is encoded as:

	11100010 10001001 10100000 = 0xE2 0x89 0xA0		11100010 10001001 10100000 = 0xE2 0x89 0xA0


			==Additional Reading==

			* [http://www.unicode.org/ Unicode.org]
			* [http://www.unicode.org/charts/ Glyph tables]
			* [http://www.cl.cam.ac.uk/~mgk25/unicode.html Markus G. Kuhn's Unicode Page] (University of Cambridge, UK)
			* [http://www.columbia.edu/kermit/utf8.html UTF-8 sampler] (Web browser test)
			* [http://www.microsoft.com/typography/unicode/cscp.htm Codepages used by OS/2 and Windows]


			[[Category:Technical]]

Pepoluan: APEtag-UTF8 moved to UTF-8: The information here applies to '''all''' UTF-8 using fields; not only APE tags.

2006-09-11T21:30:09Z

APEtag-UTF8 moved to UTF-8: The information here applies to '''all''' UTF-8 using fields; not only APE tags.

← Older revision	Revision as of 21:30, 11 September 2006
(No difference)

Rjamorim at 16:56, 27 April 2005

2005-04-27T16:56:01Z

← Older revision		Revision as of 16:56, 27 April 2005
Line 65:		Line 65:

	===Conversion scheme===		===Conversion scheme===


	{\|border="1" cellspacing="1"		{\|border="1" cellspacing="1"
Line 123:		Line 124:
	\|\|10ssssss		\|\|10ssssss
	\|}		\|}


	Only the shortest possible multibyte sequence which can represent the code number of the character can be used. Note that in multibyte sequences, the number of leading 1 bits in the first byte is identical to the number of bytes in the entire sequence.		Only the shortest possible multibyte sequence which can represent the code number of the character can be used. Note that in multibyte sequences, the number of leading 1 bits in the first byte is identical to the number of bytes in the entire sequence.

Rjamorim at 16:53, 27 April 2005

2005-04-27T16:53:15Z

← Older revision		Revision as of 16:53, 27 April 2005
Line 113:		Line 113:
	\|\|10uuuuuu		\|\|10uuuuuu
	\|\|10vvvvvv		\|\|10vvvvvv
			\|-
			\|U-04000000... U-7FFFFFFF
			\|\|0xyyyyyy zzzzzzuu uuuuvvvv vvssssss
			\|\|1111110x
			\|\|10yyyyyy
			\|\|10zzzzzz
			\|\|10uuuuuu
			\|\|10vvvvvv
			\|\|10ssssss
			\|}

			Only the shortest possible multibyte sequence which can represent the code number of the character can be used. Note that in multibyte sequences, the number of leading 1 bits in the first byte is identical to the number of bytes in the entire sequence.

			Examples: The Unicode character U+00A9 = 00010 101001 (copyright sign) is encoded in UTF-8 as

			11000010 10101001 = 0xC2 0xA9

			and character U+2260 = 0010 001001 100000 (not equal to) is encoded as:

			11100010 10001001 10100000 = 0xE2 0x89 0xA0

Jan at 16:09, 27 April 2005

2005-04-27T16:09:59Z

← Older revision		Revision as of 16:09, 27 April 2005
Line 100:		Line 100:
	\|-		\|-
	\|U-00010000... U-001FFFFF		\|U-00010000... U-001FFFFF
	\|\|~~<p style="font-family: 'fontname'>~~00000000 000xxxyy yyyyzzzz zzuuuuuu~~</p>~~		\|\|00000000 000xxxyy yyyyzzzz zzuuuuuu
	\|\|11110xxx		\|\|11110xxx
	\|\|10yyyyyy		\|\|10yyyyyy

Jan at 16:04, 27 April 2005

2005-04-27T16:04:51Z

← Older revision		Revision as of 16:04, 27 April 2005
Line 100:		Line 100:
	\|-		\|-
	\|U-00010000... U-001FFFFF		\|U-00010000... U-001FFFFF
	\|\|<p ~~<nowiki>~~style="font-family: 'fontname'>00000000 000xxxyy yyyyzzzz zzuuuuuu~~</nowiki>~~</p>		\|\|<p style="font-family: 'fontname'>00000000 000xxxyy yyyyzzzz zzuuuuuu</p>
	\|\|11110xxx		\|\|11110xxx
	\|\|10yyyyyy		\|\|10yyyyyy

Jan at 16:04, 27 April 2005

2005-04-27T16:04:24Z

← Older revision		Revision as of 16:04, 27 April 2005
Line 100:		Line 100:
	\|-		\|-
	\|U-00010000... U-001FFFFF		\|U-00010000... U-001FFFFF
	\|\|<p style="font-family: 'fontname'>00000000 000xxxyy yyyyzzzz zzuuuuuu</p>		\|\|<p <nowiki>style="font-family: 'fontname'>00000000 000xxxyy yyyyzzzz zzuuuuuu</nowiki></p>
	\|\|11110xxx		\|\|11110xxx
	\|\|10yyyyyy		\|\|10yyyyyy

Jan at 16:02, 27 April 2005

2005-04-27T16:02:50Z

← Older revision		Revision as of 16:02, 27 April 2005
Line 100:		Line 100:
	\|-		\|-
	\|U-00010000... U-001FFFFF		\|U-00010000... U-001FFFFF
	\|\|~~<nowiki>~~<p style="font-family: 'fontname'>00000000 000xxxyy yyyyzzzz zzuuuuuu</p~~></nowiki~~>		\|\|<p style="font-family: 'fontname'>00000000 000xxxyy yyyyzzzz zzuuuuuu</p>
	\|\|11110xxx		\|\|11110xxx
	\|\|10yyyyyy		\|\|10yyyyyy

Jan at 16:02, 27 April 2005

2005-04-27T16:02:22Z

← Older revision		Revision as of 16:02, 27 April 2005
Line 100:		Line 100:
	\|-		\|-
	\|U-00010000... U-001FFFFF		\|U-00010000... U-001FFFFF
	\|\|<~~nowiki></~~nowiki><p style="font-family: 'fontname'>00000000 000xxxyy yyyyzzzz zzuuuuuu</p>		\|\|<nowiki><p style="font-family: 'fontname'>00000000 000xxxyy yyyyzzzz zzuuuuuu</p></nowiki>
	\|\|11110xxx		\|\|11110xxx
	\|\|10yyyyyy		\|\|10yyyyyy

Jan at 16:02, 27 April 2005

2005-04-27T16:02:07Z

← Older revision		Revision as of 16:02, 27 April 2005
Line 100:		Line 100:
	\|-		\|-
	\|U-00010000... U-001FFFFF		\|U-00010000... U-001FFFFF
	\|\|<p style="font-family: 'fontname'>00000000 000xxxyy yyyyzzzz zzuuuuuu</p>		\|\|<nowiki></nowiki><p style="font-family: 'fontname'>00000000 000xxxyy yyyyzzzz zzuuuuuu</p>
	\|\|11110xxx		\|\|11110xxx
	\|\|10yyyyyy		\|\|10yyyyyy