utf 2.5 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100
  1. .TH UTF 6
  2. .SH NAME
  3. UTF, Unicode, ASCII, rune \- character set and format
  4. .SH DESCRIPTION
  5. The Plan 9 character set and representation are
  6. based on the Unicode Standard and on the ISO multibyte
  7. .SM UTF-8
  8. encoding (Universal Character
  9. Set Transformation Format, 8 bits wide).
  10. The Unicode Standard represents its characters in 21
  11. bits;
  12. .SM UTF-8
  13. represents such
  14. values in an 8-bit byte stream.
  15. Throughout this manual,
  16. .SM UTF-8
  17. is shortened to
  18. .SM UTF.
  19. .PP
  20. In Plan 9, a
  21. .I rune
  22. is a 21-bit quantity representing a Unicode character.
  23. Internally, programs may store characters as runes.
  24. However, any external manifestation of textual information,
  25. in files or at the interface between programs, uses a
  26. machine-independent, byte-stream encoding called
  27. .SM UTF.
  28. .PP
  29. .SM UTF
  30. is designed so the 7-bit
  31. .SM ASCII
  32. set (values hexadecimal 00 to 7F),
  33. appear only as themselves
  34. in the encoding.
  35. Runes with values above 7F appear as sequences of two or more
  36. bytes with values only from 80 to FF.
  37. .PP
  38. The
  39. .SM UTF
  40. encoding of the Unicode Standard is backward compatible with
  41. .SM ASCII\c
  42. :
  43. programs presented only with
  44. .SM ASCII
  45. work on Plan 9
  46. even if not written to deal with
  47. .SM UTF,
  48. as do
  49. programs that deal with uninterpreted byte streams.
  50. However, programs that perform semantic processing on
  51. .SM ASCII
  52. graphic
  53. characters must convert from
  54. .SM UTF
  55. to runes
  56. in order to work properly with non-\c
  57. .SM ASCII
  58. input.
  59. See
  60. .IR rune (2).
  61. .PP
  62. Letting numbers be binary,
  63. a rune x is converted to a multibyte
  64. .SM UTF
  65. sequence
  66. as follows:
  67. .PP
  68. 01. x in [000000.00000000.0bbbbbbb] → 0bbbbbbb
  69. .br
  70. 10. x in [000000.00000bbb.bbbbbbbb] → 110bbbbb, 10bbbbbb
  71. .br
  72. 11. x in [000000.bbbbbbbb.bbbbbbbb] → 1110bbbb, 10bbbbbb, 10bbbbbb
  73. .br
  74. 100. x in [bbbbbb.bbbbbbbb.bbbbbbbb] → 1110bbbb, 10bbbbbb, 10bbbbbb, 10bbbbbb
  75. .br
  76. .PP
  77. Conversion 01 provides a one-byte sequence that spans the
  78. .SM ASCII
  79. character set in a compatible way.
  80. Conversions 10, 11 and 100 represent higher-valued characters
  81. as sequences of two, three or four bytes with the high bit set.
  82. Plan 9 does not support the 5 and 6 byte sequences proposed by X-Open.
  83. When there are multiple ways to encode a value, for example rune 0,
  84. the shortest encoding is used.
  85. .PP
  86. In the inverse mapping,
  87. any sequence except those described above
  88. is incorrect and is converted to rune hexadecimal FFFD.
  89. .SH FILES
  90. .TF "/lib/unicode "
  91. .TP
  92. .B /lib/unicode
  93. table of characters and descriptions, suitable for
  94. .IR look (1).
  95. .SH "SEE ALSO"
  96. .IR ascii (1),
  97. .IR tcs (1),
  98. .IR rune (2),
  99. .IR keyboard (6),
  100. .IR "The Unicode Standard" .