1
0

utf.ms 41 KB

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073107410751076107710781079108010811082108310841085108610871088108910901091109210931094109510961097109810991100110111021103110411051106110711081109111011111112111311141115111611171118111911201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211121212131214121512161217121812191220122112221223122412251226122712281229123012311232123312341235123612371238123912401241124212431244124512461247124812491250125112521253125412551256125712581259
  1. .HTML "Hello World or Καλημέρα κόσμε or こんにちは 世界
  2. .TL
  3. Hello World
  4. .br
  5. or
  6. .br
  7. .ft R
  8. Καλημέρα κόσμε
  9. .ft
  10. .br
  11. or
  12. .br
  13. \f(Jpこんにちは 世界\fP
  14. .AU
  15. Rob Pike
  16. Ken Thompson
  17. .sp
  18. rob,ken@plan9.bell-labs.com
  19. .AB
  20. .FS
  21. Originally appeared, in a slightly different form, in
  22. .I
  23. Proc. of the Winter 1993 USENIX Conf.,
  24. .R
  25. pp. 43-50,
  26. San Diego.
  27. It has been revised to reflect the move to 21-bit Unicode.
  28. .FE
  29. Plan 9 from Bell Labs has recently been converted from ASCII
  30. to an ASCII-compatible variant of the Unicode Standard,
  31. a 16-bit (now 21-bit) character set.
  32. In this paper we explain the reasons for the change,
  33. describe the character set and representation we chose,
  34. and present the programming models and software changes
  35. that support the new text format.
  36. Although we stopped short of full internationalization\(emfor
  37. example, system error messages are in Unixese, not Japanese\(emwe
  38. believe Plan 9 is the first system to treat the representation
  39. of all major languages on a uniform, equal footing throughout all its
  40. software.
  41. .AE
  42. .SH
  43. Introduction
  44. .PP
  45. The world is multilingual but most computer systems
  46. are based on English and ASCII.
  47. The first release of Plan 9 [Pike90], a new distributed operating
  48. system from Bell Laboratories, seemed a good occasion
  49. to correct this chauvinism.
  50. It is easier to make such deep changes when building new systems than
  51. by refitting old ones.
  52. .PP
  53. The ANSI C standard [ANSIC] contains some guidance on the matter of
  54. `wide' and `multi-byte' characters but falls far short of
  55. solving the myriad associated problems.
  56. We could find no literature on how to convert a
  57. .I system
  58. to larger character sets, although some individual
  59. .I programs
  60. had been converted.
  61. This paper reports what we discovered as we
  62. explored the problem of representing multilingual
  63. text at all levels of an operating system,
  64. from the file system and kernel through
  65. the applications and up to the window system
  66. and display.
  67. .PP
  68. Plan 9 has not been `internationalized':
  69. its manuals are in English,
  70. its error messages are in English,
  71. and it can display text that goes from left to right only.
  72. But before we can address these other problems,
  73. we need to handle, uniformly and comfortably,
  74. the textual representation of all the major written languages.
  75. That subproblem is richer than we had anticipated.
  76. .SH
  77. Standards
  78. .PP
  79. Our first step was to select a standard.
  80. At the time (January 1992),
  81. there were only two viable options:
  82. ISO 10646 [ISO10646] and Unicode [Unicode].
  83. The documents describing both proposals were still in the draft stage.
  84. .PP
  85. The draft of ISO 10646 was not
  86. very attractive to us.
  87. It defined a sparse set of 32-bit characters,
  88. which would be
  89. hard to implement
  90. and have punitive storage requirements.
  91. Also, the draft attempted to
  92. mollify national interests by allocating
  93. 16-bit subspaces to national committees
  94. to partition individually.
  95. The suggested mode of use was to
  96. ``flip'' between separate national
  97. standards to implement the international standard.
  98. This did not strike us as a sound basis for a character set.
  99. As well, transmitting 32-bit values in a byte stream,
  100. such as in pipes, would be expensive and hard to implement.
  101. Since the standard does not define a byte order for such
  102. transmission, the byte stream would also have to carry
  103. state to enable the values to be recovered.
  104. .PP
  105. The Unicode Standard is a proposal by a consortium of mostly American
  106. computer companies formed
  107. to protest the technical
  108. failings of ISO 10646.
  109. It defines a uniform 16-bit code based on the
  110. principle of unification:
  111. two characters are the same if they look the
  112. same even though they are from different
  113. languages.
  114. This principle, called Han unification,
  115. allows the large Japanese, Chinese, and Korean
  116. character sets to be packed comfortably into a 16-bit representation.
  117. .PP
  118. We chose the Unicode Standard for its technical merits and because its
  119. code space was better defined.
  120. Moreover,
  121. the Unicode Consortium was derailing the
  122. ISO 10646 standard.
  123. (Now, in 1995,
  124. ISO 10646 is a standard
  125. with one 16-bit group defined,
  126. which is almost exactly the Unicode Standard.
  127. As most people expected, the two standards bodies
  128. reached a détente and
  129. ISO 10646 and Unicode represent the same character set.)
  130. .PP
  131. The Unicode Standard defines an adequate character set
  132. but an unreasonable representation.
  133. It states that all characters
  134. are 16 bits wide and are communicated and stored in
  135. 16-bit units.
  136. It also reserves a pair of characters
  137. (hexadecimal FFFE and FEFF) to detect byte order
  138. in transmitted text, requiring state in the byte stream.
  139. (The Unicode Consortium was thinking of files, not pipes.)
  140. To adopt this encoding,
  141. we would have had to convert all text going
  142. into and out of Plan 9 between ASCII and Unicode, which cannot be done.
  143. Within a single program, in command of all its input and output,
  144. it is possible to define characters as 16-bit quantities;
  145. in the context of a networked system with
  146. hundreds of applications on diverse machines
  147. by different manufacturers,
  148. it is impossible.
  149. .PP
  150. We needed a way to adapt the Unicode Standard to the tools-and-pipes
  151. model of text processing embodied by the Unix system.
  152. To do that, we
  153. needed an ASCII-compatible textual
  154. representation of Unicode characters for transmission
  155. and storage.
  156. In the draft ISO standard there was an informative
  157. (non-required)
  158. Annex
  159. called UTF
  160. that provided a byte stream encoding
  161. of the 32-bit ISO code.
  162. The encoding uses multibyte sequences composed
  163. from the 190 printable characters of Latin-1
  164. to represent character values larger
  165. than 159.
  166. .PP
  167. The UTF encoding has several good properties.
  168. By far the most important is that
  169. a byte in the ASCII range 0-127 represents
  170. itself in UTF.
  171. Thus UTF is backward compatible with ASCII.
  172. .PP
  173. UTF has other advantages.
  174. It is a byte encoding and is
  175. therefore byte-order independent.
  176. ASCII control characters appear in the byte stream
  177. only as themselves, never as an element of a sequence
  178. encoding another character,
  179. so newline bytes separate lines of UTF text.
  180. Finally, ANSI C's
  181. .CW strcmp
  182. function applied to UTF strings preserves the ordering of Unicode characters.
  183. .PP
  184. To encode and decode UTF is expensive (involving multiplication,
  185. division, and modulo operations) but workable.
  186. UTF's major disadvantage is that the encoding
  187. is not self-synchronizing.
  188. It is in general impossible to find the character
  189. boundaries in a UTF string without reading from
  190. the beginning of the string, although in practice
  191. control characters such as newlines,
  192. tabs, and blanks provide synchronization points.
  193. .PP
  194. In August 1992,
  195. X-Open circulated a proposal for another UTF-like
  196. byte encoding of Unicode characters.
  197. Their major concern was that an embedded character
  198. in a file name
  199. (in particular a slash)
  200. could be part of an escape sequence in UTF and
  201. therefore confuse a traditional file system.
  202. Their proposal would allow all 7-bit ASCII characters
  203. to represent themselves
  204. .I "and only themselves"
  205. in text.
  206. Multibyte sequences would contain only characters
  207. with the high bit set.
  208. We proposed a modification to the new UTF that
  209. would address our synchronization problem.
  210. Our proposal, which was originally known informally as UTF-2 and FSS-UTF,
  211. is now referred to as UTF-8 and has been approved by ISO to become
  212. Annex P to ISO 10646.
  213. .PP
  214. The model for text in Plan 9 is chosen from these
  215. three standards*:
  216. .FS
  217. * ``That's the nice thing about standards\(emthere's so many to choose from.'' \- Andy Tannenbaum (no, the other one)
  218. .FE
  219. the Unicode character set encoded as a byte stream by
  220. UTF-8, from
  221. (soon to be) Annex P of ISO 10646.
  222. Although this mixture may seem like a precarious position for us to adopt,
  223. it is not as bad as it sounds.
  224. ISO 10646 and the Unicode Standard have converged,
  225. other systems such as Linux have adopted the same character set and encoding,
  226. and the general feeling seems to be that Unicode and UTF-8 will be accepted as the way
  227. to exchange text between systems.
  228. The prognosis for wide acceptance is good.
  229. .PP
  230. There are a couple of aspects of the Unicode Standard we have not faced.
  231. One is the issue of right-to-left text such as Hebrew or Arabic.
  232. Since that is an issue of display, not representation, we believe
  233. we can defer that problem for the moment without affecting our
  234. ability to solve it later.
  235. Another issue is diacriticals and `combining characters',
  236. which cause overstriking of multiple Unicode characters.
  237. Although necessary for some scripts, such as Thai, Arabic, and Hebrew,
  238. such characters confuse the issues for Latin languages because they
  239. generate multiple representations for accented characters.
  240. ISO 10646 describes three levels of implementation;
  241. in Plan 9 we decided not to address the issue.
  242. Again, this can be labeled as a display issue and its finer points are still being debated,
  243. so we felt comfortable deferring. Mañana.
  244. .PP
  245. Although we converted Plan 9 in the altruistic interests of
  246. serving foreign languages, we have found the large character
  247. set attractive for other reasons. The Unicode Standard includes many
  248. characters\(emmathematical symbols, scientific notation,
  249. more general punctuation, and more\(emthat we now use
  250. daily in our work. We no longer test our imaginations
  251. to find ways to include non-ASCII symbols in our text;
  252. why type
  253. .CW :-)
  254. when you can use the character ☺?
  255. Most compelling is the ability to absorb documents
  256. and data that contain non-ASCII characters; our browser for the
  257. Oxford English Dictionary
  258. lets us see the dictionary as it really is, with pronunciation
  259. in the IPA font, foreign phrases properly rendered, and so on,
  260. .I "in plain text.
  261. .PP
  262. As of Unicode 4.0,
  263. characters are now 21 bits wide and the longest UTF-8 encoding of a character
  264. requires 4 bytes.
  265. We are adapting the system to match.
  266. .PP
  267. In the rest of this paper, except when
  268. stated otherwise, the term `UTF' refers to the UTF-8 encoding
  269. of Unicode characters as adopted by Plan 9.
  270. .SH
  271. C Compiler
  272. .PP
  273. The first program to be converted to UTF
  274. was the C Compiler.
  275. There are two levels of conversion.
  276. On the syntactic level,
  277. input to the C compiler
  278. is UTF; on the semantic level,
  279. the C language needs to define
  280. how compiled programs manipulate
  281. the UTF set.
  282. .PP
  283. The syntactic part is simple.
  284. The ANSI C language standard defines the
  285. source character set to be ASCII.
  286. Since UTF is backward compatible with ASCII,
  287. the compiler needs little change.
  288. The only places where a larger character set
  289. is allowed are in character constants, strings, and comments.
  290. Since 7-bit ASCII characters can represent only
  291. themselves in UTF,
  292. the compiler does not have to be careful while looking
  293. for the termination of a string or comment.
  294. .PP
  295. The Plan 9 compiler extends ANSI C to treat any Unicode
  296. character with a value outside of the ASCII range as
  297. an alphabetic.
  298. To a Greek programmer or an English mathematician,
  299. α is a sensible and now valid variable name.
  300. .PP
  301. On the semantic level, ANSI C allows,
  302. but does not tie down,
  303. the notion of a
  304. .I "wide character
  305. and admits string and character constants
  306. of this type.
  307. We chose the wide character type to be
  308. .CW unsigned
  309. .CW short
  310. (now
  311. .CW unsigned
  312. .CW long) .
  313. In the libraries, the word
  314. .CW Rune
  315. is now defined by a
  316. .CW typedef
  317. to be equivalent to
  318. .CW unsigned
  319. .CW long
  320. and is
  321. used to signify a Unicode character.
  322. .PP
  323. There are surprises; for example:
  324. .P1
  325. L'x' \f1is 120\fP
  326. \&'x' \f1is 120\fP
  327. L'ÿ' \f1is 255\fP
  328. \&'ÿ' \f1is -1, stdio \fPEOF\f1 (if \fPchar\f1 is signed)\fP
  329. L'\f1α\fP' \f1is 945\fP
  330. \&'\f1α\fP' \f1is illegal\fP
  331. .P2
  332. In the string constants,
  333. .P1
  334. "\f(Jpこんにちは 世界\fP"
  335. L"\f(Jpこんにちは 世界\fP",
  336. .P2
  337. the former is an array of
  338. .CW chars
  339. with 22 elements
  340. and a null byte,
  341. while the latter is an array of
  342. .CW unsigned
  343. .CW long s
  344. .CW Runes ) (
  345. with 8 elements and a null
  346. .CW Rune .
  347. .PP
  348. The Plan 9 library provides an output conversion function,
  349. .CW print
  350. (analogous to
  351. .CW printf ),
  352. with formats
  353. .CW %c ,
  354. .CW %C ,
  355. .CW %s ,
  356. and
  357. .CW %S .
  358. Since
  359. .CW print
  360. produces text, its output is always UTF.
  361. The character conversion
  362. .CW %c
  363. (lower case) masks its argument
  364. to 8 bits before converting to UTF.
  365. Thus
  366. .CW L'ÿ'
  367. and
  368. .CW 'ÿ'
  369. printed under
  370. .CW %c
  371. will be identical,
  372. but
  373. .CW L'\f1α\fP'
  374. will print as the Unicode
  375. character with decimal value 177.
  376. The character conversion
  377. .CW %C
  378. (upper case) masks its argument
  379. to 16 bits before converting to UTF.
  380. Thus
  381. .CW L'ÿ'
  382. and
  383. .CW L'\f1α\fP'
  384. will print correctly under
  385. .CW %C ,
  386. but
  387. .CW 'ÿ'
  388. will not.
  389. The conversion
  390. .CW %s
  391. (lower case)
  392. expects a pointer to
  393. .CW char
  394. and copies UTF sequences up to a null byte.
  395. The conversion
  396. .CW %S
  397. (upper case) expects a pointer to
  398. .CW Rune
  399. and
  400. performs sequential
  401. .CW %C
  402. conversions until a null
  403. .CW Rune
  404. is encountered.
  405. .PP
  406. Another problem in format conversion
  407. is the definition of
  408. .CW %10s :
  409. does the number refer to bytes or characters?
  410. We decided that such formats were most
  411. often used to align output columns and
  412. so made the number count characters.
  413. Some programs, however, use the count
  414. to place blank-padded strings
  415. in fixed-sized arrays.
  416. These programs must be found and corrected.
  417. .PP
  418. Here is a complete example:
  419. .P1
  420. #include <u.h>
  421. char c[] = "\f(Jpこんにちは 世界\fP";
  422. Rune s[] = L"\f(Jpこんにちは 世界\fP";
  423. main(void)
  424. {
  425. print("%d, %d\en", sizeof(c), sizeof(s));
  426. print("%s\en", c);
  427. print("%S\en", s);
  428. }
  429. .P2
  430. .PP
  431. This program prints
  432. .CW 23,
  433. .CW 18
  434. and then two identical lines of
  435. UTF text.
  436. In practice,
  437. .CW %S
  438. and
  439. .CW L"..."
  440. are rare in programs; one reason is
  441. that most formatted I/O is done in unconverted UTF.
  442. .SH
  443. Ramifications
  444. .PP
  445. All programs in Plan 9 now read and write text as UTF, not ASCII.
  446. This change breaks two deep-rooted symmetries implicit in most C programs:
  447. .IP 1.
  448. A character is no longer a
  449. .CW char .
  450. .IP 2.
  451. The internal representation (Rune) of a character now differs from its
  452. external representation (UTF).
  453. .PP
  454. In the sections that follow,
  455. we show how these issues were faced in the layers of
  456. system software from the operating system up to the applications.
  457. The effects are wide-reaching and often surprising.
  458. .SH
  459. Operating system
  460. .PP
  461. Since UTF is the only format for text in Plan 9,
  462. the interface to the operating system had to be converted to UTF.
  463. Text strings cross the interface in several places:
  464. command arguments,
  465. file names,
  466. user names (people can log in using their native name),
  467. error messages,
  468. and miscellaneous minor places such as commands to the I/O system.
  469. Little change was required: null-terminated UTF strings
  470. are equivalent to null-terminated ASCII strings for most purposes
  471. of the operating system.
  472. The library routines described in the next section made that
  473. change straightforward.
  474. .PP
  475. The window system, once called
  476. .CW 8.5 ,
  477. is now rightfully called
  478. .CW 8½ .
  479. .SH
  480. Libraries
  481. .PP
  482. A header file included by all programs (see [Pike92]) declares
  483. the
  484. .CW Rune
  485. type to hold 21-bit character values:
  486. .P1
  487. typedef unsigned long Rune;
  488. .P2
  489. Also defined are several constants relevant to UTF:
  490. .P1
  491. enum
  492. {
  493. UTFmax = 4, /* maximum bytes per rune */
  494. Runesync = 0x80, /* cannot be in a UTF sequence (<) */
  495. Runeself = 0x80, /* rune==UTF sequence (<) */
  496. Runeerror = 0xFFFD, /* decoding error in UTF */
  497. Runemax = 0x10FFFF, /* largest 21-bit rune */
  498. Runemask = 0x1FFFFF, /* bits used by runes (see grep) */
  499. };
  500. .P2
  501. (With the original UTF,
  502. .CW Runesync
  503. was hexadecimal 21 and
  504. .CW Runeself
  505. was A0.)
  506. .CW UTFmax
  507. bytes are sufficient
  508. to hold the UTF encoding of any Unicode character.
  509. Characters of value less than
  510. .CW Runesync
  511. only appear in a UTF string as
  512. themselves, never as part of a sequence encoding another character.
  513. Characters of value less than
  514. .CW Runeself
  515. encode into single bytes
  516. of the same value.
  517. Finally, when the library detects errors in UTF input\(embyte sequences
  518. that are not valid UTF sequences\(emit converts the first byte of the
  519. error sequence to the character
  520. .CW Runeerror .
  521. There is little a rune-oriented program can do when given bad data
  522. except exit, which is unreasonable, or carry on.
  523. Originally the conversion routines, described below,
  524. returned errors when given invalid UTF,
  525. but we found ourselves repeatedly checking for errors and ignoring them.
  526. We therefore decided to convert a bad sequence to a valid rune
  527. and continue processing.
  528. (The ANSI C routines, on the other hand, return errors.)
  529. .PP
  530. This technique does have the unfortunate property that converting
  531. invalid UTF byte strings in and out of runes does not preserve the input,
  532. but this circumstance only occurs when non-textual input is
  533. given to a textual program.
  534. The Unicode Standard defines an error character, value FFFD, to stand for
  535. characters from other sets that it does not represent.
  536. The
  537. .CW Runeerror
  538. character is a different concept, related to the encoding rather than the character set.
  539. .PP
  540. The Plan 9 C library contains a number of routines for
  541. manipulating runes.
  542. The first set converts between runes and UTF strings:
  543. .P1
  544. extern int runetochar(char*, Rune*);
  545. extern int chartorune(Rune*, char*);
  546. extern int runelen(long);
  547. extern int fullrune(char*, int);
  548. .P2
  549. .CW Runetochar
  550. translates a single
  551. .CW Rune
  552. to a UTF sequence and returns the number of bytes produced.
  553. .CW Chartorune
  554. goes the other way, reporting how many bytes were consumed.
  555. .CW Runelen
  556. returns the number of bytes in the UTF encoding of a rune.
  557. .CW Fullrune
  558. examines a UTF string up to a specified number of bytes
  559. and reports whether the string begins with a complete UTF encoding.
  560. All these routines use the
  561. .CW Runeerror
  562. character to work around encoding problems.
  563. .PP
  564. There is also a set of routines for examining null-terminated UTF strings,
  565. based on the model of the ANSI standard
  566. .CW str
  567. routines, but with
  568. .CW utf
  569. substituted for
  570. .CW str
  571. and
  572. .CW rune
  573. for
  574. .CW chr :
  575. .P1
  576. extern int utflen(char*);
  577. extern char* utfrune(char*, long);
  578. extern char* utfrrune(char*, long);
  579. extern char* utfutf(char*, char*);
  580. .P2
  581. .CW Utflen
  582. returns the number of runes in a UTF string;
  583. .CW utfrune
  584. returns a pointer to the first occurrence of a rune in a UTF string;
  585. and
  586. .CW utfrrune
  587. a pointer to the last.
  588. .CW Utfutf
  589. searches for the first occurrence of a UTF string in another UTF string.
  590. Given the synchronizing property of UTF-8,
  591. .CW utfutf
  592. is the same as
  593. .CW strstr
  594. if the arguments point to valid UTF strings.
  595. .PP
  596. It is a mistake to use
  597. .CW strchr
  598. or
  599. .CW strrchr
  600. unless searching for a 7-bit ASCII character, that is, a character
  601. less than
  602. .CW Runeself .
  603. .PP
  604. We have no routines for manipulating null-terminated arrays of
  605. .CW Runes .
  606. Although they should probably exist for completeness, we have
  607. found no need for them, for the same reason that
  608. .CW %S
  609. and
  610. .CW L"..."
  611. are rarely used.
  612. .PP
  613. Most Plan 9 programs use a new buffered I/O library, BIO, in place of
  614. Standard I/O.
  615. BIO contains routines to read and write UTF streams, converting to and from
  616. runes.
  617. .CW Bgetrune
  618. returns, as a
  619. .CW Rune
  620. within a
  621. .CW long ,
  622. the next character in the UTF input stream;
  623. .CW Bputrune
  624. takes a rune and writes its UTF representation.
  625. .CW Bungetrune
  626. puts a rune back into the input stream for rereading.
  627. .PP
  628. Plan 9 programs use a simple set of macros to process command line arguments.
  629. Converting these macros to UTF automatically updated the
  630. argument processing of most programs.
  631. In general,
  632. argument flag names can no longer be held in bytes and
  633. arrays of 256 bytes cannot be used to hold a set of flags.
  634. .PP
  635. We have done nothing analogous to ANSI C's locales, partly because
  636. we do not feel qualified to define locales and partly because we remain
  637. unconvinced of that model for dealing with the problems.
  638. That is really more an issue of internationalization than conversion
  639. to a larger character set; on the other hand,
  640. because we have chosen a single character set that encompasses
  641. most languages, some of the need for
  642. locales is eliminated.
  643. (We have a utility,
  644. .CW tcs ,
  645. that translates between UTF and other character sets.)
  646. .PP
  647. There are several reasons why our library does not follow the ANSI design
  648. for wide and multi-byte characters.
  649. The ANSI model was designed by a committee, untried, almost
  650. as an afterthought, whereas
  651. we wanted to design as we built.
  652. (We made several major changes to the interface
  653. as we became familiar with the problems involved.)
  654. We disagree with ANSI C's handling of invalid multi-byte sequences.
  655. Also, the ANSI C library is incomplete:
  656. although it contains some crucial routines for handling
  657. wide and multi-byte characters, there are some serious omissions.
  658. For example, our software can exploit
  659. the fact that UTF preserves ASCII characters in the byte stream.
  660. We could remove that assumption by replacing all
  661. calls to
  662. .CW strchr
  663. with
  664. .CW utfrune
  665. and so on.
  666. (Because of the weaker properties of the original UTF,
  667. we have actually done so.)
  668. ANSI C cannot:
  669. the standard says nothing about the representation, so portable code should
  670. .I never
  671. call
  672. .CW strchr ,
  673. yet there is no ANSI equivalent to
  674. .CW utfrune .
  675. ANSI C simultaneously invalidates
  676. .CW strchr
  677. and offers no replacement.
  678. .PP
  679. Finally, ANSI did nothing to integrate wide characters
  680. into the I/O system: it gives no method for printing
  681. wide characters.
  682. We therefore needed to invent some things and decided to invent
  683. everything.
  684. In the end, some of our entry points do correspond closely to
  685. ANSI routines\(emfor example
  686. .CW chartorune
  687. and
  688. .CW runetochar
  689. are similar to
  690. .CW mbtowc
  691. and
  692. .CW wctomb \(embut
  693. Plan 9's library defines more functionality, enough
  694. to write real applications comfortably.
  695. .SH
  696. Converting the tools
  697. .PP
  698. The source for our tools and applications had already been converted to
  699. work with Latin-1, so it was `8-bit safe', but the conversion to the Unicode
  700. Standard and UTF is more involved.
  701. Some programs needed no change at all:
  702. .CW cat ,
  703. for instance,
  704. interprets its argument strings, delivered in UTF,
  705. as file names that it passes uninterpreted to the
  706. .CW open
  707. system call,
  708. and then just copies bytes from its input to its output;
  709. it never makes decisions based on the values of the bytes.
  710. (Plan 9
  711. .CW cat
  712. has no options such as
  713. .CW -v
  714. to complicate matters.)
  715. Most programs, however, needed modest change.
  716. .PP
  717. It is difficult to
  718. find automatically the places that need attention,
  719. but
  720. .CW grep
  721. helps.
  722. Software that uses the libraries conscientiously can be searched
  723. for calls to library routines that examine bytes as characters:
  724. .CW strchr ,
  725. .CW strrchr ,
  726. .CW strstr ,
  727. etc.
  728. Replacing these by calls to
  729. .CW utfrune ,
  730. .CW utfrrune ,
  731. and
  732. .CW utfutf
  733. is enough to fix many programs.
  734. Few tools actually need to operate on runes internally;
  735. more typically they need only to look for the final slash in a file
  736. name and similar trivial tasks.
  737. Of the 170 C source programs in the top levels of
  738. .CW /sys/src/cmd ,
  739. only 23 now contain the word
  740. .CW Rune .
  741. .PP
  742. The programs that
  743. .I do
  744. store runes internally
  745. are mostly those whose
  746. .I raison
  747. .I d'être
  748. is character manipulation:
  749. .CW sam
  750. (the text editor),
  751. .CW sed ,
  752. .CW sort ,
  753. .CW tr ,
  754. .CW troff ,
  755. .CW 8½
  756. (the window system and terminal emulator),
  757. and so on.
  758. To decide whether to compute using runes
  759. or UTF-encoded byte strings requires balancing the cost of converting
  760. the data when read and written
  761. against the cost of converting relevant text on demand.
  762. For programs such as editors that run a long time with a relatively
  763. constant dataset, runes are the better choice.
  764. There are space considerations too, but they are more complicated:
  765. plain ASCII text grows when converted to runes; UTF-encoded Japanese
  766. shrinks.
  767. .PP
  768. Again, it is hard to automate the conversion of a program from
  769. .CW chars
  770. to
  771. .CW Runes .
  772. It is not enough just to change the type of variables; the assumption
  773. that bytes and characters are equivalent can be insidious.
  774. For instance, to clear a character array by
  775. .P1
  776. memset(buf, 0, BUFSIZE)
  777. .P2
  778. becomes wrong if
  779. .CW buf
  780. is changed from an array of
  781. .CW chars
  782. to an array of
  783. .CW Runes .
  784. Any program that indexes tables based on character values needs
  785. rethinking.
  786. Consider
  787. .CW tr ,
  788. which originally used multiple 256-byte arrays for the mapping.
  789. The naïve conversion would yield multiple 1,114,112-rune arrays.
  790. Instead Plan 9
  791. .CW tr
  792. saves space by building in effect
  793. a run-encoded version of the map.
  794. .PP
  795. .CW Sort
  796. has related problems.
  797. The cooperation of UTF and
  798. .CW strcmp
  799. means that a simple sort\(emone with no options\(emcan be done
  800. on the original UTF strings using
  801. .CW strcmp .
  802. With sorting options enabled, however,
  803. .CW sort
  804. may need to convert its input to runes: for example,
  805. option
  806. .CW -t\f1α\fP
  807. requires searching for alphas in the input text to
  808. crack the input into fields.
  809. The field specifier
  810. .CW +3.2
  811. refers to 2 runes beyond the third field.
  812. Some of the other options are hopelessly provincial:
  813. consider the case-folding and dictionary order options
  814. (Japanese doesn't even have an official dictionary order) or
  815. .CW -M
  816. which compares by case-insensitive English month name.
  817. Handling these options involves the
  818. larger issues of internationalization and is beyond the scope
  819. of this paper and our expertise.
  820. Plan 9
  821. .CW sort
  822. works sensibly with options that make sense relative to the input.
  823. The simple and most important options are, however, usually meaningful.
  824. In particular,
  825. .CW sort
  826. sorts UTF into the same order that
  827. .CW look
  828. expects.
  829. .PP
  830. Regular expression-matching algorithms need rethinking to
  831. be applied to UTF text.
  832. Deterministic automata are usually applied to bytes;
  833. converting them to operate on variable-sized byte sequences is awkward.
  834. On the other hand, converting the input stream to runes adds measurable
  835. expense
  836. and the state tables expand
  837. from size 256 to 1,114,112; it can be expensive just to generate them.
  838. For simple string searching,
  839. the Boyer-Moore algorithm works with UTF provided the input is
  840. guaranteed to be only valid UTF strings; however, it does not work
  841. with the old UTF encoding.
  842. At a more mundane level, even character classes are harder:
  843. the usual bit-vector representation within a non-deterministic automaton
  844. is unwieldy with 1,114,112 characters in the alphabet.
  845. .PP
  846. We compromised.
  847. An existing library for compiling and executing regular expressions
  848. was adapted to work on runes, with two entry points for searching
  849. in arrays of runes and arrays of chars (the pattern is always UTF text).
  850. Character classes are represented internally as runs of runes;
  851. the reserved value
  852. .CW FFFF
  853. marks the end of the class.
  854. Then
  855. .I all
  856. utilities that use regular expressions\(emeditors,
  857. .CW grep ,
  858. .CW awk ,
  859. etc.\(emexcept the shell, whose notation
  860. was grandfathered, were converted to use the library.
  861. For some programs, there was a concomitant loss of performance,
  862. but there was also a strong advantage.
  863. To our knowledge, Plan 9 is the only Unix-like system
  864. that has a single definition and implementation of
  865. regular expressions; patterns are written and interpreted
  866. identically by all the programs in the system.
  867. .PP
  868. A handful of programs have the notion of character built into them
  869. so strongly as to confuse the issue of what they should do with UTF input.
  870. Such programs were treated as individual special cases.
  871. For example,
  872. .CW wc
  873. is, by default, unchanged in behavior and output; a new option,
  874. .CW -r ,
  875. counts the number of correctly encoded runes\(emvalid UTF sequences\(emin
  876. its input;
  877. .CW -b
  878. the number of invalid sequences.
  879. .PP
  880. It took us several months to convert all the software in the system
  881. to the Unicode Standard and the old UTF.
  882. When we decided to convert from that to the new UTF,
  883. only three things needed to be done.
  884. First, we rewrote the library routines to encode and decode the
  885. new UTF. This took an evening.
  886. Next, we converted all the files containing UTF
  887. to the new encoding.
  888. We wrote a trivial program to look for non-ASCII bytes in
  889. text files and used a Plan 9 program called
  890. .CW tcs
  891. (translate character set) to change encodings.
  892. Finally, we recompiled all the system software;
  893. the library interface was unchanged, so recompilation was sufficient
  894. to effect the transformation.
  895. The second two steps were done concurrently and took an afternoon.
  896. We concluded that the actual encoding is relatively unimportant to the
  897. software; the adoption of large characters and a byte-stream encoding
  898. .I per
  899. .I se
  900. are much deeper issues.
  901. .SH
  902. Graphics and fonts
  903. .PP
  904. Plan 9 provides only minimal support for plain text terminals.
  905. It is instead designed to be used with all character input and
  906. output mediated by a window system such as
  907. .CW 8½ .
  908. The window system and related software are responsible for the
  909. display of UTF text as Unicode character images.
  910. For plain text, the window system must provide a user-settable
  911. .I font
  912. that provides a (possibly empty) picture for each Unicode character.
  913. Fancier applications that use bold and Italic characters
  914. need multiple fonts storing multiple pictures for each
  915. Unicode value.
  916. All the issues are apparent, though,
  917. in just the problem of
  918. displaying a single image for each character, that is, the
  919. Unicode equivalent of a plain text terminal.
  920. With 128 or even 256 characters, a font can be just
  921. an array of bitmaps. With 1,114,112 characters,
  922. a more sophisticated design is necessary. To store the ideographs
  923. for just Japanese as 16×16×1 bit images,
  924. the smallest they can reasonably be, takes over a quarter of a
  925. megabyte. Make the images a little larger, store more bits per
  926. pixel, and hold a copy in every running application, and the
  927. memory cost becomes unreasonable.
  928. .PP
  929. The structure of the bitmap graphics services is described at length elsewhere
  930. [Pike91].
  931. In summary, the memory holding the bitmaps is stored in the same machine that has
  932. the display, mouse, and keyboard: the terminal in Plan 9 terminology,
  933. the workstation in others'.
  934. Access to that memory and associated services is provided
  935. by device files served by system
  936. software on the terminal. One of those files,
  937. .CW /dev/bitblt ,
  938. interprets messages written upon it as requests for actions
  939. corresponding to entry points in the graphics library:
  940. allocate a bitmap, execute a raster operation, draw a text string, etc.
  941. The window system
  942. acts as a multiplexer that mediates access to the services
  943. and resources of the terminal by simulating in each client window
  944. a set of files mirroring those provided by the system.
  945. That is, each window has a distinct
  946. .CW /dev/mouse ,
  947. .CW /dev/bitblt ,
  948. and so on through which applications drive graphical
  949. input and output.
  950. .PP
  951. One of the resources managed by
  952. .CW 8½
  953. and the terminal is the set of active
  954. .I subfonts.
  955. Each subfont holds the
  956. bitmaps and associated data structures for a sequential set of Unicode
  957. characters.
  958. Subfonts are stored in files and loaded into the terminal by
  959. .CW 8½
  960. or an application.
  961. For example, one subfont
  962. might hold the images of the first 256 characters of the Unicode space,
  963. corresponding to the Latin-1 character set;
  964. another might hold the standard phonetic character set, Unicode characters
  965. with value 0250 to 02E9.
  966. These files are collected in directories corresponding to typefaces:
  967. .CW /lib/font/bit/pelm
  968. contains the Pellucida Monospace character set, with subfonts holding
  969. the Latin-1, Greek, Cyrillic and other components of the typeface.
  970. A suffix on subfont files encodes (in a subfont-specific
  971. way) the size of the images:
  972. .CW /lib/font/bit/pelm/latin1.9
  973. contains the Latin-1 Pellucida Monospace characters with lower
  974. case letters 9 pixels high;
  975. .CW /lib/font/bit/jis/jis5400.16
  976. contains 16-pixel high
  977. ideographs starting at Unicode value 5400.
  978. .PP
  979. The subfonts do not identify which portion of the Unicode space
  980. they cover. Instead, a
  981. font file, in plain text,
  982. describes how to assemble subfonts into a complete
  983. character set.
  984. The font file is presented as an argument to the window system
  985. to determine how plain text is displayed in text windows and
  986. applications.
  987. Here is the beginning of the font file
  988. .CW /lib/font/bit/pelm/jis.9.font ,
  989. which describes the layout of a font covering that portion of
  990. the Unicode Standard for which we have characters of typical
  991. display size, using Japanese characters
  992. to cover the Han space:
  993. .P1
  994. 18 14
  995. 0x0000 0x00FF latin1.9
  996. 0x0100 0x017E latineur.9
  997. 0x0250 0x02E9 ipa.9
  998. 0x0386 0x03F5 greek.9
  999. 0x0400 0x0475 cyrillic.9
  1000. 0x2000 0x2044 ../misc/genpunc.9
  1001. 0x2070 0x208E supsub.9
  1002. 0x20A0 0x20AA currency.9
  1003. 0x2100 0x2138 ../misc/letterlike.9
  1004. 0x2190 0x21EA ../misc/arrows
  1005. 0x2200 0x227F ../misc/math1
  1006. 0x2280 0x22F1 ../misc/math2
  1007. 0x2300 0x232C ../misc/tech
  1008. 0x2500 0x257F ../misc/chart
  1009. 0x2600 0x266F ../misc/ding
  1010. .P2
  1011. .P1
  1012. 0x3000 0x303f ../jis/jis3000.16
  1013. 0x30a1 0x30fe ../jis/katakana.16
  1014. 0x3041 0x309e ../jis/hiragana.16
  1015. 0x4e00 0x4fff ../jis/jis4e00.16
  1016. 0x5000 0x51ff ../jis/jis5000.16
  1017. \&...
  1018. .P2
  1019. The first two numbers set the interline spacing of the font (18
  1020. pixels) and the distance from the baseline to the top of the
  1021. line (14 pixels).
  1022. When characters are displayed, they are placed so as best
  1023. to fit within those constraints; characters
  1024. too large to fit will be truncated.
  1025. The rest of the file associates subfont files
  1026. with portions of Unicode space.
  1027. The first four such files are in the Pellucida Monospace typeface
  1028. and directory; others reside in other directories. The file names
  1029. are relative to the font file's own location.
  1030. .PP
  1031. There are several advantages to this two-level structure.
  1032. First, it simultaneously breaks the huge Unicode space into manageable
  1033. components and provides a unifying architecture for
  1034. assembling fonts from disjoint pieces.
  1035. Second, the structure promotes sharing.
  1036. For example, we have only one set of Japanese
  1037. characters but dozens of typefaces for the Latin-1 characters,
  1038. and this structure permits us to store only one copy of the
  1039. Japanese set but use it with any Roman typeface.
  1040. Also, customization is easy.
  1041. English-speaking users who don't need Japanese characters
  1042. but may want to read an on-line Oxford English Dictionary can
  1043. assemble a custom font with the
  1044. Latin-1 (or even just ASCII) characters and the International
  1045. Phonetic Alphabet (IPA).
  1046. Moreover, to do so requires just editing a plain text file,
  1047. not using a special font editing tool.
  1048. Finally, the structure guides the design of
  1049. caching protocols to improve performance and memory usage.
  1050. .PP
  1051. To load a complete Unicode character set into each application
  1052. would consume too
  1053. much memory and, particularly on slow terminal lines, would take
  1054. unreasonably long.
  1055. Instead, Plan 9 assembles a multi-level cache structure for
  1056. each font.
  1057. An application opens a font file, reads and parses it,
  1058. and allocates a data structure.
  1059. A message written to
  1060. .CW /dev/bitblt
  1061. allocates an associated structure held in the terminal, in particular,
  1062. a bitmap to act as a cache
  1063. for recently used character images.
  1064. Other messages copy these images to bitmaps such as the screen
  1065. by loading characters from subfonts into the cache on demand and
  1066. from there to the destination bitmap.
  1067. The protocol to draw characters is in terms of cache indices,
  1068. not Unicode character number or UTF sequences.
  1069. These details are hidden from the application, which instead
  1070. sees only a subroutine to draw a string in a bitmap from a
  1071. given font, functions to discover character size information,
  1072. and routines to allocate and to free fonts.
  1073. .PP
  1074. As needed, whole
  1075. subfonts are opened by the graphics library, read, and then downloaded
  1076. to the terminal.
  1077. They are held open by the library in an LRU-replacement list.
  1078. Even when the program closes a subfont, it is retained
  1079. in the terminal for later use.
  1080. When the application opens the subfont, it asks the terminal
  1081. if it already has a copy to avoid reading it from the file
  1082. server if possible.
  1083. This level of cache has the property that the bitmaps for, say,
  1084. all the Japanese characters are stored only once, in the terminal;
  1085. the applications read only size and width information from the terminal
  1086. and share the images.
  1087. .PP
  1088. The sizes of the character and subfont caches held by the
  1089. application are adaptive.
  1090. A simple algorithm monitors the cache miss rate to enlarge and
  1091. shrink the caches as required.
  1092. The size of the character cache is limited to 2048 images maximum,
  1093. which in practice seems enough even for Japanese text.
  1094. For plain ASCII-like text it naturally stays around 128 images.
  1095. .PP
  1096. This mechanism sounds complicated but is implemented by only about
  1097. 500 lines in the library and considerably less in each of the
  1098. terminal's graphics driver and
  1099. .CW 8½ .
  1100. It has the advantage that only characters that are
  1101. being used are loaded into memory.
  1102. It is also efficient: if the characters being drawn
  1103. are in the cache the extra overhead is negligible.
  1104. It works particularly well for alphabetic character sets,
  1105. but also adapts on demand for ideographic sets.
  1106. When a user first looks at Japanese text, it takes a few
  1107. seconds to read all the font data, but thereafter the
  1108. text is drawn almost as fast as regular text (the images
  1109. are larger, so draw a little slower).
  1110. Also, because the bitmaps are remembered by the terminal,
  1111. if a second application then looks at Japanese text
  1112. it starts faster than the first.
  1113. .PP
  1114. We considered
  1115. building a `font server'
  1116. to cache character images and associated data
  1117. for the applications, the window system, and the terminal.
  1118. We rejected this design because, although isolating
  1119. many of the problems of font management into a separate program,
  1120. it didn't simplify the applications.
  1121. Moreover, in a distributed system such as Plan 9 it is easy
  1122. to have too many special purpose servers.
  1123. Making the management of the fonts the concern of only
  1124. the essential components simplifies the system and makes
  1125. bootstrapping less intricate.
  1126. .SH
  1127. Input
  1128. .PP
  1129. A completely different problem is how to type Unicode characters
  1130. as input to the system.
  1131. We selected an unused key on our ASCII keyboards
  1132. to serve as a prefix for multi-keystroke
  1133. sequences that generate Unicode characters.
  1134. For example, the character
  1135. .CW ü
  1136. is generated by the prefix key
  1137. (typically
  1138. .CW ALT
  1139. or
  1140. .CW Compose )
  1141. followed by a double quote and a lower-case
  1142. .CW u .
  1143. When that character is read by the application, from the file
  1144. .CW /dev/cons ,
  1145. it is of course presented as its UTF encoding.
  1146. Such sequences generate characters from an arbitrary set that
  1147. includes all of Latin-1 plus a selection of mathematical
  1148. and technical characters.
  1149. An arbitrary Unicode character may be generated by typing the prefix,
  1150. an upper case X, and four hexadecimal digits that identify
  1151. the Unicode value.
  1152. .PP
  1153. These simple mechanisms are adequate for most of our day-to-day needs:
  1154. it's easy to remember to type `ALT 1 2' for ½\^ or `ALT accent letter'
  1155. for accented Latin letters.
  1156. For the occasional unusual character, the cut and paste features of
  1157. .CW 8½
  1158. serve well. A program called (perhaps misleadingly)
  1159. .CW unicode
  1160. takes as argument a hexadecimal value, and prints the UTF representation of that character,
  1161. which may then be picked up with the mouse and used as input.
  1162. .PP
  1163. These methods
  1164. are clearly unsatisfactory when working in a non-English language.
  1165. In the native country of such a language
  1166. the appropriate keyboard is likely to be at hand.
  1167. But it's also reasonable\(emespecially now that the system handles Unicode characters\(emto
  1168. work in a language foreign to the keyboard.
  1169. .PP
  1170. For alphabetic languages such as Greek or Russian, it is
  1171. straightforward to construct a program that does phonetic substitution,
  1172. so that, for example, typing a Latin `a' yields the Greek `α'.
  1173. Within Plan 9, such a program can be inserted transparently
  1174. between the real keyboard and a program such as the window system,
  1175. providing a manageable input device for such languages.
  1176. .PP
  1177. For ideographic languages such as Chinese or Japanese the problem is harder.
  1178. Native users of such languages have adopted methods for dealing with
  1179. Latin keyboards that involve a hybrid technique based on phonetics
  1180. to generate a list of possible symbols followed by menu selection to
  1181. choose the desired one.
  1182. Such methods can be
  1183. effective, but their design must be rooted in information about
  1184. the language unknown to non-native speakers.
  1185. .CW Cxterm , (
  1186. a Chinese terminal emulator built by and for
  1187. Chinese programmers,
  1188. employs such a technique
  1189. [Pong and Zhang].)
  1190. Although the technical problem of implementing such a device
  1191. is easy in Plan 9\(emit is just an elaboration of the technique for
  1192. alphabetic languages\(emour lack of familiarity with such languages
  1193. has restrained our enthusiasm for building one.
  1194. .PP
  1195. The input problem is technically the least interesting but perhaps
  1196. emotionally the most important of the problems of converting a system
  1197. to an international character set.
  1198. Beyond that remain the deeper problems of internationalization
  1199. such as multi-lingual error messages and command names,
  1200. problems we are not qualified to solve.
  1201. With the ability to treat text of most languages on an equal
  1202. footing, though, we can begin down that path.
  1203. Perhaps people in non-English speaking countries will
  1204. consider adopting Plan 9, solving the input problem locally\(emperhaps
  1205. just by plugging in their local terminals\(emand begin to use
  1206. a system with at least the capacity to be international.
  1207. .SH
  1208. Acknowledgements
  1209. .PP
  1210. Dennis Ritchie provided consultation and encouragement.
  1211. Bob Flandrena converted most of the standard tools to UTF.
  1212. Brian Kernighan suffered cheerfully with several
  1213. inadequate implementations and converted
  1214. .CW troff
  1215. to UTF.
  1216. Rich Drechsler converted his Postscript driver to UTF.
  1217. John Hobby built the Postscript ☺.
  1218. We thank them all.
  1219. .SH
  1220. References
  1221. .LP
  1222. [ANSIC] \f2American National Standard for Information Systems \-
  1223. Programming Language C\f1, American National Standards Institute, Inc.,
  1224. New York, 1990.
  1225. .LP
  1226. [ISO10646]
  1227. ISO/IEC DIS 10646-1:1993
  1228. \f2Information technology \-
  1229. Universal Multiple-Octet Coded Character Set (UCS) \(em
  1230. Part 1: Architecture and Basic Multilingual Plane\fP.
  1231. .LP
  1232. [Pike90] R. Pike, D. Presotto, K. Thompson, H. Trickey,
  1233. ``Plan 9 from Bell Labs'',
  1234. UKUUG Proc. of the Summer 1990 Conf.,
  1235. London, England,
  1236. 1990.
  1237. .LP
  1238. [Pike91] R. Pike, ``8½, The Plan 9 Window System'', USENIX Summer
  1239. Conf. Proc., Nashville, 1991, reprinted in this volume.
  1240. .LP
  1241. [Pike92] R. Pike, ``How to Use the Plan 9 C Compiler'', this volume.
  1242. .LP
  1243. [Pong and Zhang] Man-Chi Pong and Yongguang Zhang, ``cxterm:
  1244. A Chinese Terminal Emulator for the X Window System'',
  1245. .I
  1246. Software\(emPractice and Experience,
  1247. .R
  1248. Vol 22(1), 809-926, October 1992.
  1249. .LP
  1250. [Unicode]
  1251. \f2The Unicode Standard,
  1252. Worldwide Character Encoding,
  1253. Version 1.0, Volume 1\f1,
  1254. The Unicode Consortium,
  1255. Addison Wesley,
  1256. New York,
  1257. 1991.