1
0

acidpaper.ms 44 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864865866867868869870871872873874875876877878879880881882883884885886887888889890891892893894895896897898899900901902903904905906907908909910911912913914915916917918919920921922923924925926927928929930931932933934935936937938939940941942943944945946947948949950951952953954955956957958959960961962963964965966967968969970971972973974975976977978979980981982983984985986987988989990991992993994995996997998999100010011002100310041005100610071008100910101011101210131014101510161017101810191020102110221023102410251026102710281029103010311032103310341035103610371038103910401041104210431044104510461047104810491050105110521053105410551056105710581059106010611062106310641065106610671068106910701071107210731074107510761077107810791080108110821083108410851086108710881089109010911092109310941095109610971098109911001101110211031104110511061107110811091110111111121113111411151116111711181119112011211122112311241125112611271128112911301131113211331134113511361137113811391140114111421143114411451146114711481149115011511152115311541155115611571158115911601161116211631164116511661167116811691170117111721173117411751176117711781179118011811182118311841185118611871188118911901191119211931194119511961197119811991200120112021203120412051206120712081209121012111212121312141215121612171218121912201221122212231224122512261227122812291230123112321233123412351236123712381239124012411242124312441245124612471248124912501251125212531254125512561257125812591260126112621263126412651266126712681269127012711272127312741275127612771278127912801281128212831284128512861287128812891290129112921293129412951296129712981299130013011302130313041305130613071308130913101311131213131314131513161317131813191320132113221323
  1. .TL
  2. Acid: A Debugger Built From A Language
  3. .AU
  4. Phil Winterbottom
  5. philw@plan9.bell-labs.com
  6. .AB
  7. .FS
  8. Originally appeared in
  9. .I
  10. Proc. of the Winter 1994 USENIX Conf.,
  11. .R
  12. pp. 211-222,
  13. San Francisco, CA
  14. .FE
  15. Acid is an unusual source-level symbolic debugger for Plan 9. It is implemented
  16. as a language interpreter with specialized primitives that provide
  17. debugger support. Programs written in the language manipulate
  18. one or more target processes; variables in the language represent the
  19. symbols, state, and resources of those processes.
  20. This structure allows complex
  21. interaction between the debugger and the target program and
  22. provides a convenient method of parameterizing differences between
  23. machine architectures.
  24. Although some effort is required to learn
  25. the debugging language, the richness and flexibility of the
  26. debugging environment encourages new ways of reasoning about the way
  27. programs run and the conditions under which they fail.
  28. .AE
  29. .NH
  30. Introduction
  31. .PP
  32. The size and complexity
  33. of programs have increased in proportion to processor speed and memory but
  34. the interface between debugger and programmer has changed little.
  35. Graphical user interfaces have eased some of the tedious
  36. aspects of the interaction. A graphical interface is a convenient
  37. means for navigating through source and data structures but provides
  38. little benefit for process control.
  39. The introduction of a new concurrent language, Alef [Win93], emphasized the
  40. inadequacies of the existing Plan 9 [Pike90] debugger
  41. .I db ,
  42. a distant relative of
  43. .I adb ,
  44. and made it clear that a new debugger was required.
  45. .PP
  46. Current debuggers like
  47. .I dbx ,
  48. .I sdb ,
  49. and
  50. .I gdb
  51. are limited to answering only the questions their authors
  52. envisage. As a result, they supply a plethora
  53. of specialized commands, each attempting to anticipate
  54. a specific question a user may ask.
  55. When a debugging situation arises that is beyond the scope
  56. of the command set, the tool is useless.
  57. Further,
  58. it is often tedious or impossible to reproduce an anomalous state
  59. of the program, especially when
  60. the state is embedded in the program's data structures.
  61. .PP
  62. Acid applies some ideas found in CAD software used for
  63. hardware test and simulation.
  64. It is based on the notion that the state and resources of a program
  65. are best represented and manipulated by a language. The state and resources,
  66. such as memory, registers, variables, type information and source code
  67. are represented by variables in the language.
  68. Expressions provide a computation mechanism and control
  69. statements allow repetitive or selective interpretation based
  70. on the result of expression evaluation.
  71. The heart of the Acid debugger is an interpreter for a small typeless
  72. language whose operators mirror the operations
  73. of C and Alef, which in turn correspond well to the basic operations of
  74. the machine. The interpreter itself knows nothing of the underlying
  75. hardware; it deals with the program state and resources
  76. in the abstract.
  77. Fundamental routines to control
  78. processes, read files, and interface to the system are implemented
  79. as builtin functions available to the interpreter.
  80. The actual debugger functionality is coded
  81. in Acid; commands are implemented as Acid functions.
  82. .PP
  83. This language-based approach has several advantages.
  84. Most importantly, programs written in Acid, including most of the
  85. debugger itself, are inherently portable.
  86. Furthermore, Acid avoids the limitations other debuggers impose when
  87. debugging parallel programs. Instead of embedding a fixed
  88. process model in the debugger, Acid allows the
  89. programmer to adapt the debugger to handle an
  90. arbitrary process partitioning or program structure.
  91. The ability to
  92. interact dynamically with an executing process provides clear advantages
  93. over debuggers constrained to probe a static image.
  94. Finally, the Acid language is a powerful vehicle for expressing
  95. assertions about logic, process state, and the contents of data structures.
  96. When combined with dynamic interaction it allows a
  97. limited form of automated program verification without requiring
  98. modification or recompilation of the source code.
  99. The language is also an
  100. excellent vehicle for preserving a test suite for later regression testing.
  101. .PP
  102. The debugger may be customized by its users; standard
  103. functions may be modified or extended to suit a particular application
  104. or preference.
  105. For example, the kernel developers in our group require a
  106. command set supporting assembler-level debugging while the application
  107. programmers prefer source-level functionality.
  108. Although the default library is biased toward assembler-level debugging,
  109. it is easily modified to provide a convenient source-level interface.
  110. The debugger itself does not change; the user combines primitives
  111. and existing Acid functions in different ways to
  112. implement the desired interface.
  113. .NH
  114. Related Work
  115. .PP
  116. DUEL [Gol93], an extension to
  117. .I gdb
  118. [Stal91], proposes using a high level expression evaluator to solve
  119. some of these problems. The evaluator provides iterators to loop over data
  120. structures and conditionals to control evaluation of expressions.
  121. The author shows that complex state queries can be formulated
  122. by combining concise expressions but this only addresses part of the problem.
  123. A program is a dynamic entity; questions asked when the program is in
  124. a static state are meaningful only after the program has been `caught' in
  125. that state. The framework for manipulating the program is still as
  126. primitive as the underlying debugger. While DUEL provides a means to
  127. probe data structures it entirely neglects the most beneficial aspect
  128. of debugging languages: the ability to control processes. Acid is structured
  129. around a thread of control that passes between the interpreter and the
  130. target program.
  131. .PP
  132. The NeD debugger [May92] is a set of extensions to TCL [Ous90] that provide
  133. debugging primitives. The resulting language, NeDtcl, is used to implement
  134. a portable interface between a conventional debugger, pdb [May90], and
  135. a server that executes NeDtcl programs operating on the target program.
  136. Execution of the NeDtcl programs implements the debugging primitives
  137. that pdb expects.
  138. NeD is targeted at multi-process debugging across a network,
  139. and proves the flexibility of a language as a means of
  140. communication between debugging tools. Whereas NeD provides an interface
  141. between a conventional debugger and the process it debugs, Acid is the
  142. debugger itself. While NeD has some of the ideas
  143. found in Acid it is targeted toward a different purpose. Acid seeks to
  144. integrate the manipulation of a program's resources into the debugger
  145. while NeD provides a flexible interconnect between components of
  146. the debugging environment. The choice of TCL is appropriate for its use
  147. in NeD but is not suitable for Acid. Acid relies on the coupling of the type
  148. system with expression evaluation, which are the root of its design,
  149. to provide the debugging primitives.
  150. .PP
  151. Dalek [Ols90] is an event based language extension to gdb. State transitions
  152. in the target program cause events to be queued for processing by the
  153. debugging language.
  154. .PP
  155. Acid has many of the advantages of same process or
  156. .I local
  157. .I agent
  158. debuggers, like Parasight [Aral], without the need for dynamic linking or
  159. shared memory.
  160. Acid improves on the ideas of these other systems by completely integrating
  161. all aspects of the debugging process into the language environment. Of
  162. particular importance is the relationship between Acid variables,
  163. program symbols, source code, registers and type information. This
  164. integration is made possible by the design of the Acid language.
  165. .PP
  166. Interpreted languages such as Lisp and Smalltalk are able to provide
  167. richer debugging environments through more complete information than
  168. their compiled counterparts. Acid is a means to gather and represent
  169. similar information about compiled programs through cooperation
  170. with the compilation tools and library implementers.
  171. .NH
  172. Acid the Language
  173. .PP
  174. Acid is a small interpreted language targeted to its debugging task.
  175. It focuses on representing program state and addressing data rather than
  176. expressing complex computations. Program state is
  177. .I addressable
  178. from an Acid program.
  179. In addition to parsing and executing expressions and providing
  180. an architecture-independent interface to the target process,
  181. the interpreter supplies a mark-and-scan garbage collector
  182. to manage storage.
  183. .PP
  184. Every Acid session begins with the loading of the Acid libraries.
  185. These libraries contain functions, written in Acid, that provide
  186. a standard debugging environment including breakpoint management,
  187. stepping by instruction or statement, stack tracing, and
  188. access to variables, memory, and registers.
  189. The library contains 600 lines of Acid code and provides
  190. functionality similar to
  191. .I dbx .
  192. Following the loading of the system library, Acid loads
  193. user-specified libraries; this load sequence allows the
  194. user to augment or override the standard commands
  195. to customize the debugging environment. When all libraries
  196. are loaded, Acid issues an interactive prompt and begins
  197. evaluating expressions entered by the user. The Acid `commands'
  198. are actually invocations of builtin primitives or previously defined
  199. Acid functions. Acid evaluates each expression as it is entered and
  200. prints the result.
  201. .NH
  202. Types and Variables
  203. .PP
  204. Acid variables are of four basic types:
  205. .I integer ,
  206. .I string ,
  207. .I float ,
  208. and
  209. .I list .
  210. The type of a variable is inferred by the type of the right-hand side of
  211. an assignment expression.
  212. Many of the operators can be applied to more than
  213. one type; for these operators the action of the operator is determined
  214. by the type of its operands.
  215. For example,
  216. the
  217. .CW +
  218. operator adds
  219. .I integer
  220. and
  221. .I float
  222. operands, and concatenates
  223. .I string
  224. and
  225. .I list
  226. operands.
  227. Lists are the only complex type in Acid; there are no arrays, structures
  228. or pointers. Operators provide
  229. .CW head ,
  230. .CW tail ,
  231. .CW append
  232. and
  233. .CW delete
  234. operations.
  235. Lists can also be indexed like arrays.
  236. .PP
  237. Acid has two levels of scope: global and local.
  238. Function parameters and variables declared in a function body
  239. using the
  240. .CW local
  241. keyword are created at entry to the function and
  242. exist for the lifetime of a function.
  243. Global variables are created by assignment and need not be declared.
  244. All variables and functions in the program
  245. being debugged are entered in the Acid symbol table as global
  246. variables during Acid initialization.
  247. Conflicting variable names are resolved by prefixing enough `$' characters
  248. to make them unique.
  249. Syntactically, Acid variables and target program
  250. symbols are referenced identically.
  251. However, the variables are managed differently in the Acid
  252. symbol table and the user must be aware of this distinction.
  253. The value of an Acid variable is stored in the symbol
  254. table; a reference returns the value.
  255. The symbol table entry for a variable or function in the target
  256. program contains the address of that symbol in the image
  257. of the program. Thus, the value of a program variable is
  258. accessed by indirect reference through the Acid
  259. variable that has the same name; the value of an Acid variable is the
  260. address of the corresponding program variable.
  261. .NH
  262. Control Flow
  263. .PP
  264. The
  265. .CW while
  266. and
  267. .CW loop
  268. statements implement looping.
  269. The former
  270. is similar to the same statement in C.
  271. The latter evaluates starting and ending expressions yielding
  272. integers and iterates while an incrementing loop index
  273. is within the bounds of those expressions.
  274. .P1
  275. acid: i = 0; loop 1,5 do print(i=i+1)
  276. 0x00000001
  277. 0x00000002
  278. 0x00000003
  279. 0x00000004
  280. 0x00000005
  281. acid:
  282. .P2
  283. The traditional
  284. .CW if-then-else
  285. statement implements conditional execution.
  286. .NH
  287. Addressing
  288. .PP
  289. Two indirection operators allow Acid to access values in
  290. the program being debugged.
  291. The
  292. .CW *
  293. operator fetches a value from the memory image of an
  294. executing process;
  295. the
  296. .CW @
  297. operator fetches a value from the text file of the process.
  298. When either operator appears on the left side of an assignment, the value
  299. is written rather than read.
  300. .PP
  301. The indirection operator must know the size of the object
  302. referenced by a variable.
  303. The Plan 9 compilers neglect to include this
  304. information in the program symbol table, so Acid cannot
  305. derive this information implicitly.
  306. Instead Acid variables have formats.
  307. The format is a code
  308. letter specifying the printing style and the effect of some of the
  309. operators on that variable.
  310. The indirection operators look at the format code to determine the
  311. number of bytes to read or write.
  312. The format codes are derived from the format letters used by
  313. .I db .
  314. By default, symbol table variables and numeric constants
  315. are assigned the format code
  316. .CW 'X'
  317. which specifies 32-bit hexadecimal.
  318. Printing such a variable yields output of the form
  319. .CW 0x00123456 .
  320. An indirect reference through the variable fetches 32 bits
  321. of data at the address indicated by the variable.
  322. Other formats specify various data types, for example
  323. .CW i
  324. an instruction,
  325. .CW D
  326. a signed 32 bit decimal,
  327. .CW s
  328. a null-terminated string.
  329. The
  330. .CW fmt
  331. function
  332. allows the user to change the format code of a variable
  333. to control the printing format and
  334. operator side effects.
  335. This function evaluates the expression supplied as the first
  336. argument, attaches the format code supplied as the second
  337. argument to the result and returns that value.
  338. If the result is assigned to a variable,
  339. the new format code applies to
  340. that variable. For convenience, Acid provides the
  341. .CW \e
  342. operator as a shorthand infix form of
  343. .CW fmt .
  344. For example:
  345. .P1
  346. acid: x=10
  347. acid: x // print x in hex
  348. 0x0000000a
  349. acid: x = fmt(x, 'D') // make x type decimal
  350. acid: print(x, fmt(x, 'X'), x\eX) // print x in decimal & hex
  351. 10 0x0000000a 0x0000000a
  352. acid: x // print x in decimal
  353. 10
  354. acid: x\eo // print x in octal
  355. 000000000012
  356. .P2
  357. The
  358. .CW ++
  359. and
  360. .CW --
  361. operators increment or decrement a variable by an amount
  362. determined by its format code. Some formats imply a non-fixed size.
  363. For example, the
  364. .CW i
  365. format code disassembles an instruction into a string.
  366. On a 68020, which has variable length instructions:
  367. .P1
  368. acid: p=main\ei // p=addr(main), type INST
  369. acid: loop 1,5 do print(p\eX, @p++) // disassemble 5 instr's
  370. 0x0000222e LEA 0xffffe948(A7),A7
  371. 0x00002232 MOVL s+0x4(A7),A2
  372. 0x00002236 PEA 0x2f($0)
  373. 0x0000223a MOVL A2,-(A7)
  374. 0x0000223c BSR utfrrune
  375. acid:
  376. .P2
  377. Here,
  378. .CW main
  379. is the address of the function of the same name in the program under test.
  380. The loop retrieves the five instructions beginning at that address and
  381. then prints the address and the assembly language representation of each.
  382. Notice that the stride of the increment operator varies with the size of
  383. the instruction: the
  384. .CW MOVL
  385. at
  386. .CW 0x0000223a
  387. is a two byte instruction while all others are four bytes long.
  388. .PP
  389. Registers are treated as normal program variables referenced
  390. by their symbolic assembler language names.
  391. When a
  392. process stops, the register set is saved by the kernel
  393. at a known virtual address in the process memory map.
  394. The Acid variables associated with the registers point
  395. to the saved values and the
  396. .CW *
  397. indirection operator can then be used to read and write the register set.
  398. Since the registers are accessed via Acid variables they may
  399. be used in arbitrary expressions.
  400. .P1
  401. acid: PC // addr of saved PC
  402. 0xc0000f60
  403. acid: *PC
  404. 0x0000623c // contents of PC
  405. acid: *PC\ea
  406. main
  407. acid: *R1=10 // modify R1
  408. acid: asm(*PC+4) // disassemble @ PC+4
  409. main+0x4 0x00006240 MOVW R31,0x0(R29)
  410. main+0x8 0x00006244 MOVW $setR30(SB),R30
  411. main+0x10 0x0000624c MOVW R1,_clock(SB)
  412. .P2
  413. Here, the saved
  414. .CW PC
  415. is stored at address
  416. .CW 0xc0000f60 ;
  417. its current content is
  418. .CW 0x0000623c .
  419. The
  420. .CW a ' `
  421. format code converts this value to a string specifying
  422. the address as an offset beyond the nearest symbol.
  423. After setting the value of register
  424. .CW 1 ,
  425. the example uses the
  426. .CW asm
  427. command to disassemble a short section of code beginning
  428. at four bytes beyond the current value of the
  429. .CW PC .
  430. .NH
  431. Process Interface
  432. .PP
  433. A program executing under Acid is monitored through the
  434. .I proc
  435. file system interface provided by Plan 9.
  436. Textual messages written to the
  437. .CW ctl
  438. file control the execution of the process.
  439. For example writing
  440. .CW waitstop
  441. to the control file causes the write to block until the target
  442. process enters the kernel and is stopped. When the process is stopped
  443. the write completes. The
  444. .CW startstop
  445. message starts the target process and then does a
  446. .CW waitstop
  447. action.
  448. Synchronization between the debugger and the target process is determined
  449. by the actions of the various messages. Some operate asynchronously to the
  450. target process and always complete immediately, others block until the
  451. action completes. The asynchronous messages allow Acid to control
  452. several processes simultaneously.
  453. .PP
  454. The interpreter has builtin functions named after each of the control
  455. messages. The functions take a process id as argument.
  456. Any time a control message causes the program to execute instructions
  457. the interpreter performs two actions when the control operation has completed.
  458. The Acid variables pointing at the register set are fixed up to point
  459. at the saved registers, and then
  460. the user defined function
  461. .CW stopped
  462. is executed.
  463. The
  464. .CW stopped
  465. function may print the current address,
  466. line of source or instruction and return to interactive mode. Alternatively
  467. it may traverse a complex data structure, gather statistics and then set
  468. the program running again.
  469. .PP
  470. Several Acid variables are maintained by the debugger rather than the
  471. programmer.
  472. These variables allow generic Acid code to deal with the current process,
  473. architecture specifics or the symbol table.
  474. The variable
  475. .CW pid
  476. is the process id of the current process Acid is debugging.
  477. The variable
  478. .CW symbols
  479. contains a list of lists where each sublist contains the symbol
  480. name, its type and the value of the symbol.
  481. The variable
  482. .CW registers
  483. contains a list of the machine-specific register names. Global symbols in the target program
  484. can be referenced directly by name from Acid. Local variables
  485. are referenced using the colon operator as \f(CWfunction:variable\fP.
  486. .NH
  487. Source Level Debugging
  488. .PP
  489. Acid provides several builtin functions to manipulate source code.
  490. The
  491. .CW file
  492. function reads a text file, inserting each line into a list.
  493. The
  494. .CW pcfile
  495. and
  496. .CW pcline
  497. functions each take an address as an argument.
  498. The first
  499. returns a string containing the name of the source file
  500. and the second returns an integer containing the line number
  501. of the source line containing the instruction at the address.
  502. .P1
  503. acid: pcfile(main) // file containing main
  504. main.c
  505. acid: pcline(main) // line # of main in source
  506. 11
  507. acid: file(pcfile(main))[pcline(main)] // print that line
  508. main(int argc, char *argv[])
  509. acid: src(*PC) // print statements nearby
  510. 9
  511. 10 void
  512. >11 main(int argc, char *argv[])
  513. 12 {
  514. 13 int a;
  515. .P2
  516. In this example, the three primitives are combined in an expression to print
  517. a line of source code associated with an address.
  518. The
  519. .CW src
  520. function prints a few lines of source
  521. around the address supplied as its argument. A companion routine,
  522. .CW Bsrc ,
  523. communicates with the external editor
  524. .CW sam .
  525. Given an address, it loads the corresponding source file into the editor
  526. and highlights the line containing the address. This simple interface
  527. is easily extended to more complex functions.
  528. For example, the
  529. .CW step
  530. function can select the current file and line in the editor
  531. each time the target program stops, giving the user a visual
  532. trace of the execution path of the program. A more complete interface
  533. allowing two way communication between Acid and the
  534. .CW acme
  535. user interface [Pike93] is under construction. A filter between the debugger
  536. and the user interface provides interpretation of results from both
  537. sides of the interface. This allows the programming environment to
  538. interact with the debugger and vice-versa, a capability missing from the
  539. .CW sam
  540. interface.
  541. The
  542. .CW src
  543. and
  544. .CW Bsrc
  545. functions are both written in Acid code using the file and line primitives.
  546. Acid provides library functions to step through source level
  547. statements and functions. Furthermore, addresses in Acid expressions can be
  548. specified by source file and line.
  549. Source code is manipulated in the Acid
  550. .I list
  551. data type.
  552. .NH
  553. The Acid Library
  554. .PP
  555. The following examples define some useful commands and
  556. illustrate the interaction of the debugger and the interpreter.
  557. .P1
  558. defn bpset(addr) // set breakpoint
  559. {
  560. if match(addr, bplist) >= 0 then
  561. print("bkpoint already set:", addr\ea, "\en");
  562. else {
  563. *fmt(addr, bpfmt) = bpinst; // plant it
  564. bplist = append bplist, addr; // add to list
  565. }
  566. }
  567. .P2
  568. The
  569. .CW bpset
  570. function plants a break point in memory. The function starts by
  571. using the
  572. .CW match
  573. builtin to
  574. search the breakpoint list to determine if a breakpoint is already
  575. set at the address.
  576. The indirection operator, controlled by the format code returned
  577. by the
  578. .CW fmt
  579. primitive, is used to plant the breakpoint in memory.
  580. The variables
  581. .CW bpfmt
  582. and
  583. .CW bpinst
  584. are Acid global variables containing the format code specifying
  585. the size of the breakpoint instruction and the breakpoint instruction
  586. itself.
  587. These
  588. variables are set by architecture-dependent library code
  589. when the debugger first attaches to the executing image.
  590. Finally the address of the breakpoint is
  591. appended to the breakpoint list,
  592. .CW bplist .
  593. .P1
  594. defn step() // single step
  595. {
  596. local lst, lpl, addr, bput;
  597. bput = 0; // sitting on bkpoint
  598. if match(*PC, bplist) >= 0 then {
  599. bput = fmt(*PC, bpfmt); // save current addr
  600. *bput = @bput; // replace it
  601. }
  602. lst = follow(*PC); // get follow set
  603. lpl = lst;
  604. while lpl do { // place breakpoints
  605. *(head lpl) = bpinst;
  606. lpl = tail lpl;
  607. }
  608. startstop(pid); // do the step
  609. while lst do { // remove breakpoints
  610. addr = fmt(head lst, bpfmt);
  611. *addr = @addr; // replace instr.
  612. lst = tail lst;
  613. }
  614. if bput != 0 then
  615. *bput = bpinst; // restore breakpoint
  616. }
  617. .P2
  618. The
  619. .CW step
  620. function executes a single assembler instruction.
  621. If the
  622. .CW PC
  623. is sitting
  624. on a breakpoint, the address and size of
  625. the breakpoint are saved.
  626. The breakpoint instruction
  627. is then removed using the
  628. .CW @
  629. operator to fetch
  630. .CW bpfmt
  631. bytes from the text file and to place it into the memory
  632. of the executing process using the
  633. .CW *
  634. operator.
  635. The
  636. .CW follow
  637. function is an Acid
  638. builtin which returns a follow-set: a list of instruction addresses which
  639. could be executed next.
  640. If the instruction stored at the
  641. .CW PC
  642. is a branch instruction, the
  643. list contains the addresses of the next instruction and
  644. the branch destination; otherwise, it contains only the
  645. address of the next instruction.
  646. The follow-set is then used to replace each possible following
  647. instruction with a breakpoint instruction. The original
  648. instructions need not be saved; they remain
  649. in their unaltered state in the text file.
  650. The
  651. .CW startstop
  652. builtin writes the `startstop' message to the
  653. .I proc
  654. control file for the process named
  655. .CW pid .
  656. The target process executes until some condition causes it to
  657. enter the kernel, in this case, the execution of a breakpoint.
  658. When the process blocks, the debugger regains control and invokes the
  659. Acid library function
  660. .CW stopped
  661. which reports the address and cause of the blockage.
  662. The
  663. .CW startstop
  664. function completes and returns to the
  665. .CW step
  666. function where
  667. the follow-set is used to replace the breakpoints placed earlier.
  668. Finally, if the address of the original
  669. .CW PC
  670. contained a breakpoint, it is replaced.
  671. .PP
  672. Notice that this approach to process control is inherently portable;
  673. the Acid code is shared by the debuggers for all architectures.
  674. Acid variables and builtin functions provide a transparent interface
  675. to architecture-dependent values and functions. Here the breakpoint
  676. value and format are referenced through Acid variables and the
  677. .CW follow
  678. primitive masks the differences in the underlying instruction set.
  679. .PP
  680. The
  681. .CW next
  682. function, similar to the
  683. .I dbx
  684. command of the same name,
  685. is a simpler example.
  686. This function steps through
  687. a single source statement but steps over function calls.
  688. .P1
  689. defn next()
  690. {
  691. local sp, bound;
  692. sp = *SP; // save starting SP
  693. bound = fnbound(*PC); // begin & end of fn.
  694. stmnt(); // step 1 statement
  695. pc = *PC;
  696. if pc >= bound[0] && pc < bound[1] then
  697. return {};
  698. while (pc<bound[0] || pc>bound[1]) && sp>=*SP do {
  699. step();
  700. pc = *PC;
  701. }
  702. src(*PC);
  703. }
  704. .P2
  705. The
  706. .CW next
  707. function
  708. starts by saving the current stack pointer in a local variable.
  709. It then uses the Acid library function
  710. .CW fnbound
  711. to return the addresses of the first and last instructions in
  712. the current function in a list.
  713. The
  714. .CW stmnt
  715. function executes a single source statement and then uses
  716. .CW src
  717. to print a few lines of source around the new
  718. .CW PC .
  719. If the new value of the
  720. .CW PC
  721. remains in the current function,
  722. .CW next
  723. returns.
  724. When the executed statement is a function call or a return
  725. from a function, the new value of the
  726. .CW PC
  727. is outside the bounds calculated by
  728. .CW fnbound
  729. and the test of the
  730. .CW while
  731. loop is evaluated.
  732. If the statement was a return, the new value of the stack pointer
  733. is greater than the original value and the loop completes without
  734. execution.
  735. Otherwise, the loop is entered and instructions are continually
  736. executed until the value of the
  737. .CW PC
  738. is between the bounds calculated earlier. At that point, execution
  739. ceases and a few lines of source in the vicinity of the
  740. .CW PC
  741. are printed.
  742. .PP
  743. Acid provides concise and elegant expression for control and
  744. manipulation of target programs. These examples demonstrate how a
  745. few well-chosen primitives can be combined to create a rich debugging environment.
  746. .NH
  747. Dealing With Multiple Architectures
  748. .PP
  749. A single binary of Acid may be used to debug a program running on any
  750. of the five processor architectures supported by Plan 9. For example,
  751. Plan 9 allows a user on a MIPS to import the
  752. .I proc
  753. file system from an i486-based PC and remotely debug a program executing
  754. on that processor.
  755. .PP
  756. Two levels of abstraction provide this architecture independence.
  757. On the lowest level, a Plan 9 library supplies functions to
  758. decode the file header of the program being debugged and
  759. select a table of system parameters
  760. and a jump vector of architecture-dependent
  761. functions based on the magic number.
  762. Among these functions are byte-order-independent
  763. access to memory and text files, stack manipulation, disassembly,
  764. and floating point number interpretation.
  765. The second level of abstraction is supplied by Acid.
  766. It consists of primitives and approximately 200 lines
  767. of architecture-dependent Acid library code that interface the
  768. interpreter to the architecture-dependent library.
  769. This layer performs functions such as mapping register names to
  770. memory locations, supplying breakpoint values and sizes,
  771. and converting processor specific data to Acid data types.
  772. An example of the latter is the stack trace function
  773. .CW strace ,
  774. which uses the stack traversal functions in the
  775. architecture-dependent library to construct a list of lists describing
  776. the context of a process. The first level of list selects
  777. each function in the trace; subordinate lists contain the
  778. names and values of parameters and local variables of
  779. the functions. Acid commands and library functions that
  780. manipulate and display process state information operate
  781. on the list representation and are independent of the
  782. underlying architecture.
  783. .NH
  784. Alef Runtime
  785. .PP
  786. Alef is a concurrent programming language,
  787. designed specifically for systems programming, which supports both
  788. shared variable and message passing paradigms.
  789. Alef borrows the C expression syntax but implements
  790. a substantially different type system.
  791. The language provides a rich set of
  792. exception handling, process management, and synchronization
  793. primitives, which rely on a runtime system.
  794. Alef program bugs are often deadlocks, synchronization failures,
  795. or non-termination caused by locks being held incorrectly.
  796. In such cases, a process stalls deep
  797. in the runtime code and it is clearly
  798. unreasonable to expect a programmer using the language
  799. to understand the detailed
  800. internal semantics of the runtime support functions.
  801. .PP
  802. Instead, there is an Alef support library, coded in Acid, that
  803. allows the programmer to interpret the program state in terms of
  804. Alef operations. Consider the example of a multi-process program
  805. stalling because of improper synchronization. A stack trace of
  806. the program indicates that it is waiting for an event in some
  807. obscure Alef runtime
  808. synchronization function.
  809. The function itself is irrelevant to the
  810. programmer; of greater importance is the identity of the
  811. unfulfilled event.
  812. Commands in the Alef support library decode
  813. the runtime data structures and program state to report the cause
  814. of the blockage in terms of the high-level operations available to
  815. the Alef programmer.
  816. Here, the Acid language acts
  817. as a communications medium between Alef implementer and Alef user.
  818. .NH
  819. Parallel Debugging
  820. .PP
  821. The central issue in parallel debugging is how the debugger is
  822. multiplexed between the processes comprising
  823. the program.
  824. Acid has no intrinsic model of process partitioning; it
  825. only assumes that parallel programs share a symbol table,
  826. though they need not share memory.
  827. The
  828. .CW setproc
  829. primitive attaches the debugger to a running process
  830. associated with the process ID supplied as its argument
  831. and assigns that value to the global variable
  832. .CW pid ,
  833. thereby allowing simple rotation among a group of processes.
  834. Further, the stack trace primitive is driven by parameters
  835. specifying a unique process context, so it is possible to
  836. examine the state of cooperating processes without switching
  837. the debugger focus from the process of interest.
  838. Since Acid is inherently extensible and capable of
  839. dynamic interaction with subordinate processes, the
  840. programmer can define Acid commands to detect and control
  841. complex interactions between processes.
  842. In short, the programmer is free to specify how the debugger reacts
  843. to events generated in specific threads of the program.
  844. .PP
  845. The support for parallel debugging in Acid depends on a crucial kernel
  846. modification: when the text segment of a program is written (usually to
  847. place a breakpoint), the segment is cloned to prevent other threads
  848. from encountering the breakpoint. Although this incurs a slight performance
  849. penalty, it is of little importance while debugging.
  850. .NH
  851. Communication Between Tools
  852. .PP
  853. The Plan 9 Alef and C compilers do not
  854. embed detailed type information in the symbol table of an
  855. executable file.
  856. However, they do accept a command line option causing them to
  857. emit descriptions of complex data types
  858. (e.g., aggregates and abstract data types)
  859. to an auxiliary file.
  860. The vehicle for expressing this information is Acid source code.
  861. When an Acid debugging session is
  862. subsequently started, that file is loaded with the other Acid libraries.
  863. .PP
  864. For each complex object in the program the compiler generates
  865. three pieces of Acid code.
  866. The first is a table describing the size and offset of each
  867. member of the complex data type. Following is an Acid function,
  868. named the same as the object, that formats and prints each member.
  869. Finally, Acid declarations associate the
  870. Alef or C program variables of a type with the functions
  871. to print them.
  872. The three forms of declaration are shown in the following example:
  873. .P1
  874. struct Bitmap {
  875. Rectangle 0 r;
  876. Rectangle 16 clipr;
  877. 'D' 32 ldepth;
  878. 'D' 36 id;
  879. 'X' 40 cache;
  880. };
  881. .P2
  882. .P1
  883. defn
  884. Bitmap(addr) {
  885. complex Bitmap addr;
  886. print("Rectangle r {\en");
  887. Rectangle(addr.r);
  888. print("}\en");
  889. print("Rectangle clipr {\en");
  890. Rectangle(addr.clipr);
  891. print("}\en");
  892. print(" ldepth ", addr.ldepth, "\en");
  893. print(" id ", addr.id, "\en");
  894. print(" cache ", addr.cache, "\en");
  895. };
  896. complex Bitmap darkgrey;
  897. complex Bitmap Window_settag:b;
  898. .P2
  899. The
  900. .CW struct
  901. declaration specifies decoding instructions for the complex type named
  902. .CW Bitmap .
  903. Although the syntax is superficially similar to a C structure declaration,
  904. the semantics differ markedly: the C declaration specifies a layout, while
  905. the Acid declaration tells how to decode it.
  906. The declaration specifies a type, an offset, and name for each
  907. member of the complex object. The type is either the name of another
  908. complex declaration, for example,
  909. .CW Rectangle ,
  910. or a format code.
  911. The offset is the number of bytes from the start
  912. of the object to the member
  913. and the name is the member's name in the Alef or C declaration.
  914. This type description is a close match for C and Alef, but is simple enough
  915. to be language independent.
  916. .PP
  917. The
  918. .CW Bitmap
  919. function expects the address of a
  920. .CW Bitmap
  921. as its only argument.
  922. It uses the decoding information contained in the
  923. .CW Bitmap
  924. structure declaration to extract, format, and print the
  925. value of each member of the complex object pointed to by
  926. the argument.
  927. The Alef compiler emits code to call other Acid functions
  928. where a member is another complex type; here,
  929. .CW Bitmap
  930. calls
  931. .CW Rectangle
  932. to print its contents.
  933. .PP
  934. The
  935. .CW complex
  936. declarations associate Alef variables with complex types.
  937. In the example,
  938. .CW darkgrey
  939. is the name of a global variable of type
  940. .CW Bitmap
  941. in the program being debugged.
  942. Whenever the name
  943. .CW darkgrey
  944. is evaluated by Acid, it automatically calls the
  945. .CW Bitmap
  946. function with the address of
  947. .CW darkgrey
  948. as the argument.
  949. The second
  950. .CW complex
  951. declaration associates a local variable or parameter named
  952. .CW b
  953. in function
  954. .CW Window_settag
  955. with the
  956. .CW Bitmap
  957. complex data type.
  958. .PP
  959. Acid borrows the C operators
  960. .CW .
  961. and
  962. .CW ->
  963. to access the decoding parameters of a member of a complex type.
  964. Although this representation is sufficiently general for describing
  965. the decoding of both C and Alef complex data types, it may
  966. prove too restrictive for target languages with more complicated
  967. type systems.
  968. Further, the assumption that the compiler can select the proper
  969. Acid format code for each basic type in the language is somewhat
  970. naive. For example, when a member of a complex type is a pointer,
  971. it is assigned a hexadecimal type code; integer members are always
  972. assigned a decimal type code.
  973. This heuristic proves inaccurate when an integer field is a
  974. bit mask or set of bit flags which are more appropriately displayed
  975. in hexadecimal or octal.
  976. .NH
  977. Code Verification
  978. .PP
  979. Acid's ability to interact dynamically with
  980. an executing program allows passive test and
  981. verification of the target program. For example,
  982. a common concern is leak detection in programs using
  983. .CW malloc .
  984. Of interest are two items: finding memory that was allocated
  985. but never freed and detecting bad pointers passed to
  986. .CW free .
  987. An auxiliary Acid library contains Acid functions to
  988. monitor the execution of a program and detect these
  989. faults, either as they happen or in the automated
  990. post-mortem analysis of the memory arena.
  991. In the following example, the
  992. .CW sort
  993. command is run under the control of the
  994. Acid memory leak library.
  995. .P1
  996. helix% acid -l malloc /bin/sort
  997. /bin/sort: mips plan 9 executable
  998. /lib/acid/port
  999. /lib/acid/mips
  1000. /lib/acid/malloc
  1001. acid: go()
  1002. now
  1003. is
  1004. the
  1005. time
  1006. <ctrl-d>
  1007. is
  1008. now
  1009. the
  1010. time
  1011. 27680 : breakpoint _exits+0x4 MOVW $0x8,R1
  1012. acid:
  1013. .P2
  1014. The
  1015. .CW go
  1016. command creates a process and plants
  1017. breakpoints at the entry to
  1018. .CW malloc
  1019. and
  1020. .CW free .
  1021. The program is then started and continues until it
  1022. exits or stops. If the reason for stopping is anything
  1023. other than the breakpoints in
  1024. .CW malloc
  1025. and
  1026. .CW free ,
  1027. Acid prints the usual status information and returns to the
  1028. interactive prompt.
  1029. .PP
  1030. When the process stops on entering
  1031. .CW malloc ,
  1032. the debugger must capture and save the address that
  1033. .CW malloc
  1034. will return.
  1035. After saving a stack
  1036. trace so the calling routine can be identified, it places
  1037. a breakpoint at the return address and restarts the program.
  1038. When
  1039. .CW malloc
  1040. returns, the breakpoint stops the program,
  1041. allowing the debugger
  1042. to grab the address of the new memory block from the return register.
  1043. The address and stack trace are added to the list of outstanding
  1044. memory blocks, the breakpoint is removed from the return point, and
  1045. the process is restarted.
  1046. .PP
  1047. When the process stops at the beginning of
  1048. .CW free ,
  1049. the memory address supplied as the argument is compared to the list
  1050. of outstanding memory blocks. If it is not found an error message
  1051. and a stack trace of the call is reported; otherwise, the
  1052. address is deleted from the list.
  1053. .PP
  1054. When the program exits, the list of outstanding memory blocks contains
  1055. the addresses of all blocks that were allocated but never freed.
  1056. The
  1057. .CW leak
  1058. library function traverses the list producing a report describing
  1059. the allocated blocks.
  1060. .P1 1m
  1061. acid: leak()
  1062. Lost a total of 524288 bytes from:
  1063. malloc() malloc.c:32 called from dofile+0xe8 sort.c:217
  1064. dofile() sort.c:190 called from main+0xac sort.c:161
  1065. main() sort.c:128 called from _main+0x20 main9.s:10
  1066. Lost a total of 64 bytes from:
  1067. malloc() malloc.c:32 called from newline+0xfc sort.c:280
  1068. newline() sort.c:248 called from dofile+0x110 sort.c:222
  1069. dofile() sort.c:190 called from main+0xac sort.c:161
  1070. main() sort.c:128 called from _main+0x20 main9.s:10
  1071. Lost a total of 64 bytes from:
  1072. malloc() malloc.c:32 called from realloc+0x14 malloc.c:129
  1073. realloc() malloc.c:123 called from bldkey+0x358 sort.c:1388
  1074. buildkey() sort.c:1345 called from newline+0x150 sort.c:285
  1075. newline() sort.c:248 called from dofile+0x110 sort.c:222
  1076. dofile() sort.c:190 called from main+0xac sort.c:161
  1077. main() sort.c:128 called from _main+0x20 main9.s:10
  1078. acid: refs()
  1079. data...bss...stack...
  1080. acid: leak()
  1081. acid:
  1082. .P2
  1083. The presence of a block in the allocation list does not imply
  1084. it is there because of a leak; for instance, it may have been
  1085. in use when the program terminated.
  1086. The
  1087. .CW refs()
  1088. library function scans the
  1089. .I data ,
  1090. .I bss ,
  1091. and
  1092. .I stack
  1093. segments of the process looking for pointers
  1094. into the allocated blocks. When one is found, the block is deleted from
  1095. the outstanding block list.
  1096. The
  1097. .CW leak
  1098. function is used again to report the
  1099. blocks remaining allocated and unreferenced.
  1100. This strategy proves effective in detecting
  1101. disconnected (but non-circular) data structures.
  1102. .PP
  1103. The leak detection process is entirely passive.
  1104. The program is not
  1105. specially compiled and the source code is not required.
  1106. As with the Acid support functions for the Alef runtime environment,
  1107. the author of the library routines has encapsulated the
  1108. functionality of the library interface
  1109. in Acid code.
  1110. Any programmer may then check a program's use of the
  1111. library routines without knowledge of either implementation.
  1112. The performance impact of running leak detection is great
  1113. (about 10 times slower),
  1114. but it has not prevented interactive programs like
  1115. .CW sam
  1116. and the
  1117. .CW 8½
  1118. window system from being tested.
  1119. .NH
  1120. Code Coverage
  1121. .PP
  1122. Another common component of software test uses
  1123. .I coverage
  1124. analysis.
  1125. The purpose of the test is to determine which paths through the code have
  1126. not been executed while running the test suite.
  1127. This is usually
  1128. performed by a combination of compiler support and a reporting tool run
  1129. on the output generated by statements compiled into the program.
  1130. The compiler emits code that
  1131. logs the progress of the program as it executes basic blocks and writes the
  1132. results to a file. The file is then processed by the reporting tool
  1133. to determine which basic blocks have not been executed.
  1134. .PP
  1135. Acid can perform the same function in a language independent manner without
  1136. modifying the source, object or binary of the program. The following example
  1137. shows
  1138. .CW ls
  1139. being run under the control of the Acid coverage library.
  1140. .P1
  1141. philw-helix% acid -l coverage /bin/ls
  1142. /bin/ls: mips plan 9 executable
  1143. /lib/acid/port
  1144. /lib/acid/mips
  1145. /lib/acid/coverage
  1146. acid: coverage()
  1147. acid
  1148. newstime
  1149. profile
  1150. tel
  1151. wintool
  1152. 2: (error) msg: pid=11419 startstop: process exited
  1153. acid: analyse(ls)
  1154. ls.c:102,105
  1155. 102: return 1;
  1156. 103: }
  1157. 104: if(db[0].qid.path&CHDIR && dflag==0){
  1158. 105: output();
  1159. ls.c:122,126
  1160. 122: memmove(dirbuf+ndir, db, sizeof(Dir));
  1161. 123: dirbuf[ndir].prefix = 0;
  1162. 124: p = utfrrune(s, '/');
  1163. 125: if(p){
  1164. 126: dirbuf[ndir].prefix = s;
  1165. .P2
  1166. The
  1167. .CW coverage
  1168. function begins by looping through the text segment placing
  1169. breakpoints at the entry to each basic block. The start of each basic
  1170. block is found using the Acid builtin function
  1171. .CW follow .
  1172. If the list generated by
  1173. .CW follow
  1174. contains more than one
  1175. element, then the addresses mark the start of basic blocks. A breakpoint
  1176. is placed at each address to detect entry into the block. If the result
  1177. of
  1178. .CW follow
  1179. is a single address then no action is taken, and the next address is
  1180. considered. Acid maintains a list of
  1181. breakpoints already in place and avoids placing duplicates (an address may be
  1182. the destination of several branches).
  1183. .PP
  1184. After placing the breakpoints the program is set running.
  1185. Each time a breakpoint is encountered
  1186. Acid deletes the address from the breakpoint list, removes the breakpoint
  1187. from memory and then restarts the program.
  1188. At any instant the breakpoint list contains the addresses of basic blocks
  1189. which have not been executed.
  1190. The
  1191. .CW analyse
  1192. function reports the lines of source code bounded by basic blocks
  1193. whose addresses are have not been deleted from the breakpoint list.
  1194. These are the basic blocks which have not been executed.
  1195. Program performance is almost unaffected since each breakpoint is executed
  1196. only once and then removed.
  1197. .PP
  1198. The library contains a total of 128 lines of Acid code.
  1199. An obvious extension of this algorithm could be used to provide basic block
  1200. profiling.
  1201. .NH
  1202. Conclusion
  1203. .PP
  1204. Acid has two areas of weakness. As with
  1205. other language-based tools like
  1206. .I awk ,
  1207. a programmer must learn yet another language to step beyond the normal
  1208. debugging functions and use the full power of the debugger.
  1209. Second, the command line interface supplied by the
  1210. .I yacc
  1211. parser is inordinately clumsy.
  1212. Part of the problem relates directly to the use of
  1213. .I yacc
  1214. and could be circumvented with a custom parser.
  1215. However, structural problems would remain: Acid often requires
  1216. too much typing to execute a simple
  1217. command.
  1218. A debugger should prostitute itself to its users, doing whatever
  1219. is wanted with a minimum of encouragement; commands should be
  1220. concise and obvious. The language interface is more consistent than
  1221. an ad hoc command interface but is clumsy to use.
  1222. Most of these problems are addressed by an Acme interface
  1223. which is under construction. This should provide the best of
  1224. both worlds: graphical debugging and access to the underlying acid
  1225. language when required.
  1226. .PP
  1227. The name space clash between Acid variables, keywords, program variables,
  1228. and functions is unavoidable.
  1229. Although it rarely affects a debugging session, it is annoying
  1230. when it happens and is sometimes difficult to circumvent.
  1231. The current renaming scheme
  1232. is too crude; the new names are too hard to remember.
  1233. .PP
  1234. Acid has proved to be a powerful tool whose applications
  1235. have exceeded expectations.
  1236. Of its strengths, portability, extensibility and parallel debugging support
  1237. were by design and provide the expected utility.
  1238. In retrospect,
  1239. its use as a tool for code test and verification and as
  1240. a medium for communicating type information and encapsulating
  1241. interfaces has provided unanticipated benefits and altered our
  1242. view of the debugging process.
  1243. .NH
  1244. Acknowledgments
  1245. .PP
  1246. Bob Flandrena was the first user and helped prepare the paper.
  1247. Rob Pike endured three buggy Alef compilers and a new debugger
  1248. in a single sitting.
  1249. .NH
  1250. References
  1251. .LP
  1252. [Pike90] R. Pike, D. Presotto, K. Thompson, H. Trickey,
  1253. ``Plan 9 from Bell Labs'',
  1254. .I
  1255. UKUUG Proc. of the Summer 1990 Conf.,
  1256. .R
  1257. London, England,
  1258. 1990,
  1259. reprinted, in a different form, in this volume.
  1260. .LP
  1261. [Gol93] M. Golan, D. Hanson,
  1262. ``DUEL -- A Very High-Level Debugging Language'',
  1263. .I
  1264. USENIX Proc. of the Winter 1993 Conf.,
  1265. .R
  1266. San Diego, CA,
  1267. 1993.
  1268. .LP
  1269. [Lin90] M. A. Linton,
  1270. ``The Evolution of DBX'',
  1271. .I
  1272. USENIX Proc. of the Summer 1990 Conf.,
  1273. .R
  1274. Anaheim, CA,
  1275. 1990.
  1276. .LP
  1277. [Stal91] R. M. Stallman, R. H. Pesch,
  1278. ``Using GDB: A guide to the GNU source level debugger'',
  1279. Technical Report, Free Software Foundation,
  1280. Cambridge, MA,
  1281. 1991.
  1282. .LP
  1283. [Win93] P. Winterbottom,
  1284. ``Alef reference Manual'',
  1285. this volume.
  1286. .LP
  1287. [Pike93] Rob Pike,
  1288. ``Acme: A User Interface for Programmers'',
  1289. .I
  1290. USENIX Proc. of the Winter 1994 Conf.,
  1291. .R
  1292. San Francisco, CA,
  1293. reprinted in this volume.
  1294. .LP
  1295. [Ols90] Ronald A. Olsson, Richard H. Crawford, and W. Wilson Ho,
  1296. ``Dalek: A GNU, improved programmable debugger'',
  1297. .I
  1298. USENIX Proc. of the Summer 1990 Conf.,
  1299. .R
  1300. Anaheim, CA.
  1301. .LP
  1302. [May92] Paul Maybee,
  1303. ``NeD: The Network Extensible Debugger''
  1304. .I
  1305. USENIX Proc. of the Summer 1992 Conf.,
  1306. .R
  1307. San Antonio, TX.
  1308. .LP
  1309. [Aral] Ziya Aral, Ilya Gertner, and Greg Schaffer,
  1310. ``Efficient debugging primitives for multiprocessors'',
  1311. .I
  1312. Proceedings of the Third International Conference on Architectural
  1313. Support for Programming Languages and Operating Systems,
  1314. .R
  1315. SIGPLAN notices Nr. 22, May 1989.