acidpaper.ms 44 KB

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192939495969798991001011021031041051061071081091101111121131141151161171181191201211221231241251261271281291301311321331341351361371381391401411421431441451461471481491501511521531541551561571581591601611621631641651661671681691701711721731741751761771781791801811821831841851861871881891901911921931941951961971981992002012022032042052062072082092102112122132142152162172182192202212222232242252262272282292302312322332342352362372382392402412422432442452462472482492502512522532542552562572582592602612622632642652662672682692702712722732742752762772782792802812822832842852862872882892902912922932942952962972982993003013023033043053063073083093103113123133143153163173183193203213223233243253263273283293303313323333343353363373383393403413423433443453463473483493503513523533543553563573583593603613623633643653663673683693703713723733743753763773783793803813823833843853863873883893903913923933943953963973983994004014024034044054064074084094104114124134144154164174184194204214224234244254264274284294304314324334344354364374384394404414424434444454464474484494504514524534544554564574584594604614624634644654664674684694704714724734744754764774784794804814824834844854864874884894904914924934944954964974984995005015025035045055065075085095105115125135145155165175185195205215225235245255265275285295305315325335345355365375385395405415425435445455465475485495505515525535545555565575585595605615625635645655665675685695705715725735745755765775785795805815825835845855865875885895905915925935945955965975985996006016026036046056066076086096106116126136146156166176186196206216226236246256266276286296306316326336346356366376386396406416426436446456466476486496506516526536546556566576586596606616626636646656666676686696706716726736746756766776786796806816826836846856866876886896906916926936946956966976986997007017027037047057067077087097107117127137147157167177187197207217227237247257267277287297307317327337347357367377387397407417427437447457467477487497507517527537547557567577587597607617627637647657667677687697707717727737747757767777787797807817827837847857867877887897907917927937947957967977987998008018028038048058068078088098108118128138148158168178188198208218228238248258268278288298308318328338348358368378388398408418428438448458468478488498508518528538548558568578588598608618628638648658668678688698708718728738748758768778788798808818828838848858868878888898908918928938948958968978988999009019029039049059069079089099109119129139149159169179189199209219229239249259269279289299309319329339349359369379389399409419429439449459469479489499509519529539549559569579589599609619629639649659669679689699709719729739749759769779789799809819829839849859869879889899909919929939949959969979989991000100110021003100410051006100710081009101010111012101310141015101610171018101910201021102210231024102510261027102810291030103110321033103410351036103710381039104010411042104310441045104610471048104910501051105210531054105510561057105810591060106110621063106410651066106710681069107010711072107310741075107610771078107910801081108210831084108510861087108810891090109110921093109410951096109710981099110011011102110311041105110611071108110911101111111211131114111511161117111811191120112111221123112411251126112711281129113011311132113311341135113611371138113911401141114211431144114511461147114811491150115111521153115411551156115711581159116011611162116311641165116611671168116911701171117211731174117511761177117811791180118111821183118411851186118711881189119011911192119311941195119611971198119912001201120212031204120512061207120812091210121112121213121412151216121712181219122012211222122312241225122612271228122912301231123212331234123512361237123812391240124112421243124412451246124712481249125012511252125312541255125612571258125912601261126212631264126512661267126812691270127112721273127412751276127712781279128012811282128312841285128612871288128912901291129212931294129512961297129812991300130113021303130413051306130713081309131013111312131313141315131613171318131913201321132213231324
  1. .HTML "Acid: A Debugger Built From A Language
  2. .TL
  3. Acid: A Debugger Built From A Language
  4. .AU
  5. Phil Winterbottom
  6. philw@plan9.bell-labs.com
  7. .AB
  8. .FS
  9. Originally appeared in
  10. .I
  11. Proc. of the Winter 1994 USENIX Conf.,
  12. .R
  13. pp. 211-222,
  14. San Francisco, CA
  15. .FE
  16. Acid is an unusual source-level symbolic debugger for Plan 9. It is implemented
  17. as a language interpreter with specialized primitives that provide
  18. debugger support. Programs written in the language manipulate
  19. one or more target processes; variables in the language represent the
  20. symbols, state, and resources of those processes.
  21. This structure allows complex
  22. interaction between the debugger and the target program and
  23. provides a convenient method of parameterizing differences between
  24. machine architectures.
  25. Although some effort is required to learn
  26. the debugging language, the richness and flexibility of the
  27. debugging environment encourages new ways of reasoning about the way
  28. programs run and the conditions under which they fail.
  29. .AE
  30. .NH
  31. Introduction
  32. .PP
  33. The size and complexity
  34. of programs have increased in proportion to processor speed and memory but
  35. the interface between debugger and programmer has changed little.
  36. Graphical user interfaces have eased some of the tedious
  37. aspects of the interaction. A graphical interface is a convenient
  38. means for navigating through source and data structures but provides
  39. little benefit for process control.
  40. The introduction of a new concurrent language, Alef [Win93], emphasized the
  41. inadequacies of the existing Plan 9 [Pike90] debugger
  42. .I db ,
  43. a distant relative of
  44. .I adb ,
  45. and made it clear that a new debugger was required.
  46. .PP
  47. Current debuggers like
  48. .I dbx ,
  49. .I sdb ,
  50. and
  51. .I gdb
  52. are limited to answering only the questions their authors
  53. envisage. As a result, they supply a plethora
  54. of specialized commands, each attempting to anticipate
  55. a specific question a user may ask.
  56. When a debugging situation arises that is beyond the scope
  57. of the command set, the tool is useless.
  58. Further,
  59. it is often tedious or impossible to reproduce an anomalous state
  60. of the program, especially when
  61. the state is embedded in the program's data structures.
  62. .PP
  63. Acid applies some ideas found in CAD software used for
  64. hardware test and simulation.
  65. It is based on the notion that the state and resources of a program
  66. are best represented and manipulated by a language. The state and resources,
  67. such as memory, registers, variables, type information and source code
  68. are represented by variables in the language.
  69. Expressions provide a computation mechanism and control
  70. statements allow repetitive or selective interpretation based
  71. on the result of expression evaluation.
  72. The heart of the Acid debugger is an interpreter for a small typeless
  73. language whose operators mirror the operations
  74. of C and Alef, which in turn correspond well to the basic operations of
  75. the machine. The interpreter itself knows nothing of the underlying
  76. hardware; it deals with the program state and resources
  77. in the abstract.
  78. Fundamental routines to control
  79. processes, read files, and interface to the system are implemented
  80. as builtin functions available to the interpreter.
  81. The actual debugger functionality is coded
  82. in Acid; commands are implemented as Acid functions.
  83. .PP
  84. This language-based approach has several advantages.
  85. Most importantly, programs written in Acid, including most of the
  86. debugger itself, are inherently portable.
  87. Furthermore, Acid avoids the limitations other debuggers impose when
  88. debugging parallel programs. Instead of embedding a fixed
  89. process model in the debugger, Acid allows the
  90. programmer to adapt the debugger to handle an
  91. arbitrary process partitioning or program structure.
  92. The ability to
  93. interact dynamically with an executing process provides clear advantages
  94. over debuggers constrained to probe a static image.
  95. Finally, the Acid language is a powerful vehicle for expressing
  96. assertions about logic, process state, and the contents of data structures.
  97. When combined with dynamic interaction it allows a
  98. limited form of automated program verification without requiring
  99. modification or recompilation of the source code.
  100. The language is also an
  101. excellent vehicle for preserving a test suite for later regression testing.
  102. .PP
  103. The debugger may be customized by its users; standard
  104. functions may be modified or extended to suit a particular application
  105. or preference.
  106. For example, the kernel developers in our group require a
  107. command set supporting assembler-level debugging while the application
  108. programmers prefer source-level functionality.
  109. Although the default library is biased toward assembler-level debugging,
  110. it is easily modified to provide a convenient source-level interface.
  111. The debugger itself does not change; the user combines primitives
  112. and existing Acid functions in different ways to
  113. implement the desired interface.
  114. .NH
  115. Related Work
  116. .PP
  117. DUEL [Gol93], an extension to
  118. .I gdb
  119. [Stal91], proposes using a high level expression evaluator to solve
  120. some of these problems. The evaluator provides iterators to loop over data
  121. structures and conditionals to control evaluation of expressions.
  122. The author shows that complex state queries can be formulated
  123. by combining concise expressions but this only addresses part of the problem.
  124. A program is a dynamic entity; questions asked when the program is in
  125. a static state are meaningful only after the program has been `caught' in
  126. that state. The framework for manipulating the program is still as
  127. primitive as the underlying debugger. While DUEL provides a means to
  128. probe data structures it entirely neglects the most beneficial aspect
  129. of debugging languages: the ability to control processes. Acid is structured
  130. around a thread of control that passes between the interpreter and the
  131. target program.
  132. .PP
  133. The NeD debugger [May92] is a set of extensions to TCL [Ous90] that provide
  134. debugging primitives. The resulting language, NeDtcl, is used to implement
  135. a portable interface between a conventional debugger, pdb [May90], and
  136. a server that executes NeDtcl programs operating on the target program.
  137. Execution of the NeDtcl programs implements the debugging primitives
  138. that pdb expects.
  139. NeD is targeted at multi-process debugging across a network,
  140. and proves the flexibility of a language as a means of
  141. communication between debugging tools. Whereas NeD provides an interface
  142. between a conventional debugger and the process it debugs, Acid is the
  143. debugger itself. While NeD has some of the ideas
  144. found in Acid it is targeted toward a different purpose. Acid seeks to
  145. integrate the manipulation of a program's resources into the debugger
  146. while NeD provides a flexible interconnect between components of
  147. the debugging environment. The choice of TCL is appropriate for its use
  148. in NeD but is not suitable for Acid. Acid relies on the coupling of the type
  149. system with expression evaluation, which are the root of its design,
  150. to provide the debugging primitives.
  151. .PP
  152. Dalek [Ols90] is an event based language extension to gdb. State transitions
  153. in the target program cause events to be queued for processing by the
  154. debugging language.
  155. .PP
  156. Acid has many of the advantages of same process or
  157. .I local
  158. .I agent
  159. debuggers, like Parasight [Aral], without the need for dynamic linking or
  160. shared memory.
  161. Acid improves on the ideas of these other systems by completely integrating
  162. all aspects of the debugging process into the language environment. Of
  163. particular importance is the relationship between Acid variables,
  164. program symbols, source code, registers and type information. This
  165. integration is made possible by the design of the Acid language.
  166. .PP
  167. Interpreted languages such as Lisp and Smalltalk are able to provide
  168. richer debugging environments through more complete information than
  169. their compiled counterparts. Acid is a means to gather and represent
  170. similar information about compiled programs through cooperation
  171. with the compilation tools and library implementers.
  172. .NH
  173. Acid the Language
  174. .PP
  175. Acid is a small interpreted language targeted to its debugging task.
  176. It focuses on representing program state and addressing data rather than
  177. expressing complex computations. Program state is
  178. .I addressable
  179. from an Acid program.
  180. In addition to parsing and executing expressions and providing
  181. an architecture-independent interface to the target process,
  182. the interpreter supplies a mark-and-scan garbage collector
  183. to manage storage.
  184. .PP
  185. Every Acid session begins with the loading of the Acid libraries.
  186. These libraries contain functions, written in Acid, that provide
  187. a standard debugging environment including breakpoint management,
  188. stepping by instruction or statement, stack tracing, and
  189. access to variables, memory, and registers.
  190. The library contains 600 lines of Acid code and provides
  191. functionality similar to
  192. .I dbx .
  193. Following the loading of the system library, Acid loads
  194. user-specified libraries; this load sequence allows the
  195. user to augment or override the standard commands
  196. to customize the debugging environment. When all libraries
  197. are loaded, Acid issues an interactive prompt and begins
  198. evaluating expressions entered by the user. The Acid `commands'
  199. are actually invocations of builtin primitives or previously defined
  200. Acid functions. Acid evaluates each expression as it is entered and
  201. prints the result.
  202. .NH
  203. Types and Variables
  204. .PP
  205. Acid variables are of four basic types:
  206. .I integer ,
  207. .I string ,
  208. .I float ,
  209. and
  210. .I list .
  211. The type of a variable is inferred by the type of the right-hand side of
  212. an assignment expression.
  213. Many of the operators can be applied to more than
  214. one type; for these operators the action of the operator is determined
  215. by the type of its operands.
  216. For example,
  217. the
  218. .CW +
  219. operator adds
  220. .I integer
  221. and
  222. .I float
  223. operands, and concatenates
  224. .I string
  225. and
  226. .I list
  227. operands.
  228. Lists are the only complex type in Acid; there are no arrays, structures
  229. or pointers. Operators provide
  230. .CW head ,
  231. .CW tail ,
  232. .CW append
  233. and
  234. .CW delete
  235. operations.
  236. Lists can also be indexed like arrays.
  237. .PP
  238. Acid has two levels of scope: global and local.
  239. Function parameters and variables declared in a function body
  240. using the
  241. .CW local
  242. keyword are created at entry to the function and
  243. exist for the lifetime of a function.
  244. Global variables are created by assignment and need not be declared.
  245. All variables and functions in the program
  246. being debugged are entered in the Acid symbol table as global
  247. variables during Acid initialization.
  248. Conflicting variable names are resolved by prefixing enough `$' characters
  249. to make them unique.
  250. Syntactically, Acid variables and target program
  251. symbols are referenced identically.
  252. However, the variables are managed differently in the Acid
  253. symbol table and the user must be aware of this distinction.
  254. The value of an Acid variable is stored in the symbol
  255. table; a reference returns the value.
  256. The symbol table entry for a variable or function in the target
  257. program contains the address of that symbol in the image
  258. of the program. Thus, the value of a program variable is
  259. accessed by indirect reference through the Acid
  260. variable that has the same name; the value of an Acid variable is the
  261. address of the corresponding program variable.
  262. .NH
  263. Control Flow
  264. .PP
  265. The
  266. .CW while
  267. and
  268. .CW loop
  269. statements implement looping.
  270. The former
  271. is similar to the same statement in C.
  272. The latter evaluates starting and ending expressions yielding
  273. integers and iterates while an incrementing loop index
  274. is within the bounds of those expressions.
  275. .P1
  276. acid: i = 0; loop 1,5 do print(i=i+1)
  277. 0x00000001
  278. 0x00000002
  279. 0x00000003
  280. 0x00000004
  281. 0x00000005
  282. acid:
  283. .P2
  284. The traditional
  285. .CW if-then-else
  286. statement implements conditional execution.
  287. .NH
  288. Addressing
  289. .PP
  290. Two indirection operators allow Acid to access values in
  291. the program being debugged.
  292. The
  293. .CW *
  294. operator fetches a value from the memory image of an
  295. executing process;
  296. the
  297. .CW @
  298. operator fetches a value from the text file of the process.
  299. When either operator appears on the left side of an assignment, the value
  300. is written rather than read.
  301. .PP
  302. The indirection operator must know the size of the object
  303. referenced by a variable.
  304. The Plan 9 compilers neglect to include this
  305. information in the program symbol table, so Acid cannot
  306. derive this information implicitly.
  307. Instead Acid variables have formats.
  308. The format is a code
  309. letter specifying the printing style and the effect of some of the
  310. operators on that variable.
  311. The indirection operators look at the format code to determine the
  312. number of bytes to read or write.
  313. The format codes are derived from the format letters used by
  314. .I db .
  315. By default, symbol table variables and numeric constants
  316. are assigned the format code
  317. .CW 'X'
  318. which specifies 32-bit hexadecimal.
  319. Printing such a variable yields output of the form
  320. .CW 0x00123456 .
  321. An indirect reference through the variable fetches 32 bits
  322. of data at the address indicated by the variable.
  323. Other formats specify various data types, for example
  324. .CW i
  325. an instruction,
  326. .CW D
  327. a signed 32 bit decimal,
  328. .CW s
  329. a null-terminated string.
  330. The
  331. .CW fmt
  332. function
  333. allows the user to change the format code of a variable
  334. to control the printing format and
  335. operator side effects.
  336. This function evaluates the expression supplied as the first
  337. argument, attaches the format code supplied as the second
  338. argument to the result and returns that value.
  339. If the result is assigned to a variable,
  340. the new format code applies to
  341. that variable. For convenience, Acid provides the
  342. .CW \e
  343. operator as a shorthand infix form of
  344. .CW fmt .
  345. For example:
  346. .P1
  347. acid: x=10
  348. acid: x // print x in hex
  349. 0x0000000a
  350. acid: x = fmt(x, 'D') // make x type decimal
  351. acid: print(x, fmt(x, 'X'), x\eX) // print x in decimal & hex
  352. 10 0x0000000a 0x0000000a
  353. acid: x // print x in decimal
  354. 10
  355. acid: x\eo // print x in octal
  356. 000000000012
  357. .P2
  358. The
  359. .CW ++
  360. and
  361. .CW --
  362. operators increment or decrement a variable by an amount
  363. determined by its format code. Some formats imply a non-fixed size.
  364. For example, the
  365. .CW i
  366. format code disassembles an instruction into a string.
  367. On a 68020, which has variable length instructions:
  368. .P1
  369. acid: p=main\ei // p=addr(main), type INST
  370. acid: loop 1,5 do print(p\eX, @p++) // disassemble 5 instr's
  371. 0x0000222e LEA 0xffffe948(A7),A7
  372. 0x00002232 MOVL s+0x4(A7),A2
  373. 0x00002236 PEA 0x2f($0)
  374. 0x0000223a MOVL A2,-(A7)
  375. 0x0000223c BSR utfrrune
  376. acid:
  377. .P2
  378. Here,
  379. .CW main
  380. is the address of the function of the same name in the program under test.
  381. The loop retrieves the five instructions beginning at that address and
  382. then prints the address and the assembly language representation of each.
  383. Notice that the stride of the increment operator varies with the size of
  384. the instruction: the
  385. .CW MOVL
  386. at
  387. .CW 0x0000223a
  388. is a two byte instruction while all others are four bytes long.
  389. .PP
  390. Registers are treated as normal program variables referenced
  391. by their symbolic assembler language names.
  392. When a
  393. process stops, the register set is saved by the kernel
  394. at a known virtual address in the process memory map.
  395. The Acid variables associated with the registers point
  396. to the saved values and the
  397. .CW *
  398. indirection operator can then be used to read and write the register set.
  399. Since the registers are accessed via Acid variables they may
  400. be used in arbitrary expressions.
  401. .P1
  402. acid: PC // addr of saved PC
  403. 0xc0000f60
  404. acid: *PC
  405. 0x0000623c // contents of PC
  406. acid: *PC\ea
  407. main
  408. acid: *R1=10 // modify R1
  409. acid: asm(*PC+4) // disassemble @ PC+4
  410. main+0x4 0x00006240 MOVW R31,0x0(R29)
  411. main+0x8 0x00006244 MOVW $setR30(SB),R30
  412. main+0x10 0x0000624c MOVW R1,_clock(SB)
  413. .P2
  414. Here, the saved
  415. .CW PC
  416. is stored at address
  417. .CW 0xc0000f60 ;
  418. its current content is
  419. .CW 0x0000623c .
  420. The
  421. .CW a ' `
  422. format code converts this value to a string specifying
  423. the address as an offset beyond the nearest symbol.
  424. After setting the value of register
  425. .CW 1 ,
  426. the example uses the
  427. .CW asm
  428. command to disassemble a short section of code beginning
  429. at four bytes beyond the current value of the
  430. .CW PC .
  431. .NH
  432. Process Interface
  433. .PP
  434. A program executing under Acid is monitored through the
  435. .I proc
  436. file system interface provided by Plan 9.
  437. Textual messages written to the
  438. .CW ctl
  439. file control the execution of the process.
  440. For example writing
  441. .CW waitstop
  442. to the control file causes the write to block until the target
  443. process enters the kernel and is stopped. When the process is stopped
  444. the write completes. The
  445. .CW startstop
  446. message starts the target process and then does a
  447. .CW waitstop
  448. action.
  449. Synchronization between the debugger and the target process is determined
  450. by the actions of the various messages. Some operate asynchronously to the
  451. target process and always complete immediately, others block until the
  452. action completes. The asynchronous messages allow Acid to control
  453. several processes simultaneously.
  454. .PP
  455. The interpreter has builtin functions named after each of the control
  456. messages. The functions take a process id as argument.
  457. Any time a control message causes the program to execute instructions
  458. the interpreter performs two actions when the control operation has completed.
  459. The Acid variables pointing at the register set are fixed up to point
  460. at the saved registers, and then
  461. the user defined function
  462. .CW stopped
  463. is executed.
  464. The
  465. .CW stopped
  466. function may print the current address,
  467. line of source or instruction and return to interactive mode. Alternatively
  468. it may traverse a complex data structure, gather statistics and then set
  469. the program running again.
  470. .PP
  471. Several Acid variables are maintained by the debugger rather than the
  472. programmer.
  473. These variables allow generic Acid code to deal with the current process,
  474. architecture specifics or the symbol table.
  475. The variable
  476. .CW pid
  477. is the process id of the current process Acid is debugging.
  478. The variable
  479. .CW symbols
  480. contains a list of lists where each sublist contains the symbol
  481. name, its type and the value of the symbol.
  482. The variable
  483. .CW registers
  484. contains a list of the machine-specific register names. Global symbols in the target program
  485. can be referenced directly by name from Acid. Local variables
  486. are referenced using the colon operator as \f(CWfunction:variable\fP.
  487. .NH
  488. Source Level Debugging
  489. .PP
  490. Acid provides several builtin functions to manipulate source code.
  491. The
  492. .CW file
  493. function reads a text file, inserting each line into a list.
  494. The
  495. .CW pcfile
  496. and
  497. .CW pcline
  498. functions each take an address as an argument.
  499. The first
  500. returns a string containing the name of the source file
  501. and the second returns an integer containing the line number
  502. of the source line containing the instruction at the address.
  503. .P1
  504. acid: pcfile(main) // file containing main
  505. main.c
  506. acid: pcline(main) // line # of main in source
  507. 11
  508. acid: file(pcfile(main))[pcline(main)] // print that line
  509. main(int argc, char *argv[])
  510. acid: src(*PC) // print statements nearby
  511. 9
  512. 10 void
  513. >11 main(int argc, char *argv[])
  514. 12 {
  515. 13 int a;
  516. .P2
  517. In this example, the three primitives are combined in an expression to print
  518. a line of source code associated with an address.
  519. The
  520. .CW src
  521. function prints a few lines of source
  522. around the address supplied as its argument. A companion routine,
  523. .CW Bsrc ,
  524. communicates with the external editor
  525. .CW sam .
  526. Given an address, it loads the corresponding source file into the editor
  527. and highlights the line containing the address. This simple interface
  528. is easily extended to more complex functions.
  529. For example, the
  530. .CW step
  531. function can select the current file and line in the editor
  532. each time the target program stops, giving the user a visual
  533. trace of the execution path of the program. A more complete interface
  534. allowing two way communication between Acid and the
  535. .CW acme
  536. user interface [Pike93] is under construction. A filter between the debugger
  537. and the user interface provides interpretation of results from both
  538. sides of the interface. This allows the programming environment to
  539. interact with the debugger and vice-versa, a capability missing from the
  540. .CW sam
  541. interface.
  542. The
  543. .CW src
  544. and
  545. .CW Bsrc
  546. functions are both written in Acid code using the file and line primitives.
  547. Acid provides library functions to step through source level
  548. statements and functions. Furthermore, addresses in Acid expressions can be
  549. specified by source file and line.
  550. Source code is manipulated in the Acid
  551. .I list
  552. data type.
  553. .NH
  554. The Acid Library
  555. .PP
  556. The following examples define some useful commands and
  557. illustrate the interaction of the debugger and the interpreter.
  558. .P1
  559. defn bpset(addr) // set breakpoint
  560. {
  561. if match(addr, bplist) >= 0 then
  562. print("bkpoint already set:", addr\ea, "\en");
  563. else {
  564. *fmt(addr, bpfmt) = bpinst; // plant it
  565. bplist = append bplist, addr; // add to list
  566. }
  567. }
  568. .P2
  569. The
  570. .CW bpset
  571. function plants a break point in memory. The function starts by
  572. using the
  573. .CW match
  574. builtin to
  575. search the breakpoint list to determine if a breakpoint is already
  576. set at the address.
  577. The indirection operator, controlled by the format code returned
  578. by the
  579. .CW fmt
  580. primitive, is used to plant the breakpoint in memory.
  581. The variables
  582. .CW bpfmt
  583. and
  584. .CW bpinst
  585. are Acid global variables containing the format code specifying
  586. the size of the breakpoint instruction and the breakpoint instruction
  587. itself.
  588. These
  589. variables are set by architecture-dependent library code
  590. when the debugger first attaches to the executing image.
  591. Finally the address of the breakpoint is
  592. appended to the breakpoint list,
  593. .CW bplist .
  594. .P1
  595. defn step() // single step
  596. {
  597. local lst, lpl, addr, bput;
  598. bput = 0; // sitting on bkpoint
  599. if match(*PC, bplist) >= 0 then {
  600. bput = fmt(*PC, bpfmt); // save current addr
  601. *bput = @bput; // replace it
  602. }
  603. lst = follow(*PC); // get follow set
  604. lpl = lst;
  605. while lpl do { // place breakpoints
  606. *(head lpl) = bpinst;
  607. lpl = tail lpl;
  608. }
  609. startstop(pid); // do the step
  610. while lst do { // remove breakpoints
  611. addr = fmt(head lst, bpfmt);
  612. *addr = @addr; // replace instr.
  613. lst = tail lst;
  614. }
  615. if bput != 0 then
  616. *bput = bpinst; // restore breakpoint
  617. }
  618. .P2
  619. The
  620. .CW step
  621. function executes a single assembler instruction.
  622. If the
  623. .CW PC
  624. is sitting
  625. on a breakpoint, the address and size of
  626. the breakpoint are saved.
  627. The breakpoint instruction
  628. is then removed using the
  629. .CW @
  630. operator to fetch
  631. .CW bpfmt
  632. bytes from the text file and to place it into the memory
  633. of the executing process using the
  634. .CW *
  635. operator.
  636. The
  637. .CW follow
  638. function is an Acid
  639. builtin which returns a follow-set: a list of instruction addresses which
  640. could be executed next.
  641. If the instruction stored at the
  642. .CW PC
  643. is a branch instruction, the
  644. list contains the addresses of the next instruction and
  645. the branch destination; otherwise, it contains only the
  646. address of the next instruction.
  647. The follow-set is then used to replace each possible following
  648. instruction with a breakpoint instruction. The original
  649. instructions need not be saved; they remain
  650. in their unaltered state in the text file.
  651. The
  652. .CW startstop
  653. builtin writes the `startstop' message to the
  654. .I proc
  655. control file for the process named
  656. .CW pid .
  657. The target process executes until some condition causes it to
  658. enter the kernel, in this case, the execution of a breakpoint.
  659. When the process blocks, the debugger regains control and invokes the
  660. Acid library function
  661. .CW stopped
  662. which reports the address and cause of the blockage.
  663. The
  664. .CW startstop
  665. function completes and returns to the
  666. .CW step
  667. function where
  668. the follow-set is used to replace the breakpoints placed earlier.
  669. Finally, if the address of the original
  670. .CW PC
  671. contained a breakpoint, it is replaced.
  672. .PP
  673. Notice that this approach to process control is inherently portable;
  674. the Acid code is shared by the debuggers for all architectures.
  675. Acid variables and builtin functions provide a transparent interface
  676. to architecture-dependent values and functions. Here the breakpoint
  677. value and format are referenced through Acid variables and the
  678. .CW follow
  679. primitive masks the differences in the underlying instruction set.
  680. .PP
  681. The
  682. .CW next
  683. function, similar to the
  684. .I dbx
  685. command of the same name,
  686. is a simpler example.
  687. This function steps through
  688. a single source statement but steps over function calls.
  689. .P1
  690. defn next()
  691. {
  692. local sp, bound;
  693. sp = *SP; // save starting SP
  694. bound = fnbound(*PC); // begin & end of fn.
  695. stmnt(); // step 1 statement
  696. pc = *PC;
  697. if pc >= bound[0] && pc < bound[1] then
  698. return {};
  699. while (pc<bound[0] || pc>bound[1]) && sp>=*SP do {
  700. step();
  701. pc = *PC;
  702. }
  703. src(*PC);
  704. }
  705. .P2
  706. The
  707. .CW next
  708. function
  709. starts by saving the current stack pointer in a local variable.
  710. It then uses the Acid library function
  711. .CW fnbound
  712. to return the addresses of the first and last instructions in
  713. the current function in a list.
  714. The
  715. .CW stmnt
  716. function executes a single source statement and then uses
  717. .CW src
  718. to print a few lines of source around the new
  719. .CW PC .
  720. If the new value of the
  721. .CW PC
  722. remains in the current function,
  723. .CW next
  724. returns.
  725. When the executed statement is a function call or a return
  726. from a function, the new value of the
  727. .CW PC
  728. is outside the bounds calculated by
  729. .CW fnbound
  730. and the test of the
  731. .CW while
  732. loop is evaluated.
  733. If the statement was a return, the new value of the stack pointer
  734. is greater than the original value and the loop completes without
  735. execution.
  736. Otherwise, the loop is entered and instructions are continually
  737. executed until the value of the
  738. .CW PC
  739. is between the bounds calculated earlier. At that point, execution
  740. ceases and a few lines of source in the vicinity of the
  741. .CW PC
  742. are printed.
  743. .PP
  744. Acid provides concise and elegant expression for control and
  745. manipulation of target programs. These examples demonstrate how a
  746. few well-chosen primitives can be combined to create a rich debugging environment.
  747. .NH
  748. Dealing With Multiple Architectures
  749. .PP
  750. A single binary of Acid may be used to debug a program running on any
  751. of the five processor architectures supported by Plan 9. For example,
  752. Plan 9 allows a user on a MIPS to import the
  753. .I proc
  754. file system from an i486-based PC and remotely debug a program executing
  755. on that processor.
  756. .PP
  757. Two levels of abstraction provide this architecture independence.
  758. On the lowest level, a Plan 9 library supplies functions to
  759. decode the file header of the program being debugged and
  760. select a table of system parameters
  761. and a jump vector of architecture-dependent
  762. functions based on the magic number.
  763. Among these functions are byte-order-independent
  764. access to memory and text files, stack manipulation, disassembly,
  765. and floating point number interpretation.
  766. The second level of abstraction is supplied by Acid.
  767. It consists of primitives and approximately 200 lines
  768. of architecture-dependent Acid library code that interface the
  769. interpreter to the architecture-dependent library.
  770. This layer performs functions such as mapping register names to
  771. memory locations, supplying breakpoint values and sizes,
  772. and converting processor specific data to Acid data types.
  773. An example of the latter is the stack trace function
  774. .CW strace ,
  775. which uses the stack traversal functions in the
  776. architecture-dependent library to construct a list of lists describing
  777. the context of a process. The first level of list selects
  778. each function in the trace; subordinate lists contain the
  779. names and values of parameters and local variables of
  780. the functions. Acid commands and library functions that
  781. manipulate and display process state information operate
  782. on the list representation and are independent of the
  783. underlying architecture.
  784. .NH
  785. Alef Runtime
  786. .PP
  787. Alef is a concurrent programming language,
  788. designed specifically for systems programming, which supports both
  789. shared variable and message passing paradigms.
  790. Alef borrows the C expression syntax but implements
  791. a substantially different type system.
  792. The language provides a rich set of
  793. exception handling, process management, and synchronization
  794. primitives, which rely on a runtime system.
  795. Alef program bugs are often deadlocks, synchronization failures,
  796. or non-termination caused by locks being held incorrectly.
  797. In such cases, a process stalls deep
  798. in the runtime code and it is clearly
  799. unreasonable to expect a programmer using the language
  800. to understand the detailed
  801. internal semantics of the runtime support functions.
  802. .PP
  803. Instead, there is an Alef support library, coded in Acid, that
  804. allows the programmer to interpret the program state in terms of
  805. Alef operations. Consider the example of a multi-process program
  806. stalling because of improper synchronization. A stack trace of
  807. the program indicates that it is waiting for an event in some
  808. obscure Alef runtime
  809. synchronization function.
  810. The function itself is irrelevant to the
  811. programmer; of greater importance is the identity of the
  812. unfulfilled event.
  813. Commands in the Alef support library decode
  814. the runtime data structures and program state to report the cause
  815. of the blockage in terms of the high-level operations available to
  816. the Alef programmer.
  817. Here, the Acid language acts
  818. as a communications medium between Alef implementer and Alef user.
  819. .NH
  820. Parallel Debugging
  821. .PP
  822. The central issue in parallel debugging is how the debugger is
  823. multiplexed between the processes comprising
  824. the program.
  825. Acid has no intrinsic model of process partitioning; it
  826. only assumes that parallel programs share a symbol table,
  827. though they need not share memory.
  828. The
  829. .CW setproc
  830. primitive attaches the debugger to a running process
  831. associated with the process ID supplied as its argument
  832. and assigns that value to the global variable
  833. .CW pid ,
  834. thereby allowing simple rotation among a group of processes.
  835. Further, the stack trace primitive is driven by parameters
  836. specifying a unique process context, so it is possible to
  837. examine the state of cooperating processes without switching
  838. the debugger focus from the process of interest.
  839. Since Acid is inherently extensible and capable of
  840. dynamic interaction with subordinate processes, the
  841. programmer can define Acid commands to detect and control
  842. complex interactions between processes.
  843. In short, the programmer is free to specify how the debugger reacts
  844. to events generated in specific threads of the program.
  845. .PP
  846. The support for parallel debugging in Acid depends on a crucial kernel
  847. modification: when the text segment of a program is written (usually to
  848. place a breakpoint), the segment is cloned to prevent other threads
  849. from encountering the breakpoint. Although this incurs a slight performance
  850. penalty, it is of little importance while debugging.
  851. .NH
  852. Communication Between Tools
  853. .PP
  854. The Plan 9 Alef and C compilers do not
  855. embed detailed type information in the symbol table of an
  856. executable file.
  857. However, they do accept a command line option causing them to
  858. emit descriptions of complex data types
  859. (e.g., aggregates and abstract data types)
  860. to an auxiliary file.
  861. The vehicle for expressing this information is Acid source code.
  862. When an Acid debugging session is
  863. subsequently started, that file is loaded with the other Acid libraries.
  864. .PP
  865. For each complex object in the program the compiler generates
  866. three pieces of Acid code.
  867. The first is a table describing the size and offset of each
  868. member of the complex data type. Following is an Acid function,
  869. named the same as the object, that formats and prints each member.
  870. Finally, Acid declarations associate the
  871. Alef or C program variables of a type with the functions
  872. to print them.
  873. The three forms of declaration are shown in the following example:
  874. .P1
  875. struct Bitmap {
  876. Rectangle 0 r;
  877. Rectangle 16 clipr;
  878. 'D' 32 ldepth;
  879. 'D' 36 id;
  880. 'X' 40 cache;
  881. };
  882. .P2
  883. .P1
  884. defn
  885. Bitmap(addr) {
  886. complex Bitmap addr;
  887. print("Rectangle r {\en");
  888. Rectangle(addr.r);
  889. print("}\en");
  890. print("Rectangle clipr {\en");
  891. Rectangle(addr.clipr);
  892. print("}\en");
  893. print(" ldepth ", addr.ldepth, "\en");
  894. print(" id ", addr.id, "\en");
  895. print(" cache ", addr.cache, "\en");
  896. };
  897. complex Bitmap darkgrey;
  898. complex Bitmap Window_settag:b;
  899. .P2
  900. The
  901. .CW struct
  902. declaration specifies decoding instructions for the complex type named
  903. .CW Bitmap .
  904. Although the syntax is superficially similar to a C structure declaration,
  905. the semantics differ markedly: the C declaration specifies a layout, while
  906. the Acid declaration tells how to decode it.
  907. The declaration specifies a type, an offset, and name for each
  908. member of the complex object. The type is either the name of another
  909. complex declaration, for example,
  910. .CW Rectangle ,
  911. or a format code.
  912. The offset is the number of bytes from the start
  913. of the object to the member
  914. and the name is the member's name in the Alef or C declaration.
  915. This type description is a close match for C and Alef, but is simple enough
  916. to be language independent.
  917. .PP
  918. The
  919. .CW Bitmap
  920. function expects the address of a
  921. .CW Bitmap
  922. as its only argument.
  923. It uses the decoding information contained in the
  924. .CW Bitmap
  925. structure declaration to extract, format, and print the
  926. value of each member of the complex object pointed to by
  927. the argument.
  928. The Alef compiler emits code to call other Acid functions
  929. where a member is another complex type; here,
  930. .CW Bitmap
  931. calls
  932. .CW Rectangle
  933. to print its contents.
  934. .PP
  935. The
  936. .CW complex
  937. declarations associate Alef variables with complex types.
  938. In the example,
  939. .CW darkgrey
  940. is the name of a global variable of type
  941. .CW Bitmap
  942. in the program being debugged.
  943. Whenever the name
  944. .CW darkgrey
  945. is evaluated by Acid, it automatically calls the
  946. .CW Bitmap
  947. function with the address of
  948. .CW darkgrey
  949. as the argument.
  950. The second
  951. .CW complex
  952. declaration associates a local variable or parameter named
  953. .CW b
  954. in function
  955. .CW Window_settag
  956. with the
  957. .CW Bitmap
  958. complex data type.
  959. .PP
  960. Acid borrows the C operators
  961. .CW .
  962. and
  963. .CW ->
  964. to access the decoding parameters of a member of a complex type.
  965. Although this representation is sufficiently general for describing
  966. the decoding of both C and Alef complex data types, it may
  967. prove too restrictive for target languages with more complicated
  968. type systems.
  969. Further, the assumption that the compiler can select the proper
  970. Acid format code for each basic type in the language is somewhat
  971. naive. For example, when a member of a complex type is a pointer,
  972. it is assigned a hexadecimal type code; integer members are always
  973. assigned a decimal type code.
  974. This heuristic proves inaccurate when an integer field is a
  975. bit mask or set of bit flags which are more appropriately displayed
  976. in hexadecimal or octal.
  977. .NH
  978. Code Verification
  979. .PP
  980. Acid's ability to interact dynamically with
  981. an executing program allows passive test and
  982. verification of the target program. For example,
  983. a common concern is leak detection in programs using
  984. .CW malloc .
  985. Of interest are two items: finding memory that was allocated
  986. but never freed and detecting bad pointers passed to
  987. .CW free .
  988. An auxiliary Acid library contains Acid functions to
  989. monitor the execution of a program and detect these
  990. faults, either as they happen or in the automated
  991. post-mortem analysis of the memory arena.
  992. In the following example, the
  993. .CW sort
  994. command is run under the control of the
  995. Acid memory leak library.
  996. .P1
  997. helix% acid -l malloc /bin/sort
  998. /bin/sort: mips plan 9 executable
  999. /lib/acid/port
  1000. /lib/acid/mips
  1001. /lib/acid/malloc
  1002. acid: go()
  1003. now
  1004. is
  1005. the
  1006. time
  1007. <ctrl-d>
  1008. is
  1009. now
  1010. the
  1011. time
  1012. 27680 : breakpoint _exits+0x4 MOVW $0x8,R1
  1013. acid:
  1014. .P2
  1015. The
  1016. .CW go
  1017. command creates a process and plants
  1018. breakpoints at the entry to
  1019. .CW malloc
  1020. and
  1021. .CW free .
  1022. The program is then started and continues until it
  1023. exits or stops. If the reason for stopping is anything
  1024. other than the breakpoints in
  1025. .CW malloc
  1026. and
  1027. .CW free ,
  1028. Acid prints the usual status information and returns to the
  1029. interactive prompt.
  1030. .PP
  1031. When the process stops on entering
  1032. .CW malloc ,
  1033. the debugger must capture and save the address that
  1034. .CW malloc
  1035. will return.
  1036. After saving a stack
  1037. trace so the calling routine can be identified, it places
  1038. a breakpoint at the return address and restarts the program.
  1039. When
  1040. .CW malloc
  1041. returns, the breakpoint stops the program,
  1042. allowing the debugger
  1043. to grab the address of the new memory block from the return register.
  1044. The address and stack trace are added to the list of outstanding
  1045. memory blocks, the breakpoint is removed from the return point, and
  1046. the process is restarted.
  1047. .PP
  1048. When the process stops at the beginning of
  1049. .CW free ,
  1050. the memory address supplied as the argument is compared to the list
  1051. of outstanding memory blocks. If it is not found an error message
  1052. and a stack trace of the call is reported; otherwise, the
  1053. address is deleted from the list.
  1054. .PP
  1055. When the program exits, the list of outstanding memory blocks contains
  1056. the addresses of all blocks that were allocated but never freed.
  1057. The
  1058. .CW leak
  1059. library function traverses the list producing a report describing
  1060. the allocated blocks.
  1061. .P1 1m
  1062. acid: leak()
  1063. Lost a total of 524288 bytes from:
  1064. malloc() malloc.c:32 called from dofile+0xe8 sort.c:217
  1065. dofile() sort.c:190 called from main+0xac sort.c:161
  1066. main() sort.c:128 called from _main+0x20 main9.s:10
  1067. Lost a total of 64 bytes from:
  1068. malloc() malloc.c:32 called from newline+0xfc sort.c:280
  1069. newline() sort.c:248 called from dofile+0x110 sort.c:222
  1070. dofile() sort.c:190 called from main+0xac sort.c:161
  1071. main() sort.c:128 called from _main+0x20 main9.s:10
  1072. Lost a total of 64 bytes from:
  1073. malloc() malloc.c:32 called from realloc+0x14 malloc.c:129
  1074. realloc() malloc.c:123 called from bldkey+0x358 sort.c:1388
  1075. buildkey() sort.c:1345 called from newline+0x150 sort.c:285
  1076. newline() sort.c:248 called from dofile+0x110 sort.c:222
  1077. dofile() sort.c:190 called from main+0xac sort.c:161
  1078. main() sort.c:128 called from _main+0x20 main9.s:10
  1079. acid: refs()
  1080. data...bss...stack...
  1081. acid: leak()
  1082. acid:
  1083. .P2
  1084. The presence of a block in the allocation list does not imply
  1085. it is there because of a leak; for instance, it may have been
  1086. in use when the program terminated.
  1087. The
  1088. .CW refs()
  1089. library function scans the
  1090. .I data ,
  1091. .I bss ,
  1092. and
  1093. .I stack
  1094. segments of the process looking for pointers
  1095. into the allocated blocks. When one is found, the block is deleted from
  1096. the outstanding block list.
  1097. The
  1098. .CW leak
  1099. function is used again to report the
  1100. blocks remaining allocated and unreferenced.
  1101. This strategy proves effective in detecting
  1102. disconnected (but non-circular) data structures.
  1103. .PP
  1104. The leak detection process is entirely passive.
  1105. The program is not
  1106. specially compiled and the source code is not required.
  1107. As with the Acid support functions for the Alef runtime environment,
  1108. the author of the library routines has encapsulated the
  1109. functionality of the library interface
  1110. in Acid code.
  1111. Any programmer may then check a program's use of the
  1112. library routines without knowledge of either implementation.
  1113. The performance impact of running leak detection is great
  1114. (about 10 times slower),
  1115. but it has not prevented interactive programs like
  1116. .CW sam
  1117. and the
  1118. .CW 8½
  1119. window system from being tested.
  1120. .NH
  1121. Code Coverage
  1122. .PP
  1123. Another common component of software test uses
  1124. .I coverage
  1125. analysis.
  1126. The purpose of the test is to determine which paths through the code have
  1127. not been executed while running the test suite.
  1128. This is usually
  1129. performed by a combination of compiler support and a reporting tool run
  1130. on the output generated by statements compiled into the program.
  1131. The compiler emits code that
  1132. logs the progress of the program as it executes basic blocks and writes the
  1133. results to a file. The file is then processed by the reporting tool
  1134. to determine which basic blocks have not been executed.
  1135. .PP
  1136. Acid can perform the same function in a language independent manner without
  1137. modifying the source, object or binary of the program. The following example
  1138. shows
  1139. .CW ls
  1140. being run under the control of the Acid coverage library.
  1141. .P1
  1142. philw-helix% acid -l coverage /bin/ls
  1143. /bin/ls: mips plan 9 executable
  1144. /lib/acid/port
  1145. /lib/acid/mips
  1146. /lib/acid/coverage
  1147. acid: coverage()
  1148. acid
  1149. newstime
  1150. profile
  1151. tel
  1152. wintool
  1153. 2: (error) msg: pid=11419 startstop: process exited
  1154. acid: analyse(ls)
  1155. ls.c:102,105
  1156. 102: return 1;
  1157. 103: }
  1158. 104: if(db[0].qid.path&CHDIR && dflag==0){
  1159. 105: output();
  1160. ls.c:122,126
  1161. 122: memmove(dirbuf+ndir, db, sizeof(Dir));
  1162. 123: dirbuf[ndir].prefix = 0;
  1163. 124: p = utfrrune(s, '/');
  1164. 125: if(p){
  1165. 126: dirbuf[ndir].prefix = s;
  1166. .P2
  1167. The
  1168. .CW coverage
  1169. function begins by looping through the text segment placing
  1170. breakpoints at the entry to each basic block. The start of each basic
  1171. block is found using the Acid builtin function
  1172. .CW follow .
  1173. If the list generated by
  1174. .CW follow
  1175. contains more than one
  1176. element, then the addresses mark the start of basic blocks. A breakpoint
  1177. is placed at each address to detect entry into the block. If the result
  1178. of
  1179. .CW follow
  1180. is a single address then no action is taken, and the next address is
  1181. considered. Acid maintains a list of
  1182. breakpoints already in place and avoids placing duplicates (an address may be
  1183. the destination of several branches).
  1184. .PP
  1185. After placing the breakpoints the program is set running.
  1186. Each time a breakpoint is encountered
  1187. Acid deletes the address from the breakpoint list, removes the breakpoint
  1188. from memory and then restarts the program.
  1189. At any instant the breakpoint list contains the addresses of basic blocks
  1190. which have not been executed.
  1191. The
  1192. .CW analyse
  1193. function reports the lines of source code bounded by basic blocks
  1194. whose addresses are have not been deleted from the breakpoint list.
  1195. These are the basic blocks which have not been executed.
  1196. Program performance is almost unaffected since each breakpoint is executed
  1197. only once and then removed.
  1198. .PP
  1199. The library contains a total of 128 lines of Acid code.
  1200. An obvious extension of this algorithm could be used to provide basic block
  1201. profiling.
  1202. .NH
  1203. Conclusion
  1204. .PP
  1205. Acid has two areas of weakness. As with
  1206. other language-based tools like
  1207. .I awk ,
  1208. a programmer must learn yet another language to step beyond the normal
  1209. debugging functions and use the full power of the debugger.
  1210. Second, the command line interface supplied by the
  1211. .I yacc
  1212. parser is inordinately clumsy.
  1213. Part of the problem relates directly to the use of
  1214. .I yacc
  1215. and could be circumvented with a custom parser.
  1216. However, structural problems would remain: Acid often requires
  1217. too much typing to execute a simple
  1218. command.
  1219. A debugger should prostitute itself to its users, doing whatever
  1220. is wanted with a minimum of encouragement; commands should be
  1221. concise and obvious. The language interface is more consistent than
  1222. an ad hoc command interface but is clumsy to use.
  1223. Most of these problems are addressed by an Acme interface
  1224. which is under construction. This should provide the best of
  1225. both worlds: graphical debugging and access to the underlying acid
  1226. language when required.
  1227. .PP
  1228. The name space clash between Acid variables, keywords, program variables,
  1229. and functions is unavoidable.
  1230. Although it rarely affects a debugging session, it is annoying
  1231. when it happens and is sometimes difficult to circumvent.
  1232. The current renaming scheme
  1233. is too crude; the new names are too hard to remember.
  1234. .PP
  1235. Acid has proved to be a powerful tool whose applications
  1236. have exceeded expectations.
  1237. Of its strengths, portability, extensibility and parallel debugging support
  1238. were by design and provide the expected utility.
  1239. In retrospect,
  1240. its use as a tool for code test and verification and as
  1241. a medium for communicating type information and encapsulating
  1242. interfaces has provided unanticipated benefits and altered our
  1243. view of the debugging process.
  1244. .NH
  1245. Acknowledgments
  1246. .PP
  1247. Bob Flandrena was the first user and helped prepare the paper.
  1248. Rob Pike endured three buggy Alef compilers and a new debugger
  1249. in a single sitting.
  1250. .NH
  1251. References
  1252. .LP
  1253. [Pike90] R. Pike, D. Presotto, K. Thompson, H. Trickey,
  1254. ``Plan 9 from Bell Labs'',
  1255. .I
  1256. UKUUG Proc. of the Summer 1990 Conf.,
  1257. .R
  1258. London, England,
  1259. 1990,
  1260. reprinted, in a different form, in this volume.
  1261. .LP
  1262. [Gol93] M. Golan, D. Hanson,
  1263. ``DUEL -- A Very High-Level Debugging Language'',
  1264. .I
  1265. USENIX Proc. of the Winter 1993 Conf.,
  1266. .R
  1267. San Diego, CA,
  1268. 1993.
  1269. .LP
  1270. [Lin90] M. A. Linton,
  1271. ``The Evolution of DBX'',
  1272. .I
  1273. USENIX Proc. of the Summer 1990 Conf.,
  1274. .R
  1275. Anaheim, CA,
  1276. 1990.
  1277. .LP
  1278. [Stal91] R. M. Stallman, R. H. Pesch,
  1279. ``Using GDB: A guide to the GNU source level debugger'',
  1280. Technical Report, Free Software Foundation,
  1281. Cambridge, MA,
  1282. 1991.
  1283. .LP
  1284. [Win93] P. Winterbottom,
  1285. ``Alef reference Manual'',
  1286. this volume.
  1287. .LP
  1288. [Pike93] Rob Pike,
  1289. ``Acme: A User Interface for Programmers'',
  1290. .I
  1291. USENIX Proc. of the Winter 1994 Conf.,
  1292. .R
  1293. San Francisco, CA,
  1294. reprinted in this volume.
  1295. .LP
  1296. [Ols90] Ronald A. Olsson, Richard H. Crawford, and W. Wilson Ho,
  1297. ``Dalek: A GNU, improved programmable debugger'',
  1298. .I
  1299. USENIX Proc. of the Summer 1990 Conf.,
  1300. .R
  1301. Anaheim, CA.
  1302. .LP
  1303. [May92] Paul Maybee,
  1304. ``NeD: The Network Extensible Debugger''
  1305. .I
  1306. USENIX Proc. of the Summer 1992 Conf.,
  1307. .R
  1308. San Antonio, TX.
  1309. .LP
  1310. [Aral] Ziya Aral, Ilya Gertner, and Greg Schaffer,
  1311. ``Efficient debugging primitives for multiprocessors'',
  1312. .I
  1313. Proceedings of the Third International Conference on Architectural
  1314. Support for Programming Languages and Operating Systems,
  1315. .R
  1316. SIGPLAN notices Nr. 22, May 1989.