harvey/sys/doc/sam/sam.ms

.Vx 17 11 November 87 1 32 "ROB PIKE" "THE TEXT EDITOR SAM"
.ds DY "31 May 1987
.ds DR "Revised 1 July 1987
.de CW		\" puts first arg in CW font, same as UL; maintains font
\%\&\\$3\f(CW\\$1\fP\&\\$2
..
.de Cs
.br
.fi
.ft 2
.ps -2
.vs -2
..
.de Ce
.br
.nf
.ft 1
.ps
.vs
.sp
..
.TL
The Text Editor \&\f(CWsam\fP
.AU
Rob Pike
rob@plan9.bell-labs.com
.AB
.LP
.CW Sam
is an interactive multi-file text editor intended for
bitmap displays.
A textual command language
supplements the mouse-driven, cut-and-paste interface
to make complex or
repetitive editing tasks easy to specify.
The language is characterized by the composition of regular expressions
to describe the structure of the text being modified.
The treatment of files as a database, with changes logged
as atomic transactions, guides the implementation and
makes a general `undo' mechanism straightforward.
.PP
.CW Sam
is implemented as two processes connected by a low-bandwidth stream,
one process handling the display and the other the editing
algorithms.  Therefore it can run with the display process
in a bitmap terminal and the editor on a local host,
with both processes on a bitmap-equipped host, or with
the display process in the terminal and the editor in a
remote host.
By suppressing the display process,
it can even run without a bitmap terminal.
.PP
This paper is reprinted from Software\(emPractice and Experience,
Vol 17, number 11, pp. 813-845, November 1987.
The paper has not been updated for the Plan 9 manuals.  Although
.CW Sam
has not changed much since the paper was written, the system around it certainly has.
Nonetheless, the description here still stands as the best introduction to the editor.
.AE
.SH
Introduction
.LP
.CW Sam
is an interactive text editor that combines cut-and-paste interactive editing with
an unusual command language based on the composition of regular expressions.
It is written as two programs: one, the `host part,' runs on a UNIX system
and implements the command language and provides file access; the other, the
`terminal part,' runs asynchronously
on a machine with a mouse and bitmap display
and supports the display and interactive editing.
The host part may be even run in isolation on an ordinary terminal
to edit text using the command
language, much like a traditional line editor,
without assistance from a mouse or display.
Most often,
the terminal part runs on a Blit\u\s-4\&1\s+4\d terminal
(actually on a Teletype DMD 5620, the production version of the Blit), whose
host connection is an ordinary 9600 bps RS232 link;
on the SUN computer the host and display processes run on a single machine,
connected by a pipe.
.PP
.CW Sam
edits uninterpreted
ASCII text.
It has no facilities for multiple fonts, graphics or tables,
unlike MacWrite,\u\s-4\&2\s+4\d Bravo,\u\s-4\&3\s+4\d Tioga\u\s-4\&4\s+4\d
or Lara.\u\s-4\&5\s+4\d
Also unlike them, it has a rich command language.
(Throughout this paper, the phrase
.I
command language
.R
refers to
textual commands; commands activated from the mouse form the
.I mouse
.I language. )
.CW Sam
developed as an editor for use by programmers, and tries to join
the styles of the UNIX text editor
.CW ed \u\s-4\&6,7\s+4\d
with that of interactive cut-and-paste editors by
providing a comfortable mouse-driven interface
to a program with a solid command language driven by regular expressions.
The command language developed more than the mouse language, and
acquired a notation for describing the structure of files
more richly than as a sequence of lines,
using a dataflow-like syntax for specifying changes.
.PP
The interactive style was influenced by
.CW jim ,\u\s-4\&1\s+4\d
an early cut-and-paste editor for the Blit, and by
.CW mux ,\u\s-4\&8\s+4\d
the Blit window system.
.CW Mux
merges the original Blit window system,
.CW mpx ,\u\s-4\&1\s+4\d
with cut-and-paste editing, forming something like a
multiplexed version of
.CW jim
that edits the output of (and input to) command sessions rather than files.
.PP
The first part of this paper describes the command language, then the mouse
language, and explains how they interact.
That is followed by a description of the implementation,
first of the host part, then of the terminal part.
A principle that influenced the design of
.CW sam
is that it should have no explicit limits, such as upper limits on
file size or line length.
A secondary consideration is that it be efficient.
To honor these two goals together requires a method for efficiently
manipulating
huge strings (files) without breaking them into lines,
perhaps while making thousands of changes
under control of the command language.
.CW Sam 's
method is to
treat the file as a transaction database, implementing changes as atomic
updates.  These updates may be unwound easily to `undo' changes.
Efficiency is achieved through a collection of caches that minimizes
disc traffic and data motion, both within the two parts of the program
and between them.
.PP
The terminal part of
.CW sam
is fairly straightforward.
More interesting is how the two halves of the editor stay
synchronized when either half may initiate a change.
This is achieved through a data structure that organizes the
communications and is maintained in parallel by both halves.
.PP
The last part of the paper chronicles the writing of
.CW sam
and discusses the lessons that were learned through its development and use.
.PP
The paper is long, but is composed largely of two papers of reasonable length:
a description of the user interface of
.CW sam
and a discussion of its implementation.
They are combined because the implementation is strongly influenced by
the user interface, and vice versa.
.SH
The Interface
.LP
.CW Sam
is a text editor for multiple files.
File names may be provided when it is invoked:
.P1
sam file1 file2 ...
.P2
and there are commands
to add new files and discard unneeded ones.
Files are not read until necessary
to complete some command.
Editing operations apply to an internal copy
made when the file is read; the UNIX file associated with the copy
is changed only by an explicit command.
To simplify the discussion, the internal copy is here called a
.I file ,
while the disc-resident original is called a
.I
disc file.
.R
.PP
.CW Sam
is usually connected to a bitmap display that presents a cut-and-paste
editor driven by the mouse.
In this mode, the command language is still available:
text typed in a special window, called the
.CW sam
.I window,
is interpreted
as commands to be executed in the current file.
Cut-and-paste editing may be used in any window \(em even in the
.CW sam
window to construct commands.
The other mode of operation, invoked by starting
.CW sam
with the option
.CW -d
(for `no download'),
does not use the mouse or bitmap display, but still permits
editing using the textual command language, even on an ordinary terminal,
interactively or from a script.
.PP
The following sections describe first the command language (under
.CW sam\ -d
and in the
.CW sam
window), and then the mouse interface.
These two languages are nearly independent, but connect through the
.I current
.I text,
described below.
.SH 2
The Command Language
.LP
A file consists of its contents, which are an array of characters
(that is, a string); the
.I name
of the associated disc file; the
.I
modified bit
.R
that states whether the contents match those of
the disc file;
and a substring of the contents, called the
.I
current text
.R
or
.I dot
(see Figures 1 and 2).
If the current text is a null string, dot falls between characters.
The
.I value
of dot is the location of the current text; the
.I contents
of dot are the characters it contains.
.CW Sam
imparts to the text no two-dimensional interpretation such as columns
or fields; text is always one-dimensional.
Even the idea of a `line' of text as understood by most UNIX programs
\(em a sequence of characters terminated by a newline character \(em
is only weakly supported.
.PP
The
.I
current file
.R
is the file to which editing commands refer.
The current text is therefore dot in the current file.
If a command doesn't explicitly name a particular file or piece of text,
the command is assumed to apply to the current text.
For the moment, ignore the presence of multiple files and consider
editing a single file.
.KF L
.BP fig1.ps 3.5i
.Cs
Figure 1. A typical
.CW sam
screen, with the editing menu presented.
The
.CW sam
(command language) window is in the middle, with file windows above and below.
(The user interface makes it easy to create these abutting windows.)
The partially obscured window is a third file window.
The uppermost window is that to which typing and mouse operations apply,
as indicated by its heavy border.
Each window has its current text highlighted in reverse video.
The
.CW sam
window's current text is the null string on the last visible line,
indicated by a vertical bar.
See also Figure 2.
.Ce
.KE
.PP
Commands have one-letter names.
Except for non-editing commands such as writing
the file to disc, most commands make some change
to the text in dot and leave dot set to the text resulting from the change.
For example, the delete command,
.CW d ,
deletes the text in dot, replacing it by the null string and setting dot
to the result.
The change command,
.CW c ,
replaces dot by text delimited by an arbitrary punctuation character,
conventionally
a slash.  Thus,
.P1
c/Peter/
.P2
replaces the text in dot by the string
.CW Peter .
Similarly,
.P1
a/Peter/
.P2
(append) adds the string after dot, and
.P1
i/Peter/
.P2
(insert) inserts before dot.
All three leave dot set to the new text,
.CW Peter .
.PP
Newlines are part of the syntax of commands:
the newline character lexically terminates a command.
Within the inserted text, however, newlines are never implicit.
But since it is often convenient to insert multiple lines of text,
.CW sam
has a special
syntax for that case:
.P1
a
some lines of text
to be inserted in the file,
terminated by a period
on a line by itself
\&.
.P2
In the one-line syntax, a newline character may be specified by a C-like
escape, so
.P1
c/\en/
.P2
replaces dot by a single newline character.
.PP
.CW Sam
also has a substitute command,
.CW s :
.P1
s/\f2expression\fP/\f2replacement\fP/
.P2
substitutes the replacement text for the first match, in dot,
of the regular expression.
Thus, if dot is the string
.CW Peter ,
the command
.P1
s/t/st/
.P2
changes it to
.CW Pester .
In general,
.CW s
is unnecessary, but it was inherited from
.CW ed
and it has some convenient variations.
For instance, the replacement text may include the matched text,
specified by
.CW & :
.P1
s/Peter/Oh, &, &, &, &!/
.P2
.PP
There are also three commands that apply programs
to text:
.P1
< \f2UNIX program\fP
.P2
replaces dot by the output of the UNIX program.
Similarly, the
.CW >
command
runs the program with dot as its standard input, and
.CW |
does both.  For example,
.P1
| sort
.P2
replaces dot by the result of applying the standard sorting utility to it.
Again, newlines have no special significance for these
.CW sam
commands.
The text acted upon and resulting from these commands is not necessarily
bounded by newlines, although for connection with UNIX programs,
newlines may be necessary to obey conventions.
.PP
One more command:
.CW p
prints the contents of dot.
Table I summarizes
.CW sam 's
commands.
.KF
.TS
center;
c s
lfCW l.
Table I. \f(CWSam\fP commands
.sp .4
.ft CW
_
.ft
.sp .4
\f1Text commands\fP	
.sp .4
_
.sp .4
a/\f2text\fP/	Append text after dot
c/\f2text\fP/	Change text in dot
i/\f2text\fP/	Insert text before dot
d	Delete text in dot
s/\f2regexp\fP/\f2text\fP/	Substitute text for match of regular expression in dot
m \f2address\fP	Move text in dot after address
t \f2address\fP	Copy text in dot after address
.sp .4
_
.sp .4
\f1Display commands\fP	
.sp .4
_
.sp .2
p	Print contents of dot
\&=	Print value (line numbers and character numbers) of dot
.sp .4
_
.sp .4
\f1File commands\fP
.sp .4
_
.sp .2
b \f2file-list\fP	Set current file to first file in list that \f(CWsam\fP has in menu
B \f2file-list\fP	Same as \f(CWb\fP, but load new files
n	Print menu lines of all files
D \f2file-list\fP	Delete named files from \f(CWsam\fP
.sp .4
_
.sp .4
\f1I/O commands\fP	
.sp .4
_
.sp .2
e \f2filename\fP	Replace file with named disc file
r \f2filename\fP	Replace dot by contents of named disc file
w \f2filename\fP	Write file to named disc file
f \f2filename\fP	Set file name and print new menu line
< \f2UNIX-command\fP	Replace dot by standard output of command
> \f2UNIX-command\fP	Send dot to standard input of command
| \f2UNIX-command\fP	Replace dot by result of command applied to dot
! \f2UNIX-command\fP	Run the command
.sp .4
_
.sp .4
\f1Loops and conditionals\fP	
.sp .4
_
.sp .2
x/\f2regexp\fP/ \f2command\fP	For each match of regexp, set dot and run command
y/\f2regexp\fP/ \f2command\fP	Between adjacent matches of regexp, set dot and run command
X/\f2regexp\fP/ \f2command\fP	Run command in each file whose menu line matches regexp
Y/\f2regexp\fP/ \f2command\fP	Run command in each file whose menu line does not match
g/\f2regexp\fP/ \f2command\fP	If dot contains a match of regexp, run command
v/\f2regexp\fP/ \f2command\fP	If dot does not contain a match of regexp, run command
.sp .4
_
.sp .4
\f1Miscellany\fP	
.sp .4
_
.sp .2
k	Set address mark to value of dot
q	Quit
u \f2n\fP	Undo last \f2n\fP (default 1) changes
{ }	Braces group commands
.sp .3
.ft CW
_
.ft
.TE
.sp
.KE
.PP
The value of dot may be changed by
specifying an
.I address
for the command.
The simplest address is a line number:
.P1
3
.P2
refers to the third line of the file, so
.P1
3d
.P2
deletes the third line of the file, and implicitly renumbers
the lines so the old line 4 is now numbered 3.
(This is one of the few places where
.CW sam
deals with lines directly.)
Line
.CW 0
is the null string at the beginning of the file.
If a command consists of only an address, a
.CW p
command is assumed, so typing an unadorned
.CW 3
prints line 3 on the terminal.
There are a couple of other basic addresses:
a period addresses dot itself; and
a dollar sign
.CW $ ) (
addresses the null string at the end of the file.
.PP
An address is always a single substring of the file.
Thus, the address
.CW 3
addresses the characters
after the second newline of
the file through the third newline of the file.
A
.I
compound address
.R
is constructed by the comma operator
.P1
\f2address1\fP,\f2address2\fP
.P2
and addresses the substring of the file from the beginning of
.I address1
to the end of
.I address2 .
For example, the command
.CW 3,5p
prints the third through fifth lines of the file and
.CW .,$d
deletes the text from the beginning of dot to the end of the file.
.PP
These addresses are all absolute positions in the file, but
.CW sam
also has relative addresses, indicated by
.CW +
or
.CW - .
For example,
.P1
$-3
.P2
is the third line before the end of the file and
.P1
\&.+1
.P2
is the line after dot.
If no address appears to the left of the
.CW +
or
.CW - ,
dot is assumed;
if nothing appears to the right,
.CW 1
is assumed.
Therefore,
.CW .+1
may be abbreviated to just a plus sign.
.PP
The
.CW +
operator acts relative to the end of its first argument, while the
.CW -
operator acts relative to the beginning.  Thus
.CW .+1
addresses the first line after dot,
.CW .-
addresses the first line before dot, and
.CW +-
refers to the line containing the end of dot.  (Dot may span multiple lines, and
.CW +
selects the line after the end of dot, then
.CW -
backs up one line.)
.PP
The final type of address is a regular expression, which addresses the
text matched by the expression.  The expression is enclosed in slashes, as in
.P1
/\f2expression\fP/
.P2
The expressions are the same as those in the UNIX program
.CW egrep ,\u\s-4\&6,7\s+4\d
and include closures, alternations, and so on.
They find the
.I
leftmost longest
.R
string that matches the expression, that is,
the first match after the point where the search is started,
and if more than one match begins at the same spot, the longest such match.
(I assume familiarity with the syntax for regular expressions in UNIX programs.\u\s-4\&9\s+4\d)
For example,
.P1
/x/
.P2
matches the next
.CW x
character in the file,
.P1
/xx*/
.P2
matches the next run of one or more
.CW x 's,
and
.P1
/x|Peter/
.P2
matches the next
.CW x
or
.CW Peter .
For compatibility with other UNIX programs, the `any character' operator,
a period,
does not match a newline, so
.P1
/.*/
.P2
matches the text from dot to the end of the line, but excludes the newline
and so will not match across
the line boundary.
.PP
Regular expressions are always relative addresses.
The direction is forwards by default,
so
.CW /Peter/
is really an abbreviation for
.CW +/Peter/ .
The search can be reversed with a minus sign, so
.P1
.CW -/Peter/
.P2
finds the first
.CW Peter
before dot.
Regular expressions may be used with other address forms, so
.CW 0+/Peter/
finds the first
.CW Peter
in the file and
.CW $-/Peter/
finds the last.
Table II summarizes
.CW sam 's
addresses.
.KF
.TS
center;
c s
lfCW l.
Table II. \f(CWSam\fP addresses
.sp .4
.ft CW
_
.ft
.sp .4
\f1Simple addresses\fP	
.sp .4
_
.sp .2
#\f2n\fP	The empty string after character \f2n\fP
\f2n\fP	Line \f2n\fP.
/\f2regexp\fP/	The first following match of the regular expression
-/\f2regexp\fP/	The first previous match of the regular expression
$	The null string at the end of the file
\&.	Dot
\&'	The address mark, set by \f(CWk\fP command
"\f2regexp\fP"	Dot in the file whose menu line matches regexp
.sp .4
_
.sp .4
\f1Compound addresses\fP	
.sp .4
_
.sp .2
\f2a1\fP+\f2a2\fP	The address \f2a2\fP evaluated starting at right of \f2a1\fP
\f2a1\fP-\f2a2\fP	\f2a2\fP evaluated in the reverse direction starting at left of \f2a1\fP
\f2a1\fP,\f2a2\fP	From the left of \f2a1\fP to the right of \f2a2\fP (default \f(CW0,$\fP)
\f2a1\fP;\f2a2\fP	Like \f(CW,\fP but sets dot after evaluating \f2a1\fP
.sp .4
_
.sp .4
.T&
c s.
T{
The operators
.CW +
and
.CW -
are high precedence, while
.CW ,
and
.CW ;
are low precedence.
In both
.CW +
and
.CW -
forms,
.I a2
defaults to 1 and
.I a1
defaults to dot.
If both
.I a1
and
.I a2
are present,
.CW +
may be elided.
T}
.sp .5
.ft CW
_
.ft
.TE
.sp
.KE
.PP
The language discussed so far will not seem novel
to people who use UNIX text editors
such as
.CW ed
or
.CW vi .\u\s-4\&9\s+4\d
Moreover, the kinds of editing operations these commands allow, with the exception
of regular expressions and line numbers,
are clearly more conveniently handled by a mouse-based interface.
Indeed,
.CW sam 's
mouse language (discussed at length below) is the means by which
simple changes are usually made.
For large or repetitive changes, however, a textual language
outperforms a manual interface.
.PP
Imagine that, instead of deleting just one occurrence of the string
.CW Peter ,
we wanted to eliminate every
.CW Peter .
What's needed is an iterator that runs a command for each occurrence of some
text.
.CW Sam 's
iterator is called
.CW x ,
for extract:
.P1
x/\f2expression\fP/ \f2command\fP
.P2
finds all matches in dot of the specified expression, and for each
such match, sets dot to the text matched and runs the command.
So to delete all the
.CW Peters:
.P1
0,$ x/Peter/ d
.P2
(Blanks in these examples are to improve readability;
.CW sam
neither requires nor interprets them.)
This searches the entire file
.CW 0,$ ) (
for occurrences of the string
.CW Peter ,
and runs the
.CW d
command with dot set to each such occurrence.
(By contrast, the comparable
.CW ed
command would delete all
.I lines
containing
.CW Peter ;
.CW sam
deletes only the
.CW Peters .)
The address
.CW 0,$
is commonly used, and may be abbreviated to just a comma.
As another example,
.P1
, x/Peter/ p
.P2
prints a list of
.CW Peters,
one for each appearance in the file, with no intervening text (not even newlines
to separate the instances).
.PP
Of course, the text extracted by
.CW x
may be selected by a regular expression,
which complicates deciding what set of matches is chosen \(em
matches may overlap.  This is resolved by generating the matches
starting from the beginning of dot using the leftmost-longest rule,
and searching for each match starting from the end of the previous one.
Regular expressions may also match null strings, but a null match
adjacent to a non-null match is never selected; at least one character
must intervene.
For example,
.P1
, c/AAA/
x/B*/ c/-/
, p
.P2
produces as output
.P1
-A-A-A-
.P2
because the pattern
.CW B*
matches the null strings separating the
.CW A 's.
.PP
The
.CW x
command has a complement,
.CW y ,
with similar syntax, that executes the command with dot set to the text
.I between
the matches of the expression.
For example,
.P1
, c/AAA/
y/A/ c/-/
, p
.P2
produces the same result as the example above.
.PP
The
.CW x
and
.CW y
commands are looping constructs, and
.CW sam
has a pair of conditional commands to go with them.
They have similar syntax:
.P1
g/\f2expression\fP/ \f2command\fP
.P2
(guard)
runs the command exactly once if dot contains a match of the expression.
This is different from
.CW x ,
which runs the command for
.I each
match:
.CW x
loops;
.CW g
merely tests, without changing the value of dot.
Thus,
.P1
, x/Peter/ d
.P2
deletes all occurrences of
.CW Peter ,
but
.P1
, g/Peter/ d
.P2
deletes the whole file (reduces it to a null string) if
.CW Peter
occurs anywhere in the text.
The complementary conditional is
.CW v ,
which runs the command if there is
.I no
match of the expression.
.PP
These control-structure-like commands may be composed to construct more
involved operations.  For example, to print those lines of text that
contain the string
.CW Peter :
.P1
, x/.*\en/ g/Peter/ p
.P2
The
.CW x
breaks the file into lines, the
.CW g
selects those lines containing
.CW Peter ,
and the
.CW p
prints them.
This command gives an address for the
.CW x
command (the whole file), but because
.CW g
does not have an explicit address, it applies to the value of
dot produced by the
.CW x
command, that is, to each line.
All commands in
.CW sam
except for the command to write a file to disc use dot for the
default address.
.PP
Composition may be continued indefinitely.
.P1
, x/.*\en/ g/Peter/ v/SaltPeter/ p
.P2
prints those lines containing
.CW Peter
but
.I not
those containing
.CW SaltPeter .
.SH 2
Structural Regular Expressions
.LP
Unlike other UNIX text editors,
including the non-interactive ones such as
.CW sed
and
.CW awk ,\u\s-4\&7\s+4\d
.CW sam
is good for manipulating files with multi-line `records.'
An example is an on-line phone book composed of records,
separated by blank lines, of the form
.P1
Herbert Tic
44 Turnip Ave., Endive, NJ
201-5555642

Norbert Twinge
16 Potato St., Cabbagetown, NJ
201-5553145

\&...
.P2
The format may be encoded as a regular expression:
.P1
(.+\en)+
.P2
that is, a sequence of one or more non-blank lines.
The command to print Mr. Tic's entire record is then
.P1
, x/(.+\en)+/ g/^Herbert Tic$/ p
.P2
and that to extract just the phone number is
.P1
, x/(.+\en)+/ g/^Herbert Tic$/ x/^[0-9]*-[0-9]*\en/ p
.P2
The latter command breaks the file into records,
chooses Mr. Tic's record,
extracts the phone number from the record,
and finally prints the number.
.PP
A more involved problem is that of
renaming a particular variable, say
.CW n ,
to
.CW num
in a C program.
The obvious first attempt,
.P1
, x/n/ c/num/
.P2
is badly flawed: it changes not only the variable
.CW n
but any letter
.CW n
that appears.
We need to extract all the variables, and select those that match
.CW n
and only
.CW n :
.P1
, x/[A-Za-z_][A-Za-z_0-9]*/ g/n/ v/../ c/num/
.P2
The pattern
.CW [A-Za-z_][A-Za-z_0-9]*
matches C identifiers.
Next
.CW g/n/
selects those containing an
.CW n .
Then
.CW v/../
rejects those containing two (or more) characters, and finally
.CW c/num/
changes the remainder (identifiers
.CW n )
to
.CW num .
This version clearly works much better, but there may still be problems.
For example, in C character and string constants, the sequence
.CW \en
is interpreted as a newline character, and we don't want to change it to
.CW \enum.
This problem can be forestalled with a
.CW y
command:
.P1
, y/\e\en/ x/[A-Za-z_][A-Za-z_0-9]*/ g/n/ v/../ c/num/
.P2
(the second
.CW \e
is necessary because of lexical conventions in regular expressions),
or we could even reject character constants and strings outright:
.P1 0
,y/'[^']*'/ y/"[^"]*"/ x/[A-Za-z_][A-Za-z_0-9]*/ g/n/ v/../ c/num/
.P2
The
.CW y
commands in this version exclude from consideration all character constants
and strings.
The only remaining problem is to deal with the possible occurrence of
.CW \e'
or
.CW \e"
within these sequences, but it's easy to see how to resolve this difficulty.
.PP
The point of these composed commands is successive refinement.
A simple version of the command is tried, and if it's not good enough,
it can be honed by adding a clause or two.
(Mistakes can be undone; see below.
Also, the mouse language makes it unnecessary to retype the command each time.)
The resulting chains of commands are somewhat reminiscent of
shell pipelines.\u\s-4\&7\s+4\d
Unlike pipelines, though, which pass along modified
.I data ,
.CW sam
commands pass a
.I view
of the data.
The text at each step of the command is the same, but which pieces
are selected is refined step by step until the correct piece is
available to the final step of the command line, which ultimately makes the change.
.PP
In other UNIX programs, regular expressions are used only for selection,
as in the
.CW sam
.CW g
command, never for extraction as in the
.CW x
or
.CW y
command.
For example, patterns in
.CW awk \u\s-4\&7\s+4\d
are used to select lines to be operated on, but cannot be used
to describe the format of the input text, or to handle newline-free text.
The use of regular expressions to describe the structure of a piece
of text rather than its contents, as in the
.CW x
command, 
has been given a name:
.I
structural regular expressions.
.R
When they are composed, as in the above example,
they are pleasantly expressive.
Their use is discussed at greater length elsewhere.\u\s-4\&10\s+4\d
.PP
.SH 2
Multiple files
.LP
.CW Sam
has a few other commands, mostly relating to input and output.
.P1
e discfilename
.P2
replaces the contents and name of the current file with those of the named
disc file;
.P1
w discfilename
.P2
writes the contents to the named disc file; and
.P1
r discfilename
.P2
replaces dot with the contents of the named disc file.
All these commands use the current file's name if none is specified.
Finally,
.P1
f discfilename
.P2
changes the name associated with the file and displays the result:
.P1
\&'-. discfilename
.P2
This output is called the file's
.I
menu line,
.R
because it is the contents of the file's line in the button 3 menu (described
in the
next section).
The first three characters are a concise notation for the state of the file.
The apostrophe signifies that the file is modified.
The minus sign indicates the number of windows
open on the file (see the next section):
.CW -
means none,
.CW +
means one, and
.CW *
means more than one.
Finally, the period indicates that this is the current file.
These characters are useful for controlling the
.CW X
command, described shortly.
.PP
.CW Sam
may be started with a set of disc files (such as all the source for
a program) by invoking it with a list of file names as arguments, and
more may be added or deleted on demand.
.P1
B discfile1 discfile2 ...
.P2
adds the named files to
.CW sam 's
list, and
.P1
D discfile1 discfile2 ...
.P2
removes them from
.CW sam 's
memory (without effect on associated disc files).
Both these commands have a syntax for using the shell\u\s-4\&7\s+4\d
(the UNIX command interpreter) to generate the lists:
.P1
B <echo *.c
.P2
will add all C source files, and
.P1
B <grep -l variable *.c
.P2
will add all C source files referencing a particular variable
(the UNIX command
.CW grep\ -l
lists all files in its arguments that contain matches of
the specified regular expression).
Finally,
.CW D
without arguments deletes the current file.
.PP
There are two ways to change which file is current:
.P1
b filename
.P2
makes the named file current.
The
.CW B
command
does the same, but also adds any new files to
.CW sam 's
list.
(In practice, of course, the current file
is usually chosen by mouse actions, not by textual commands.)
The other way is to use a form of address that refers to files:
.P1
"\f2expression\fP" \f2address\fP
.P2
refers to the address evaluated in the file whose menu line
matches the expression (there must be exactly one match).
For example,
.P1
"peter.c" 3
.P2
refers to the third line of the file whose name matches
.CW peter.c .
This is most useful in the move
.CW m ) (
and copy
.CW t ) (
commands:
.P1
0,$ t "peter.c" 0
.P2
makes a copy of the current file at the beginning of
.CW peter.c .
.PP
The
.CW X
command
is a looping construct, like
.CW x ,
that refers to files instead of strings:
.P1
X/\f2expression\fP/ \f2command\fP
.P2
runs the command in all
files whose menu lines match the expression.  The best example is
.P1
X/'/ w
.P2
which writes to disc all modified files.
.CW Y
is the complement of
.CW X :
it runs the command on all files whose menu lines don't match the expression:
.P1
Y/\e.c/ D
.P2
deletes all files that don't have
.CW \&.c
in their names, that is, it keeps all C source files and deletes the rest.
.PP
Braces allow commands to be grouped, so
.P1
{
	\f2command1\fP
	\f2command2\fP
}
.P2
is syntactically a single command that runs two commands.
Thus,
.P1
X/\e.c/ ,g/variable/ {
	f
	, x/.*\en/ g/variable/ p
}
.P2
finds all occurrences of
.CW variable
in C source files, and prints
out the file names and lines of each match.
The precise semantics of compound operations is discussed in the implementation
sections below.
.PP
Finally,
the undo command,
.CW u ,
undoes the last command,
no matter how many files were affected.
Multiple undo operations move further back in time, so
.P1
u
u
.P2
(which may be abbreviated
.CW u2 )
undoes the last two commands.  An undo may not be undone, however, nor
may any command that adds or deletes files.
Everything else is undoable, though, including for example
.CW e
commands:
.P1
e filename
u
.P2
restores the state of the file completely, including its name, dot,
and modified bit.  Because of the undo, potentially dangerous commands
are not guarded by confirmations.  Only
.CW D ,
which destroys the information necessary to restore itself, is protected.
It will not delete a modified file, but a second
.CW D
of the same file will succeed regardless.
The
.CW q
command, which exits
.CW sam ,
is similarly guarded.
.SH 2
Mouse Interface
.LP
.CW Sam
is most commonly run
connected to a bitmap display and mouse for interactive editing.
The only difference in the command language
between regular, mouse-driven
.CW sam
and
.CW sam\ -d
is that if an address
is provided without a command,
.CW sam\ -d
will print the text referenced by the address, but
regular
.CW sam
will highlight it on the screen \(em in fact,
dot is always highlighted (see Figure 2).
.WS 1
.KF
.BP fig3.ps 2.04i
.Cs
Figure 2. A
.CW sam
window.  The scroll bar down the left
represents the file, with the bubble showing the fraction
visible in the window.
The scroll bar may be manipulated by the mouse for convenient browsing.
The current text,
which is highlighted, need not fit on a line.  Here it consists of one partial
line, one complete line, and final partial line.
.Ce
.KE
.PP
Each file may have zero or more windows open on the display.
At any time, only one window in all of
.CW sam
is the
.I
current window,
.R
that is, the window to which typing and mouse actions refer;
this may be the
.CW sam
window (that in which commands may be typed)
or one of the file windows.
When a file has multiple windows, the image of the file in each window
is always kept up to date.
The current file is the last file affected by a command,
so if the
.CW sam
window is current,
the current window is not a window on the current file.
However, each window on a file has its own value of dot,
and when switching between windows on a single file,
the file's value of dot is changed to that of the window.
Thus, flipping between windows behaves in the obvious, convenient way.
.PP
The mouse on the Blit has three buttons, numbered left to right.
Button 3 has a list of commands to manipulate windows,
followed by a list of `menu lines' exactly as printed by the
.CW f
command, one per file (not one per window).
These menu lines are sorted by file name.
If the list is long, the Blit menu software will make it more manageable
by generating a scrolling menu instead of an unwieldy long list.
Using the menu to select a file from the list makes that file the current
file, and the most recently current window in that file the current window.
But if that file is already current, selecting it in the menu cycles through
the windows on the file; this simple trick avoids a special menu to
choose windows on a file.
If there is no window open on the file,
.CW sam
changes the mouse cursor to prompt the user to create one.
.PP
The commands on the button 3 menu are straightforward (see Figure 3), and
are like the commands to manipulate windows in
.CW mux ,\u\s-4\&8\s+4\d
the Blit's window system.
.CW New
makes a new file, and gives it one empty window, whose size is determined
by a rectangle swept by the mouse.
.CW Zerox
prompts for a window to be selected, and
makes a clone of that window; this is how multiple windows are created on one file.
.CW Reshape
changes the size of the indicated window, and
.CW close
deletes it.  If that is the last window open on the file,
.CW close
first does a
.CW D
command on the file.
.CW Write
is identical to a
.CW w
command on the file; it is in the menu purely for convenience.
Finally,
.CW ~~sam~~
is a menu item that appears between the commands and the file names.
Selecting it makes the
.CW sam
window the current window,
causing subsequent typing to be interpreted as commands.
.KF
.BP fig2.ps 2.74i
.Cs
Figure 3. The menu on button 3.
The black rectangle on the left is a scroll bar; the menu is limited to
the length shown to prevent its becoming unwieldy.
Above the
.CW ~~sam~~
line is a list of commands;
beneath it is a list of files, presented exactly as with the
.CW f
command.
.Ce
.KE
.PP
When
.CW sam
requests that a window be swept, in response to
.CW new ,
.CW zerox
or
.CW reshape ,
it changes the mouse cursor from the usual arrow to a box with
a small arrow.
In this state, the mouse may be used to indicate an arbitrary rectangle by
pressing button 3 at one corner and releasing it at the opposite corner.
More conveniently,
button 3 may simply be clicked,
whereupon
.CW sam
creates the maximal rectangle that contains the cursor
and abuts the
.CW sam
window.
By placing the
.CW sam
window in the middle of the screen, the user can define two regions (one above,
one below) in which stacked fully-overlapping
windows can be created with minimal fuss (see Figure 1).
This simple user interface trick makes window creation noticeably easier.
.PP
The cut-and-paste editor is essentially the same as that in Smalltalk-80.\u\s-4\&11\s+4\d
The text in dot is always highlighted on the screen.
When a character is typed it replaces dot, and sets dot to the null
string after the character.  Thus, ordinary typing inserts text.
Button 1 is used for selection:
pressing the button, moving the mouse, and lifting the button
selects (sets dot to) the text between the points where the
button was pressed and released.
Pressing and releasing at the same point selects a null string; this
is called clicking.  Clicking twice quickly, or
.I
double clicking,
.R
selects larger objects;
for example, double clicking in a word selects the word,
double clicking just inside an opening bracket selects the text
contained in the brackets (handling nested brackets correctly),
and similarly for
parentheses, quotes, and so on.
The double-clicking rules reflect a bias toward
programmers.
If
.CW sam
were intended more for word processing, double-clicks would probably
select linguistic structures such as sentences.
.PP
If button 1 is pressed outside the current window, it makes the indicated
window current.
This is the easiest way to switch between windows and files.
.PP
Pressing button 2 brings up a menu of editing functions (see Figure 4).
These mostly apply to the selected text:
.CW cut
deletes the selected text, and remembers it in a hidden buffer called the
.I
snarf buffer,
.R
.CW paste
replaces the selected text by the contents of the snarf buffer,
.CW snarf
just copies the selected text to the snarf buffer,
.CW look
searches forward for the next literal occurrence of the selected text, and
.CW <mux>
exchanges snarf buffers with the window system in which
.CW sam
is running.
Finally, the last regular expression used appears as a menu entry
to search
forward for the next occurrence of a match for the expression.
.WS 1
.KF
.BP fig4.ps 1.20i
.Cs
Figure 4. The menu on button 2.
The bottom entry tracks the most recently used regular expression, which may
be literal text.
.Ce
.KE
.PP
The relationship between the command language and the mouse language is
entirely due to the equality of dot and the selected text chosen
with button 1 on the mouse.
For example, to make a set of changes in a C subroutine, dot can be
set by double clicking on the left brace that begins the subroutine,
which sets dot for the command language.
An address-free command then typed in the
.CW sam
window will apply only to the text between the opening and closing
braces of the function.
The idea is to select what you want, and then say what you want
to do with it, whether invoked by a menu selection or by a typed command.
And of course, the value of dot is highlighted on
the display after the command completes.
This relationship between mouse interface and command language
is clumsy to explain, but comfortable, even natural, in practice.
.SH
The Implementation
.LP
The next few sections describe how
.CW sam
is put together, first the host part,
then the inter-component communication,
then the terminal part.
After explaining how the command language is implemented,
the discussion follows (roughly) the path of a character
from the temporary file on disc to the screen.
The presentation centers on the data structures,
because that is how the program was designed and because
the algorithms are easy to provide, given the right data
structures.
.SH 2
Parsing and execution
.LP
The command language is interpreted by parsing each command with a
table-driven recursive
descent parser, and when a complete command is assembled, invoking a top-down
executor.
Most editors instead employ a simple character-at-a-time
lexical scanner.
Use of a parser makes it
easy and unambiguous to detect when a command is complete,
which has two advantages.
First, escape conventions such as backslashes to quote
multiple-line commands are unnecessary;  if the command isn't finished,
the parser keeps reading.  For example, a multiple-line append driven by an
.CW x
command is straightforward:
.P1
x/.*\en/ g/Peter/ a
one line about Peter
another line about Peter
\&.
.P2
Other UNIX editors would require a backslash after all but the last line.
.PP
The other advantage is specific to the two-process structure of
.CW sam .
The host process must decide when a command is completed so the
command interpreter can be called.  This problem is easily resolved
by having the lexical analyzer read the single stream of events from the
terminal, directly executing all typing and mouse commands,
but passing to the parser characters typed to the
.CW sam
command window.
This scheme is slightly complicated by the availability of cut-and-paste
editing in the
.CW sam
window, but that difficulty is resolved by applying the rules
used in
.CW mux :
when a newline is typed to the
.CW sam
window, all text between the newline and the previously typed newline
is made available to the parser.
This permits arbitrary editing to be done to a command before
typing newline and thereby requesting execution.
.PP
The parser is driven by a table because the syntax of addresses
and commands is regular enough
to be encoded compactly.  There are few special cases, such as the
replacement text in a substitution, so the syntax of almost all commands
can be encoded with a few flags.
These include whether the command allows an address (for example,
.CW e
does not), whether it takes a regular expression (as in
.CW x
and
.CW s ),
whether it takes replacement text (as in
.CW c
or
.CW i ),
which may be multi-line, and so on.
The internal syntax of regular expressions is handled by a separate
parser; a regular expression is a leaf of the command parse tree.
Regular expressions are discussed fully in the next section.
.PP
The parser table also has information about defaults, so the interpreter
is always called with a complete tree.  For example, the parser fills in
the implicit
.CW 0
and
.CW $
in the abbreviated address
.CW ,
(comma),
inserts a
.CW +
to the left of an unadorned regular expression in an address,
and provides the usual default address
.CW .
(dot) for commands that expect an address but are not given one.
.PP
Once a complete command is parsed, the evaluation is easy.
The address is evaluated left-to-right starting from the value of dot,
with a mostly ordinary expression evaluator.
Addresses, like many of the data structures in
.CW sam ,
are held in a C structure and passed around by value:
.P1
typedef long Posn;    /* Position in a file */
typedef struct Range{
        Posn    p1, p2;
}Range;
typedef struct Address{
        Range   r;
        File    *f;
}Address;
.P2
An address is encoded as a substring (character positions
.CW p1
to
.CW p2 )
in a file
.CW f .
(The data type
.CW File
is described in detail below.)
.PP
The address interpreter is an
.CW Address -valued
function that traverses the parse tree describing an address (the
parse tree for the address has type
.CW Addrtree ):
.P1
Address
address(ap, a, sign)
	Addrtree *ap;
	Address a;
	int sign;
{
	Address a2;
	do
		switch(ap->type){
		case '.':
			a=a.f->dot;
			break;
		case '$':
			a.r.p1=a.r.p2=a.f->nbytes;
			break;
		case '"':	
			a=matchfile(a, ap->aregexp)->dot; 
			break;
		case ',':
			a2=address(ap->right, a, 0);
			a=address(ap->left, a, 0);
			if(a.f!=a2.f || a2.r.p2<a.r.p1)
				error(Eorder);
			a.r.p2=a2.r.p2;
			return a;
		/* and so on */
		}
	while((ap=ap->right)!=0);
	return a;
}
.P2
.PP
Throughout, errors are handled by a non-local
.CW goto
(a
.CW setjmp/longjmp
in C terminology)
hidden in a routine called
.CW error
that immediately aborts the execution, retracts any
partially made changes (see the section below on `undoing'), and
returns to the top level of the parser.
The argument to
.CW error
is an enumeration type that
is translated to a terse but possibly helpful
message such as `?addresses out of order.'
Very common messages are kept short; for example the message for
a failed regular expression search is `?search.'
.PP
Character addresses such as
.CW #3
are trivial to implement, as the
.CW File
data structure is accessible by character number.
However,
.CW sam
keeps no information about the position of newlines \(em it is too
expensive to track dynamically \(em so line addresses are computed by reading
the file, counting newlines.  Except in very large files, this has proven
acceptable: file access is fast enough to make the technique practical,
and lines are not central to the structure of the command language.
.PP
The command interpreter, called
.CW cmdexec ,
is also straightforward.  The parse table includes a
function to call to interpret a particular command.  That function
receives as arguments
the calculated address
for the command
and the command tree (of type
.CW Cmdtree ),
which may contain information such as the subtree for compound commands.
Here, for example, is the function for the
.CW g
and
.CW v
commands:
.P1
int
g_cmd(a, cp)
	Address a;
	Cmdtree *cp;
{
	compile(cp->regexp);
	if(execute(a.f, a.r.p1, a.r.p2)!=(cp->cmdchar=='v')){
		a.f->dot=a;
		return cmdexec(a, cp->subcmd);
	}
	return TRUE;	/* cause execution to continue */
}
.P2
.CW Compile "" (
and
.CW execute
are part of the regular expression code, described in the next section.)
Because the parser and the
.CW File
data structure do most of the work, most commands
are similarly brief.
.SH 2
Regular expressions
.LP
The regular expression code in
.CW sam
is an interpreted, rather than compiled on-the-fly, implementation of Thompson's
non-deterministic finite automaton algorithm.\u\s-4\&12\s+4\d
The syntax and semantics of the expressions are as in the UNIX program
.CW egrep ,
including alternation, closures, character classes, and so on.
The only changes in the notation are two additions:
.CW \en
is translated to, and matches, a newline character, and
.CW @
matches any character.  In
.CW egrep ,
the character
.CW \&.
matches any character except newline, and in
.CW sam
the same rule seemed safest, to prevent idioms like
.CW \&.*
from spanning newlines.
.CW Egrep
expressions are arguably too complicated for an interactive editor \(em
certainly it would make sense if all the special characters were two-character
sequences, so that most of the punctuation characters wouldn't have
peculiar meanings \(em but for an interesting command language, full
regular expressions are necessary, and
.CW egrep
defines the full regular expression syntax for UNIX programs.
Also, it seemed superfluous to define a new syntax, since various UNIX programs
.CW ed , (
.CW egrep
and
.CW vi )
define too many already.
.PP
The expressions are compiled by a routine,
.CW compile ,
that generates the description of the non-deterministic finite state machine.
A second routine,
.CW execute ,
interprets the machine to generate the leftmost-longest match of the
expression in a substring of the file.
The algorithm is described elsewhere.\u\s-4\&12,13\s+4\d
.CW Execute
reports
whether a match was found, and sets a global variable,
of type
.CW Range ,
to the substring matched.
.PP
A trick is required to evaluate the expression in reverse, such as when
searching backwards for an expression.
For example,
.P1
-/P.*r/
.P2
looks backwards through the file for a match of the expression.
The expression, however, is defined for a forward search.
The solution is to construct a machine identical to the machine
for a forward search except for a reversal of all the concatenation
operators (the other operators are symmetric under direction reversal),
to exchange the meaning of the operators
.CW ^
and
.CW $ ,
and then to read the file backwards, looking for the
usual earliest longest match.
.PP
.CW Execute
generates only one match each time it is called.
To interpret looping constructs such as the
.CW x
command,
.CW sam
must therefore synchronize between
calls of
.CW execute
to avoid
problems with null matches.
For example, even given the leftmost-longest rule,
the expression
.CW a*
matches three times in the string
.CW ab
(the character
.CW a ,
the null string between the
.CW a
and
.CW b ,
and the final null string).
After returning a match for the
.CW a ,
.CW sam
must not match the null string before the
.CW b .
The algorithm starts
.CW execute
at the end of its previous match, and
if the match it returns
is null and abuts the previous match, rejects the match and advances
the initial position one character.
.SH 2
Memory allocation
.LP
The C language has no memory allocation primitives, although a standard
library routine,
.CW malloc ,
provides adequate service for simple programs.
For specific uses, however,
it can be better to write a custom allocator.
The allocator (or rather, pair of allocators) described here
work in both the terminal and host parts of
.CW sam .
They are designed for efficient manipulation of strings,
which are allocated and freed frequently and vary in length from essentially
zero to 32 Kbytes (very large strings are written to disc).
More important, strings may be large and change size often,
so to minimize memory usage it is helpful to reclaim and to coalesce the
unused portions of strings when they are truncated.
.PP
Objects to be allocated in
.CW sam
are of two flavors:
the first is C
.CW structs ,
which are small and often addressed by pointer variables;
the second is variable-sized arrays of characters
or integers whose
base pointer is always used to access them.
The memory allocator in
.CW sam
is therefore in two parts:
first, a traditional first-fit allocator that provides fixed storage for
.CW structs ;
and second, a garbage-compacting allocator that reduces storage
overhead for variable-sized objects, at the cost of some bookkeeping.
The two types of objects are allocated from adjoining arenas, with
the garbage-compacting allocator controlling the arena with higher addresses.
Separating into two arenas simplifies compaction and prevents fragmentation due
to immovable objects.
The access rules for garbage-compactable objects
(discussed in the next paragraph) allow them to be relocated, so when
the first-fit arena needs space, it moves the garbage-compacted arena
to higher addresses to make room.  Storage is therefore created only
at successively higher addresses, either when more garbage-compacted
space is needed or when the first-fit arena pushes up the other arena.
.PP
Objects that may be compacted declare to the
allocator a cell that is guaranteed to be the sole repository of the
address of the object whenever a compaction can occur.
The compactor can then update the address when the object is moved.
For example, the implementation of type
.CW List
(really a variable-length array)
is:
.P1
typedef struct List{
        int     nused;
        long    *ptr;
}List;
.P2
The
.CW ptr
cell must always be used directly, and never copied.  When a
.CW List
is to be created the
.CW List
structure is allocated in the ordinary first-fit arena
and its
.CW ptr
is allocated in the garbage-compacted arena.
A similar data type for strings, called
.CW String ,
stores variable-length character arrays of up to 32767 elements.
.PP
A related matter of programming style:
.CW sam
frequently passes structures by value, which
simplifies the code.
Traditionally, C programs have
passed structures by reference, but implicit allocation on
the stack is easier to use.
Structure passing is a relatively new feature of C
(it is not in the 
standard reference manual for C\u\s-4\&14\s+4\d), and is poorly supported in most
commercial C compilers.
It's convenient and expressive, though,
and simplifies memory management by
avoiding the allocator altogether
and eliminating pointer aliases.
.SH 2
Data structures for manipulating files
.LP
Experience with
.CW jim
showed that the requirements
of the file data structure were few, but strict.
First, files need to be read and written quickly;
adding a fresh file must be painless.
Second, the implementation must place no arbitrary upper limit on
the number or sizes of files.  (It should be practical to edit many files,
and files up to megabytes in length should be handled gracefully.)
This implies that files be stored on disc, not in main memory.
(Aficionados of virtual memory may argue otherwise, but the
implementation of virtual
memory in our system is not something to depend on
for good performance.)
Third, changes to files need be made by only two primitives:
deletion and insertion.
These are inverses of each other,
which simplifies the implementation of the undo operation.
Finally,
it must be easy and efficient to access the file, either
forwards or backwards, a byte at a time.
.PP
The
.CW File
data type is constructed from three simpler data structures that hold arrays
of characters.
Each of these types has an insertion and deletion operator, and the
insertion and deletion operators of the
.CW File
type itself are constructed from them.
.PP
The simplest type is the
.CW String ,
which is used to hold strings in main memory.
The code that manages
.CW Strings
guarantees that they will never be longer
than some moderate size, and in practice they are rarely larger than 8 Kbytes.
.CW Strings
have two purposes: they hold short strings like file names with little overhead,
and because they are deliberately small, they are efficient to modify.
They are therefore used as the data structure for in-memory caches.
.PP
The disc copy of the file is managed by a data structure called a
.CW Disc ,
which corresponds to a temporary file.  A
.CW Disc
has no storage in main memory other than bookkeeping information;
the actual data being held is all on the disc.
To reduce the number of open files needed,
.CW sam
opens a dozen temporary UNIX files and multiplexes the
.CW Discs
upon them.
This permits many files to
be edited; the entire
.CW sam
source (48 files) may be edited comfortably with a single
instance of
.CW sam .
Allocating one temporary file per
.CW Disc
would strain the operating system's limit on the number of open files.
Also, spreading the traffic among temporary files keeps the files shorter,
and shorter files are more efficiently implemented by the UNIX
I/O subsystem.
.PP
A
.CW Disc
is an array of fixed-length blocks, each of which contains
between 1 and 4096 characters of active data.
(The block size of our UNIX file system is 4096 bytes.)
The block addresses within the temporary file and the length of each
block are stored in a
.CW List .
When changes are made the live part of blocks may change size.
Blocks are created and coalesced when necessary to try to keep the sizes
between 2048 and 4096 bytes.
An actively changing part of the
.CW Disc
therefore typically has about a kilobyte of slop that can be
inserted or deleted
without changing more than one block or affecting the block order.
When an insertion would overflow a block, the block is split, a new one
is allocated to receive the overflow, and the memory-resident list of blocks
is rearranged to reflect the insertion of the new block.
.PP
Obviously, going to the disc for every modification to the file is
prohibitively expensive.
The data type
.CW Buffer
consists of a
.CW Disc
to hold the data and a
.CW String
that acts as a cache.
This is the first of a series of caches throughout the data structures in
.CW sam.
The caches not only improve performance, they provide a way to organize
the flow of data, particularly in the communication between the host
and terminal.
This idea is developed below, in the section on communications.
.PP
To reduce disc traffic, changes to a
.CW Buffer
are mediated by a variable-length string, in memory, that acts as a cache.
When an insertion or deletion is made to a
.CW Buffer ,
if the change can be accommodated by the cache, it is done there.
If the cache becomes bigger than a block because of an insertion,
some of it is written to the
.CW Disc
and deleted from the cache.
If the change does not intersect the cache, the cache is flushed.
The cache is only loaded at the new position if the change is smaller than a block;
otherwise, it is sent directly to the
.CW Disc .
This is because
large changes are typically sequential,
whereupon the next change is unlikely to overlap the current one.
.PP
A
.CW File
comprises a
.CW String
to hold the file name and some ancillary data such as dot and the modified bit.
The most important components, though, are a pair of
.CW Buffers ,
one called the transcript and the other the contents.
Their use is described in the next section.
.PP
The overall structure is shown in Figure 5.
Although it may seem that the data is touched many times on its
way from the
.CW Disc ,
it is read (by one UNIX system call) directly into the cache of the
associated
.CW Buffer ;
no extra copy is done.
Similarly, when flushing the cache, the text is written
directly from the cache to disc.
Most operations act directly on the text in the cache.
A principle applied throughout
.CW sam
is that the fewer times the data is copied, the faster the program will run
(see also the paper by Waite\u\s-4\&15\s+4\d).
.KF
.PS
copy "fig5.pic"
.PE
.Cs
Figure 5. File data structures.
The temporary files are stored in the standard repository for such files
on the host system.
.Ce
.KE
.PP
The contents of a
.CW File
are accessed by a routine that
copies to a buffer a substring of a file starting at a specified offset.
To read a byte at a time, a
.CW File "" per-
array is loaded starting from a specified initial position,
and bytes may then be read from the array.
The implementation is done by a macro similar to the C standard I/O
.CW getc
macro.\u\s-4\&14\s+4\d
Because the reading may be done at any address, a minor change to the
macro allows the file to be read backwards.
This array is read-only; there is no
.CW putc .
.SH 2
Doing and undoing
.LP
.CW Sam
has an unusual method for managing changes to files.
The command language makes it easy to specify multiple variable-length changes
to a file millions of bytes long, and such changes
must be made efficiently if the editor is to be practical.
The usual techniques for inserting and deleting strings
are inadequate under these conditions.
The
.CW Buffer
and
.CW Disc
data structures are designed for efficient random access to long strings,
but care must be taken to avoid super-linear behavior when making
many changes simultaneously.
.PP
.CW Sam
uses a two-pass algorithm for making changes, and treats each file as a database
against which transactions are registered.
Changes are not made directly to the contents.
Instead, when a command is started, a `mark' containing
a sequence number is placed in the transcript
.CW Buffer ,
and each change made to the file, either an insertion or deletion
or a change to the file name,
is appended to the end of the transcript.
When the command is complete, the transcript is rewound to the
mark and applied to the contents.
.PP
One reason for separating evaluation from
application in this way is to simplify tracking the addresses of changes
made in the middle of a long sequence.
The two-pass algorithm also allows all changes to apply to the
.I original
data: no change can affect another change made in the same command.
This is particularly important when evaluating an
.CW x
command because it prevents regular expression matches
from stumbling over changes made earlier in the execution.
Also, the two-pass
algorithm is cleaner than the way other UNIX editors allow changes to
affect each other;
for example,
.CW ed 's
idioms to do things like delete every other line
depend critically on the implementation.
Instead,
.CW sam 's
simple model, in which all changes in a command occur effectively
simultaneously, is easy to explain and to understand.
.PP
The records in the transcript are of the form ``delete substring from
locations
123 to 456'' and ``insert 11 characters `hello there' at location 789.''
(It is an error if the changes are not at monotonically greater
positions through the file.)
While the update is occurring, these numbers must be
offset by earlier changes, but that is straightforward and
local to the update routine;
moreover, all the numbers have been computed
before the first is examined.
.PP
Treating the file as a transaction system has another advantage:
undo is trivial.
All it takes is to invert the transcript after it has been
implemented, converting insertions
into deletions and vice versa, and saving them in a holding
.CW Buffer .
The `do' transcript can then be deleted from
the transcript
.CW Buffer
and replaced by the `undo' transcript.
If an undo is requested, the transcript is rewound and the undo transcript
executed.
Because the transcript
.CW Buffer
is not truncated after each command, it accumulates
successive changes.
A sequence of undo commands
can therefore back up the file arbitrarily,
which is more helpful than the more commonly implemented self-inverse form of undo.
.CW Sam "" (
provides no way to undo an undo, but if it were desired,
it would be easy to provide by re-interpreting the `do' transcript.)
Each mark in the transcript contains a sequence number and the offset into
the transcript of the previous mark, to aid in unwinding the transcript.
Marks also contain the value of dot and the modified bit so these can be
restored easily.
Undoing multiple files is easy; it merely demands undoing all files whose
latest change has the same sequence number as the current file.
.PP
Another benefit of having a transcript is that errors encountered in the middle
of a complicated command need not leave the files in an intermediate state.
By rewinding the transcript to the mark beginning the command,
the partial command can be trivially undone.
.PP
When the update algorithm was first implemented, it was unacceptably slow,
so a cache was added to coalesce nearby changes,
replacing multiple small changes by a single larger one.
This reduced the number
of insertions into the transaction
.CW Buffer ,
and made a dramatic improvement in performance,
but made it impossible
to handle changes in non-monotonic order in the file; the caching method
only works if changes don't overlap.
Before the cache was added, the transaction could in principle be sorted
if the changes were out of order, although
this was never done.
The current status is therefore acceptable performance with a minor
restriction on global changes, which is sometimes, but rarely, an annoyance.
.PP
The update algorithm obviously paws the data more than simpler
algorithms, but it is not prohibitively expensive;
the caches help.
(The principle of avoiding copying the data is still honored here,
although not as piously:
the data is moved from contents' cache to
the transcript's all at once and through only one internal buffer.)
Performance figures confirm the efficiency.
To read from a dead start a hundred kilobyte file on a VAX-11/750
takes 1.4 seconds of user time, 2.5 seconds of system time,
and 5 seconds of real time.
Reading the same file in
.CW ed
takes 6.0 seconds of user time, 1.7 seconds of system time,
and 8 seconds of real time.
.CW Sam
uses about half the CPU time.
A more interesting example is the one stated above:
inserting a character between every pair of characters in the file.
The
.CW sam
command is
.P1
,y/@/ a/x/
.P2
and takes 3 CPU seconds per kilobyte of input file, of which
about a third is spent in the regular expression code.
This translates to about 500 changes per second.
.CW Ed
takes 1.5 seconds per kilobyte to make a similar change (ignoring newlines),
but cannot undo it.
The same example in
.CW ex ,\u\s-4\&9\s+4\d
a variant of
.CW ed
done at the University of California at Berkeley,
which allows one level of undoing, again takes 3 seconds.
In summary,
.CW sam 's
performance is comparable to that of other UNIX editors, although it solves
a harder problem.
.SH 2
Communications
.LP
The discussion so far has described the implementation of the host part of
.CW sam ;
the next few sections explain how a machine with mouse and bitmap display
can be engaged to improve interaction.
.CW Sam
is not the first editor to be written as two processes,\u\s-4\&16\s+4\d
but its implementation
has some unusual aspects.
.PP
There are several ways
.CW sam 's
host and terminal parts may be connected.
The first and simplest is to forgo the terminal part and use the host
part's command language to edit text on an ordinary terminal.
This mode is invoked by starting
.CW sam
with the
.CW -d
option.
With no options,
.CW sam
runs separate host and terminal programs,
communicating with a message protocol over the physical
connection that joins them.
Typically, the connection is an RS-232 link between a Blit
(the prototypical display for
.CW sam )
and a host running
the Ninth Edition of the UNIX operating system.\u\s-4\&8\s+4\d
(This is the version of the system used in the Computing Sciences Research
Center at AT&T Bell Laboratories [now Lucent Technologies, Bell Labs], where I work.  Its relevant
aspects are discussed in the Blit paper.\u\s-4\&1\s+4\d)
The implementation of
.CW sam
for the SUN computer runs both processes on the same machine and
connects them by a pipe.
.PP
The low bandwidth of an RS-232 link
necessitated the split between
the two programs.
The division is a mixed blessing:
a program in two parts is much harder to write and to debug
than a self-contained one,
but the split makes several unusual configurations possible.
The terminal may be physically separated from the host, allowing the conveniences
of a mouse and bitmap display to be taken home while leaving the files at work.
It is also possible to run the host part on a remote machine:
.P1
sam -r host
.P2
connects to the terminal in the usual way, and then makes a call
across the network to establish the host part of
.CW sam
on the named machine.
Finally, it cross-connects the I/O to join the two parts.
This allows
.CW sam
to be run on machines that do not support bitmap displays;
for example,
.CW sam
is the editor of choice on our Cray X-MP/24.
.CW Sam
.CW -r
involves
.I three
machines: the remote host, the terminal, and the local host.
The local host's job is simple but vital: it passes the data
between the remote host and terminal.
.PP
The host and terminal exchange messages asynchronously
(rather than, say, as remote procedure calls) but there is no
error detection or correction
because, whatever the configuration, the connection is reliable.
Because the terminal handles mundane interaction tasks such as
popping up menus and interpreting the responses, the messages are about
data, not actions.
For example, the host knows nothing about what is displayed on the screen,
and when the user types a character, the message sent to the host says
``insert a one-byte string at location 123 in file 7,'' not ``a character
was typed at the current position in the current file.''
In other words, the messages look very much like the transaction records
in the transcripts.
.PP
Either the host or terminal part of
.CW sam
may initiate a change to a file.
The command language operates on the host, while typing and some
mouse operations are executed directly in the terminal to optimize response.
Changes initiated by the host program must be transmitted to the terminal,
and
vice versa.
(A token is exchanged to determine which end is in control,
which means that characters typed while a time-consuming command runs
must be buffered and do not appear until the command is complete.)
To maintain consistent information,
the host and terminal track changes through a per-file
data structure that records what portions of the file
the terminal has received.
The data structure, called a
.CW Rasp
(a weak pun: it's a file with holes)
is held and updated by both the host and terminal.
A
.CW Rasp
is a list of
.CW Strings
holding those parts of the file known to the terminal,
separated by counts of the number of bytes in the interstices.
Of course, the host doesn't keep a separate copy of the data (it only needs
the lengths of the various pieces),
but the structure is the same on both ends.
.PP
The
.CW Rasp
in the terminal doubles as a cache.
Since the terminal keeps the text for portions of the file it has displayed,
it need not request data from the host when revisiting old parts of the file
or redrawing obscured windows, which speeds things up considerably
over low-speed links.
.PP
It's trivial for the terminal to maintain its
.CW Rasp ,
because all changes made on the terminal apply to parts of the file
already loaded there.
Changes made by the host are compared against the
.CW Rasp
during the update sequence after each command.
Small changes to pieces of the file loaded in the terminal
are sent in their entirety.
Larger changes, and changes that fall entirely in the holes,
are transmitted as messages without literal data:
only the lengths of the deleted and inserted strings are transmitted.
When a command is completed, the terminal examines its visible
windows to see if any holes in their
.CW Rasps
intersect the visible portion of the file.
It then requests the missing data from the host,
along with up to 512 bytes of surrounding data, to minimize
the number of messages when visiting a new portion of the file.
This technique provides a kind of two-level lazy evaluation for the terminal.
The first level sends a minimum of information about
parts of the file not being edited interactively;
the second level waits until a change is displayed before
transmitting the new data.
Of course,
performance is also helped by having the terminal respond immediately to typing
and simple mouse requests.
Except for small changes to active pieces of the file, which are
transmitted to the terminal without negotiation,
the terminal is wholly responsible for deciding what is displayed;
the host uses the
.CW Rasp
only to tell the terminal what might be relevant.
.PP
When a change is initiated by the host,
the messages to the terminal describing the change
are generated by the routine that applies the transcript of the changes
to the contents of the
.CW File .
Since changes are undone by the same update routine,
undoing requires
no extra code in the communications;
the usual messages describing changes to the file are sufficient
to back up the screen image.
.PP
The
.CW Rasp
is a particularly good example of the way caches are used in
.CW sam .
First, it facilitates access to the active portion of the text by placing
the busy text in main memory.
In so doing, it provides efficient access
to a large data structure that does not fit in memory.
Since the form of data is to be imposed by the user, not by the program,
and because characters will frequently be scanned sequentially,
files are stored as flat objects.
Caches help keep performance good and linear when working with such
data.
.PP
Second, the
.CW Rasp
and several of the other caches have some
.I read-ahead;
that is, the cache is loaded with more information than is needed for
the job immediately at hand.
When manipulating linear structures, the accesses are usually sequential,
and read-ahead can significantly reduce the average time to access the
next element of the object.
Sequential access is a common mode for people as well as programs;
consider scrolling through a document while looking for something.
.PP
Finally, like any good data structure,
the cache guides the algorithm, or at least the implementation.
The
.CW Rasp
was actually invented to control the communications between the host and
terminal parts, but I realized very early that it was also a form of
cache.  Other caches were more explicitly intended to serve a double
purpose: for example, the caches in
.CW Files
that coalesce updates not only reduce traffic to the
transcript and contents
.CW Buffers ,
they also clump screen updates so that complicated changes to the
screen are achieved in
just a few messages to the terminal.
This saved me considerable work: I did not need to write special
code to optimize the message traffic to the
terminal.
Caches pay off in surprising ways.
Also, they tend to be independent, so their performance improvements
are multiplicative.
.SH 2
Data structures in the terminal
.LP
The terminal's job is to display and to maintain a consistent image of
pieces of the files being edited.
Because the text is always in memory, the data structures are
considerably simpler than those in the host part.
.PP
.CW Sam
typically has far more windows than does
.CW mux ,
the window system within which its Blit implementation runs.
.CW Mux
has a fairly small number of asynchronously updated windows;
.CW sam
needs a large number of synchronously updated windows that are
usually static and often fully obscured.
The different tradeoffs guided
.CW sam
away from the memory-intensive implementation of windows, called
.CW Layers ,\u\s-4\&17\s+4\d
used in
.CW mux.
Rather than depending on a complete bitmap image of the display for each window,
.CW sam
regenerates the image from its in-memory text
(stored in the
.CW Rasp )
when necessary, although it will use such an image if it is available.
Like
.CW Layers ,
though,
.CW sam
uses the screen bitmap as active storage in which to update the image using
.CW bitblt .\u\s-4\&18,19\s+4\d
The resulting organization, pictured in Figure 6,
has a global array of windows, called
.CW Flayers ,
each of which holds an image of a piece of text held in a data structure
called a
.CW Frame ,
which in turn represents
a rectangular window full of text displayed in some
.CW Bitmap .
Each
.CW Flayer
appears in a global list that orders them all front-to-back
on the display, and simultaneously as an element of a per-file array
that holds all the open windows for that file.
The complement in the terminal of the
.CW File
on the host is called a
.CW Text ;
each connects its
.CW Flayers
to the associated
.CW Rasp .
.KF
.PS
copy "fig6.pic"
.PE
.Cs
Figure 6. Data structures in the terminal.
.CW Flayers
are also linked together into a front-to-back list.
.CW Boxes
are discussed in the next section.
.Ce
.KE
.PP
The
.CW Bitmap
for a
.CW Frame
contains the image of the text.
For a fully visible window, the
.CW Bitmap
will be the screen (or at least the
.CW Layer
in which
.CW sam
is being run),
while for partially obscured windows the
.CW Bitmap
will be off-screen.
If the window is fully obscured, the
.CW Bitmap
will be null.
.PP
The
.CW Bitmap
is a kind of cache.
When making changes to the display, most of the original image will
look the same in the final image, and the update algorithms exploit this.
The
.CW Frame
software updates the image in the
.CW Bitmap
incrementally; the
.CW Bitmap
is not just an image, it is a data structure.\u\s-4\&18,19\s+4\d
The job of the software that updates the display is therefore
to use as much as possible of the existing image (converting the
text from ASCII characters to pixels is expensive) in a sort of two-dimensional
string insertion algorithm.
The details of this process are described in the next section.
.PP
The
.CW Frame
software has no code to support overlapping windows;
its job is to keep a single
.CW Bitmap
up to date.
It falls to the
.CW Flayer
software to multiplex the various
.CW Bitmaps
onto the screen.
The problem of maintaining overlapping
.CW Flayers
is easier than for
.CW Layers \u\s-4\&17\s+4\d
because changes are made synchronously and because the contents of the window
can be reconstructed from the data stored in the
.CW Frame ;
the
.CW Layers
software
makes no such assumptions.
In
.CW sam ,
the window being changed is almost always fully visible, because the current
window is always fully visible, by construction.
However, when multi-file changes are being made, or when
more than one window is open on a file,
it may be necessary to update partially obscured windows.
.PP
There are three cases: the window is 
fully visible, invisible (fully obscured), or partially visible.
If fully visible, the
.CW Bitmap
is part of the screen, so when the
.CW Flayer
update routine calls the
.CW Frame
update routine, the screen will be updated directly.
If the window is invisible,
there is no associated
.CW Bitmap ,
and all that is necessary is to update the
.CW Frame
data structure, not the image.
If the window is partially visible, the
.CW Frame
routine is called to update the image in the off-screen
.CW Bitmap ,
which may require regenerating it from the text of the window.
The
.CW Flayer
code then clips this
.CW Bitmap
against the
.CW Bitmaps
of all
.CW Frames
in front of the
.CW Frame
being modified, and the remainder is copied to the display.
.PP
This is much faster than recreating the image off-screen
for every change, or clipping all the changes made to the image
during its update.
Unfortunately, these caches can also consume prohibitive amounts of
memory, so they are freed fairly liberally \(em after every change to the
front-to-back order of the
.CW Flayers .
The result is that
the off-screen
.CW Bitmaps
exist only while multi-window changes are occurring,
which is the only time the performance improvement they provide is needed.
Also, the user interface causes fully-obscured windows to be the
easiest to make \(em
creating a canonically sized and placed window requires only a button click
\(em which reduces the need for caching still further.
.PP
.SH 2
Screen update
.LP
Only two low-level primitives are needed for incremental update:
.CW bitblt ,
which copies rectangles of pixels, and
.CW string
(which in turn calls
.CW bitblt ),
which draws a null-terminated character string in a
.CW Bitmap .
A
.CW Frame
contains a list of
.CW Boxes ,
each of which defines a horizontal strip of text in the window
(see Figure 7).
A
.CW Box
has a character string
.CW str ,
and a
.CW Rectangle
.CW rect
that defines the location of the strip in the window.
(The text in
.CW str
is stored in the
.CW Box
separately from the
.CW Rasp
associated with the window's file, so
.CW Boxes
are self-contained.)
The invariant is that
the image of the
.CW Box
can be reproduced by calling
.CW string
with argument
.CW str
to draw the string in
.CW rect ,
and the resulting picture fits perfectly within
.CW rect .
In other words, the
.CW Boxes
define the tiling of the window.
The tiling may be complicated by long lines of text, which
are folded onto the next line.
Some editors use horizontal scrolling to avoid this complication,
but to be comfortable this technique requires that lines not be
.I too
long;
.CW sam
has no such restriction.
Also, and perhaps more importantly, UNIX programs and terminals traditionally fold
long lines to make their contents fully visible.
.PP
Two special kinds of
.CW Boxes
contain a single
character: either a newline or a tab.
Newlines and tabs are white space.
A newline
.CW Box
always extends to the right edge of the window,
forcing the following
.CW Box
to the next line.
The width of a tab depends on where it is located:
it forces the next
.CW Box
to begin at a tab location.
Tabs also
have a minimum width equivalent to a blank (blanks are
drawn by
.CW string
and are not treated specially); newlines have a minimum width of zero.
.KF
.PS
copy "fig7.pic"
.PE
.sp .5
.Cs
Figure 7. A line of text showing its
.CW Boxes .
The first two blank
.CW Boxes
contain tabs; the last contains a newline.
Spaces are handled as ordinary characters.
.Ce
.KE
.PP
The update algorithms always use the
.CW Bitmap
image of the text (either the display or cache
.CW Bitmap );
they never examine the characters within a
.CW Box
except when the
.CW Box
needs to be split in two.
Before a change, the window consists of a tiling of
.CW Boxes ;
after the change the window is tiled differently.
The update algorithms rearrange the tiles in place, without
backup storage.
The algorithms are not strictly optimal \(em for example, they can
clear a pixel that is later going to be written upon \(em
but they never move a tile that doesn't need to be moved,
and they move each tile at most once.
.CW Frinsert
on a Blit can absorb over a thousand characters a second if the strings
being inserted are a few tens of characters long.
.PP
Consider
.CW frdelete .
Its job is to delete a substring from a
.CW Frame
and restore the image of the
.CW Frame .
The image of a substring has a peculiar shape (see Figure 2) comprising
possibly a partial line,
zero or more full lines,
and possibly a final partial line.
For reference, call this the
.I
Z-shape.
.R
.CW Frdelete
begins by splitting, if necessary, the
.CW Boxes
containing the ends of
the substring so the substring begins and ends on
.CW Box
boundaries.
Because the substring is being deleted, its image is not needed,
so the Z-shape is then cleared.
Then, tiles (that is, the images of
.CW Boxes )
are copied, using
.CW bitblt ,
from immediately after the Z-shape to
the beginning of the Z-shape,
resulting in a new Z-shape.
.CW Boxes "" (
whose contents would span two lines in the new position must first be split.)
.PP
Copying the remainder of the
.CW Frame
tile by tile
this way will clearly accomplish the deletion but eventually,
typically when the copying algorithm encounters a tab or newline,
the old and new
.CW x
coordinates of the tile
to be copied are the same.
This correspondence implies
that the Z-shape has its beginning and ending edges aligned
vertically, and a sequence of at most two
.CW bitblts
can be used to copy the remaining tiles.
The last step is to clear out the resulting empty space at the bottom
of the window;
the number of lines to be cleared is the number of complete lines in the
Z-shape closed by the final
.CW bitblts.
The final step is to merge horizontally adjacent
.CW Boxes
of plain text.
The complete source to
.CW frdelete
is less than 100 lines of C.
.PP
.CW frinsert
is more complicated because it must do four passes:
one to construct the
.CW Box
list for the inserted string,
one to reconnoitre,
one to copy (in opposite order to
.CW frdelete )
the
.CW Boxes
to make the hole for the new text,
and finally one to copy the new text into place.
Overall, though,
.CW frinsert
has a similar flavor to
.CW frdelete ,
and needn't be described further.
.CW Frinsert
and its subsidiary routines comprise 211 lines of C.
.PP
The terminal source code is 3024 lines of C,
and the host source is 5797 lines.
.SH
Discussion
.SH 2
History
.LP
The immediate ancestor of
.CW sam
was the original text editor for the Blit, called
.CW jim .
.CW Sam
inherited
.CW jim 's
two-process structure and mouse language almost unchanged, but
.CW jim
suffered from several drawbacks that were addressed in the design of
.CW sam .
The most important of these was the lack of a command language.
Although
.CW jim
was easy to use for simple editing, it provided no direct help with
large or repetitive editing tasks.  Instead, it provided a command to pass
selected text through a shell pipeline,
but this was no more satisfactory than could be expected of a stopgap measure.
.PP
.CW Jim
was written primarily as a vehicle for experimenting with a mouse-based
interface to text, and the experiment was successful.
.CW Jim
had some spin-offs:
.CW mux ,
the second window system for the Blit, is essentially a multiplexed
version of the terminal part of
.CW jim ;
and the debugger
.CW pi 's
user interface\u\s-4\&20\s+4\d was closely modeled on
.CW jim 's.
But after a couple of years,
.CW jim
had become difficult to maintain and limiting to use,
and its replacement was overdue.
.PP
I began the design of
.CW sam
by asking
.CW jim
customers what they wanted.
This was probably a mistake; the answers were essentially a list of features
to be found in other editors, which did not provide any of the
guiding principles I was seeking.
For instance, one common request was for a ``global substitute,''
but no one suggested how to provide it within a cut-and-paste editor.
I was looking for a scheme that would
support such specialized features comfortably in the context of some
general command language.
Ideas were not forthcoming, though, particularly given my insistence
on removing all limits on file sizes, line lengths and so on.
Even worse, I recognized that, since the mouse could easily
indicate a region of the screen that was not an integral number of lines,
the command language would best forget about newlines altogether,
and that meant the command language had to treat the file as a single
string, not an array of lines.
.PP
Eventually, I decided that thinking was not getting me very far and it was
time to try building.
I knew that the terminal part could be built easily \(em
that part of
.CW jim
behaved acceptably well \(em and that most of the hard work was going
to be in the host part: the file interface, command interpreter and so on.
Moreover, I had some ideas about how the architecture of
.CW jim
could be improved without destroying its basic structure, which I liked
in principle but which hadn't worked out as well as I had hoped.
So I began by designing the file data structure,
starting with the way
.CW jim
worked \(em comparable to a single structure merging
.CW Disc
and
.CW Buffer ,
which I split to make the cache more general
\(em and thinking about how global substitute could be implemented.
The answer was clearly that it had to be done in two passes,
and the transcript-oriented implementation fell out naturally.
.PP
.CW Sam
was written bottom-up,
starting from the data structures and algorithms for manipulating text,
through the command language and up to the code for maintaining
the display.
In retrospect, it turned out well, but this implementation method is
not recommended in general.
There were several times when I had a large body of interesting code
assembled and no clue how to proceed with it.
The command language, in particular, took almost a year to figure out,
but can be implemented (given what was there at the beginning of that year)
in a day or two.  Similarly, inventing the
.CW Rasp
data structure delayed the
connection of the host and terminal pieces by another few months.
.CW Sam
took about two years to write, although only about four months were
spent actually working on it.
.PP
Part of the design process was unusual:
the subset of the protocol that maintains the
.CW Rasp
was simulated, debugged
and verified by an automatic protocol analyzer,\u\s-4\&21\s+4\d and was bug-free
from the start.
The rest of the protocol, concerned mostly
with keeping menus up to date,
was unfortunately too unwieldy for such analysis,
and was debugged by more traditional methods, primarily
by logging in a file all messages in and out of the host.
.SH 2
Reflections
.LP
.CW Sam
is essentially the only interactive editor used by the sixty or so members of
the computing science research center in which I work.
The same could not be said of
.CW jim ;
the lack of a command language kept some people from adopting it.
The union of a user interface as comfortable as
.CW jim 's
with a command language as powerful as
.CW ed 's†
.FS
.vs 9
†The people who criticize
.CW ed
as an interactive program often forget that it and its close relative
.CW sed \u\s-4\&7\s+4\d
still thrive as programmable editors.  The strength of these programs is
independent of their convenience for interactive editing.
.br
.vs
.FE
is essential to
.CW sam 's
success.
When
.CW sam
was first made available to the
.CW jim
community,
almost everyone switched to it within two or three days.
In the months that followed, even people who had never adopted
.CW jim
started using
.CW sam
exclusively.
.PP
To be honest,
.CW ed
still gets occasional use, but usually when
something quick needs to be done and the overhead of
downloading the terminal part of
.CW sam
isn't worth the trouble.
Also, as a `line' editor,
.CW sam
.CW -d
is a bit odd;
when using a good old ASCII terminal, it's comforting to have
a true line editor.
But it is fair to say that
.CW sam 's
command language has displaced
.CW ed 's
for most of the complicated editing that has kept line editors
(that is, command-driven editors) with us.
.PP
.CW Sam 's
command language is even fancier than
.CW ed 's,
and most
.CW sam
customers don't come near to using all its capabilities.
Does it need to be so sophisticated?
I think the answer is yes, for two reasons.
.PP
First, the
.I model
for
.CW sam 's
command language is really relatively simple, and certainly simpler than that of
.CW ed .
For instance, there is only one kind of textual loop in
.CW sam
\(em the
.CW x
command \(em
while
.CW ed
has three (the
.CW g
command, the global flag on substitutions, and the implicit loop over
lines in multi-line substitutions).
Also,
.CW ed 's
substitute command is necessary to make changes within lines, but in
.CW sam
the
.CW s
command is more of a familiar convenience than a necessity;
.CW c
and
.CW t
can do all the work.
.PP
Second,
given a community that expects an editor to be about as powerful as
.CW ed ,
it's hard to see how
.CW sam
could really be much simpler and still satisfy that expectation.
People want to do ``global substitutes,'' and most are content
to have the recipe for that and a few other fancy changes.
The sophistication of the command language is really just a veneer
over a design that makes it possible to do global substitutes
in a screen editor.
Some people will always want something more, however, and it's gratifying to
be able to provide it.
The real power of
.CW sam 's
command language comes from composability of the operators, which is by
nature orthogonal to the underlying model.
In other words,
.CW sam
is not itself complex, but it makes complex things possible.
If you don't want to do anything complex, you can ignore the
complexity altogether, and many people do so.
.PP
Sometimes I am asked the opposite question: why didn't I just make
.CW sam
a real programmable editor, with macros and variables and so on?
The main reason is a matter of taste: I like the editor
to be the same every time I use it.
There is one technical reason, though:
programmability in editors is largely a workaround for insufficient
interactivity.
Programmable editors are used to make particular, usually short-term,
things easy to do, such as by providing shorthands for common actions.
If things are generally easy to do in the first place,
shorthands are not as helpful.
.CW Sam
makes common editing operations very easy, and the solutions to
complex editing problems seem commensurate with the problems themselves.
Also, the ability to edit the
.CW sam
window makes it easy to repeat commands \(em it only takes a mouse button click
to execute a command again.
.SH 2
Pros and cons
.LP
.CW Sam
has several other good points,
and its share of problems.
Among the good things is the idea of
structural regular expressions,
whose usefulness has only begun to be explored.
They were arrived at serendipitously when I attempted to distill the essence of
.CW ed 's
way of doing global substitution and recognized that the looping command in
.CW ed
was implicitly imposing a structure (an array of lines) on the file.
.PP
Another of
.CW sam 's
good things is its undo capability.
I had never before used an editor with a true undo,
but I would never go back now.
Undo
.I must
be done well, but if it is, it can be relied on.
For example,
it's safe to experiment if you're not sure how to write some intricate command,
because if you make a mistake, it can be fixed simply and reliably.
I learned two things about undo from writing
.CW sam :
first, it's easy to provide if you design it in from the beginning, and
second, it's necessary, particularly if the system has some subtle
properties that may be unfamiliar or error-prone for users.
.PP
.CW Sam 's
lack of internal limits and sizes is a virtue.
Because it avoids all fixed-size tables and data structures,
.CW sam
is able to make global changes to files that some of our other
tools cannot even read.
Moreover, the design keeps the performance linear when doing such
operations, although I must admit
.CW sam
does get slow when editing a huge file.
.PP
Now, the problems.
Externally, the most obvious is that it is poorly integrated into the
surrounding window system.
By design, the user interface in
.CW sam
feels almost identical to that of
.CW mux ,
but a thick wall separates text in
.CW sam
from the programs running in
.CW mux .
For instance, the `snarf buffer' in
.CW sam
must be maintained separately from that in
.CW mux .
This is regrettable, but probably necessary given the unusual configuration
of the system, with a programmable terminal on the far end of an RS-232 link.
.PP
.CW Sam
is reliable; otherwise, people wouldn't use it.
But it was written over such a long time, and has so many new (to me)
ideas in it, that I would like to see it done over again to clean
up the code and remove many of the lingering problems in the implementation.
The worst part is in the interconnection of the host and terminal parts,
which might even be able to go away in a redesign for a more
conventional window system.
The program must be split in two to use the terminal effectively,
but the low bandwidth of the connection forces the separation to
occur in an inconvenient part of the design if performance is to be acceptable.
A simple remote procedure call
protocol driven by the host, emitting only graphics
commands, would be easy to write but wouldn't have nearly the
necessary responsiveness.  On the other hand, if the terminal were in control
and requested much simpler file services from the host, regular expression
searches would require that the terminal read the entire file over its RS-232
link, which would be unreasonably slow.
A compromise in which either end can take control is necessary.
In retrospect, the communications protocol should have been
designed and verified formally, although I do not know of any tool
that can adequately relate the protocol to
its implementation.
.PP
Not all of
.CW sam 's
users are comfortable with its command language, and few are adept.
Some (venerable) people use a sort of
.CW ed \& ``
subset'' of
.CW sam 's
command language,
and even ask why
.CW sam 's
command language is not exactly
.CW ed 's.
(The reason, of course, is that
.CW sam 's
model for text does not include newlines, which are central to
.CW ed .
Making the text an array of newlines to the command language would
be too much of a break from the seamless model provided by the mouse.
Some editors, such as
.CW vi ,
are willing to make this break, though.)
The difficulty is that
.CW sam 's
syntax is so close to
.CW ed 's
that people believe it
.I should
be the same.
I thought, with some justification in hindsight,
that making
.CW sam
similar to
.CW ed
would make it easier to learn and to accept.
But I may have overstepped and raised the users'
expectations too much.
It's hard to decide which way to resolve this problem.
.PP
Finally, there is a tradeoff in
.CW sam
that was decided by the environment in which it runs:
.CW sam
is a multi-file editor, although in a different system there might instead be
multiple single-file editors.
The decision was made primarily because starting a new program in a Blit is
time-consuming.
If the choice could be made freely, however, I would
still choose the multi-file architecture, because it allows
groups of files to be handled as a unit;
the usefulness of the multi-file commands is incontrovertible.
It is delightful to have the source to an entire program
available at your fingertips.
.SH
Acknowledgements
.LP
Tom Cargill suggested the idea behind the
.CW Rasp
data structure.
Norman Wilson and Ken Thompson influenced the command language.
This paper was improved by comments from
Al Aho,
Jon Bentley,
Chris Fraser,
Gerard Holzmann,
Brian Kernighan,
Ted Kowalski,
Doug McIlroy
and
Dennis Ritchie.