Mk is a tool for describing and maintaining dependencies between files. It is similar to the UNIX program make, but provides several extensions. Mk's flexible rule specifications, implied dependency derivation, and parallel execution of maintenance actions are well-suited to the Plan 9 environment. Almost all Plan 9 maintenance procedures are automated using mk.
This document describes how mk, a program functionally similar to make [Feld79], is used to maintain dependencies between files in Plan 9. Mk provides several extensions to the capabilities of its predecessor that work well in Plan 9's distributed, multi-architecture environment. It exploits the power of multiprocessors by executing maintenance actions in parallel and interacts with the Plan 9 command interpreter rc to provide a powerful set of maintenance tools. It accepts pattern-based dependency specifications that are not limited to describing rules for program construction. The result is a tool that is flexible enough to perform many maintenance tasks including database maintenance, hardware design, and document production.
This document begins by discussing the syntax of the control file, the pattern matching capabilities, and the special rules for maintaining archives. A brief description of mk's algorithm for deriving dependencies is followed by a discussion of the conventions used to resolve ambiguous specifications. The final sections describe parallel execution and special features.
An earlier paper [Hume87] provides a detailed discussion of mk's design and an appendix summarizes the differences between mk and make.
Mk reads a file describing relationships among files and executes commands to bring the files up to date. The specification file, called a mkfile, contains three types of statements: assignments, includes, and rules. Assignment and include statements are similar to those in C. Rules specify dependencies between a target and its prerequisites. When the target and prerequisites are files, their modification times determine if they are out of date. Rules often contain a recipe, an rc(1) script that produces the target from the prerequisites.
This simple mkfile produces an executable from a C source file:
CC=pcc f1: f1.c $CC -o f1 f1.c
The native Plan 9 environment requires executables for all architectures, not only the current one. The Plan 9 version of the same mkfile looks like:
</$objtype/mkfile f1: f1.$O $LD $LDFLAGS -o f1 f1.$O f1.$O: f1.c $CC $CFLAGS f1.c
% objtype=mips mk vc -w f1.c vl $LDFLAGS -o f1 f1.k %
We can extend the mkfile to build two programs:
</$objtype/mkfile ALL=f1 f2 all:V: $ALL f1: f1.$O $LD $LDFLAGS -o f1 f1.$O f1.$O: f1.c $CC $CFLAGS f1.c f2: f2.$O $LD $LDFLAGS -o f2 f2.$O f2.$O: f2.c $CC $CFLAGS f2.c
Mk does not distinguish between its internal variables and rc variables in the environment. When mk starts, it imports each environment variable into a mk variable of the same name. Before executing a recipe, mk exports all variables, including those inherited from the environment, to the environment in which rc executes the recipe.
There are several ways for a variable to take a value. It can be set with an assignment statement, inherited from the environment, or specified on the command line. Mk also maintains several special internal variables that are described in mk(1). Assignments have the following decreasing order of precedence:
All variable values are strings. They can be used for pattern matching and comparison but not for arithmetic. A list is a string containing several values separated by white space. Each member is handled individually during pattern matching, target selection, and prerequisite evaluation.
A namelist is a list produced by transforming the members of an existing list. The transform applies a pattern to each member, replacing each matched string with a new string, much as in the substitute command in sam(1) or ed(1). The syntax is
${var:A%B=C%D}
s/A(.*)B/C\1D/
Namelists are useful for generating a list based on a predictable transformation. For example,
SRC=a.c b.c c.c OBJ=${SRC:%.c=%.v}
Command output is assigned to a variable using the normal rc syntax:
var=`{rc command}
TARG=`{ls -d *.[cy] | sed 's/..$//'}
The include statement replaces itself with the contents of a file. It is functionally similar to the C #include statement but uses a different syntax:
<filename
Unlike make, mk has no built-in rules. Instead, the include statement allows generic rules to be imported from a prototype mkfile; most Plan 9 mkfiles use this approach [Flan95].
A rule has four elements: targets, prerequisites, attributes, and a recipe. It has the form:
targets:attributes:prerequisites recipe
Normally the target is a file that depends on one or more prerequisite files. Mk compares the modification times of each target and each prerequisite; a target is considered out of date when it does not exist or when a prerequisite has been modified more recently. When a target is out of date, mk executes the recipe to bring it up to date. When the recipe completes, the modification time of the target is checked and used in later dependency evaluations. If the recipe does not update the target, evaluation continues with the out of date target.
A prerequisite of one rule may be the target of another. When this happens, the rules cascade to define a multi-step procedure. For example, an executable target depends on prerequisite object files, each of which is a target in a rule with a C source file as the prerequisite. Mk follows a chain of dependencies until it encounters a prerequisite that is not a target of another rule or it finds a target that is up to date. It then executes the recipes in reverse order to produce the desired target.
The rule header is evaluated when the rule is read. Variables are replaced by their values, namelists are generated, and commands are replaced by their output at this time.
Most attributes modify mk's evaluation of a rule. An attribute is usually a single letter but some are more complicated. This paper only discusses commonly used attributes; see mk(1) for a complete list.
The V attribute identifies a virtual target; that is, a target that is not a file. For example,
clean:V: rm *.$O $O.out
default:QV: echo 'No default target; use mk all or mk install'
The recipe is an rc script. It is optional but when it is missing, the rule is handled specially, as described later. Unlike make, mk executes recipes without interpretation. After stripping the first white space character from each line it passes the entire recipe to rc on standard input. Since mk does not interpret a recipe, escape conventions are exactly those of rc. Scripts for awk and sed commands can be embedded exactly as they would be entered from the command line. Mk invokes rc with the -e flag, which causes rc to stop if any command in the recipe exits with a non-zero status; the E attribute overrides this behavior and allows rc to continue executing in the face of errors. Before a recipe is executed, variables are exported to the environment where they are available to rc. Commands in the recipe may not read from standard input because mk uses it internally.
References to a variable can yield different values depending on the location of the reference in the mkfile. Mk resolves variable references in assignment statements and rule headers when the statement is read. Variable references in recipes are evaluated by rc when the recipe is executed; this happens after the entire mkfile has been read. The value of a variable in a recipe is the last value assigned in the file. For example,
STRING=all all:VQ: echo $STRING STRING=none
A metarule is a rule based on a pattern. The pattern selects a class of target(s) and identifies related prerequisites. Mk metarules may select targets and prerequisites based on any criterion that can be described by a pattern, not just the suffix transformations associated with program construction.
Metarule patterns are either intrinsic or regular expressions conforming to the syntax of regexp(6). The intrinsic patterns are shorthand for common regular expressions. The intrinsic pattern % matches one or more of anything; it is equivalent to the regular expression `.+'. The other intrinsic pattern, &, matches one or more of any characters except `/' and `.'. It matches a portion of a path and is equivalent to the regular expression `[^./]+'. An intrinsic pattern in a prerequisite references the string matched by the same intrinsic pattern in the target. For example, the rule
%.v: %.c
%.$O: %.c $CC $CFLAGS $stem.c
Metarules simplify the mkfile for building programs f1 and f2:
</$objtype/mkfile ALL=f1 f2 all:V: $ALL %: %.$O $LD -o $target $prereq %.$O: %.c $CC $CFLAGS $stem.c clean:V: rm -f $ALL *.[$OS]
A regular expression metarule must have an R attribute. Prerequisites may reference matching substrings in the target using the form n where n is a digit from 1 to 9 specifying the nth parenthesized sub-expression. In a recipe, $stemn is the equivalent reference. For example, a compile rule could be specified using regular expressions:
(.+)\.$O:R: \1.c $CC $CFLAGS $stem1.c
Mk provides a special mechanism for maintaining an archive. An archive member is referenced using the form lib(file) where lib is the name of the archive and file is the name of the member. Two rules define the dependency between an object file and its membership in an archive:
$LIB(foo.8):N: foo.8 $LIB: $LIB(foo.8) ar rv $LIB foo.8
A metarule generalizes library maintenance:
LIB=lib.a OBJS=etoa.$O atoe.$O ebcdic.$O $LIB(%):N: % $LIB: ${OBJS:%=$LIB(%)} ar rv $LIB $OBJS
$LIB: ${OBJS:%=$LIB(%)} ar rv $LIB `{membername $newprereq}
The mkfile
</$objtype/mkfile LIB=lib.a OBJS=etoa.$O atoe.$O ebcdic.$O prog: main.$O $LIB $LD -o $target $prereq $LIB(%):N: % $LIB: ${OBJS:%=$LIB(%)} ar rv $LIB $OBJS
For each target of interest, mk uses the rules in a mkfile to build a data structure called a dependency graph. The nodes of the graph represent targets and prerequisites; a directed arc from one node to another indicates that the file associated with the first node depends on the file associated with the second. When the mkfile has been completely read, the graph is analyzed. In the first step, implied dependencies are resolved by computing the transitive closure of the graph. This calculation extends the graph to include all targets that are potentially derivable from the rules in the mkfile. Next the graph is checked for cycles; make accepts cyclic dependencies, but mk does not allow them. Subsequent steps prune subgraphs that are irrelevant for producing the desired target and verify that there is only one way to build it. The recipes associated with the nodes on the longest path between the target and an out of date prerequisite are then executed in reverse order.
The transitive closure calculation is sensitive to metarules; the patterns often select many potential targets and cause the graph to grow rapidly. Fortunately, dependencies associated with the desired target usually form a small part of the graph, so, after pruning, analysis is tractable. For example, the rules
%: x.% recipe1 x.%: %.k recipe2 %.k: %.f recipe3
Mk avoids infinite cycles by evaluating each metarule once. Thus, the rule
%: %.z cp $prereq $prereq.z
There must be only one way to build each target. However, during evaluation metarule patterns often select potential targets that conflict with the targets of other rules. Mk uses several conventions to resolve ambiguities and to select the proper dependencies.
When a target selects more than one rule, mk chooses a regular rule over a metarule. For example, the mkfile
</$objtype/mkfile FILES=f1.$O f2.$O f3.$O prog: $FILES $LD -o $target $prereq %.$O: %.c $CC $CFLAGS $stem.c f2.$O: f2.c $CC f2.c
When a rule has a target and prerequisites but no recipe, those prerequisites are added to all other rules with recipes that have the same target. All prerequisites, regardless of where they were specified, are exported to the recipe in variable $prereq. For example, in
</$objtype/mkfile FILES=f1.$O f2.$O f3.$O prog: $FILES $LD -o $target $prereq %.$O: hdr.h %.$O: %.c $CC $CFLAGS $stem.c
When a target is virtual and there is no other rule with the same target, mk evaluates each prerequisite. For example, adding the rule
all:V: prog
When two rules have identical rule headers and both have recipes, the later rule replaces the former one. For example, if a file named mkrules contains
$O.out: $OFILES $LD $LFLAGS $OFILES %.$O: %.c $CC $CFLAGS $stem.c
OFILES=f1.$O f2.$O f3.$O <mkrules $O.out: $OFILES $LD $LFLAGS -l $OFILES -lbio -lc
<mkrules $O.out:Q: $OFILES ;
When a rule has no prerequisites, the recipe is executed only when the target does not exist. For example,
marker: touch $target
clean:V: rm -f [$OS].out *.[$OS]
clean tidy nuke:V: rm -f [$OS].out *.[$OS]
A rule applies to a target only when its prerequisites exist or can be derived. More than one rule may have the same target as long as only one rule with a recipe remains applicable after the dependency evaluation completes. For example, consider a program built from C and assembler source files. Two rules produce object files:
%.$O: %.c $CC $CFLAGS $stem.c %.$O: %.s $AS $AFLAGS $stem.s
In Plan 9, many programs consist of portable code stored in one directory and architecture-specific source stored in another. For example, the mkfile
</$objtype/mkfile FILES=f1.$O f2.$O f3.$O f3.$O prog: $FILES $LD -o $target $prereq %.$O: %.$c $CC $CFLAGS $stem.c %.$O: ../port/%.c $CC $CFLAGS ../port/$stem.c
f2.$O: f2.c $CC $CFLAGS $f2.c
Mk's heuristics can produce unintended results when rules are not carefully specified. For example, the rules that build object files from C or assembler source files
%.$O: %.c $CC $CFLAGS $stem.c %.$O: %.s $AS $AFLAGS $stem.s
%.$O: %.c hdr.h $CC $CFLAGS $stem.c
don't know how to make 'file.c'
%.$O: hdr.h %.$O: %.c $CC $CFLAGS $stem.c
Metarule patterns should be as restrictive as possible to prevent conflicts with other rules. Consider the mkfile
</$objtype/mkfile BIN=/$objtype/bin PROG=foo install:V: $BIN/$PROG %: %.c $CC $stem.c $LD -o $target $stem.$O $BIN/%: % mv $stem $target
mk: ambiguous recipes for /mips/bin/foo: /mips/bin/foo <-(mkfile:8)- /mips/bin/foo.c <-(mkfile:12)- foo.c /mips/bin/foo <-(mkfile:12)- foo <-(mkfile:8)- foo.c
&: &.c $CC $stem.c $LD -o $target $stem.$O
Mk does not build a missing intermediate file if a target is up to date with the prerequisites of the intermediate. For example, when an executable is up to date with its source file, mk does not compile the source to create a missing object file. The evaluation only applies when a target is considered up to date by pretending that the intermediate exists. Thus, it does not apply when the intermediate is a command line target or when it has no prerequisites.
This capability is useful for maintaining archives. We can modify the archive update recipe to remove object files after they are archived:
$LIB(%):N: % $LIB: ${OBJS:%=$LIB(%)} names=`{membername $newprereq} ar rv $LIB $names rm -f $names
Sometimes the modification time is not useful for deciding when a target and prerequisite are out of date. The P attribute replaces the default mechanism with the result of a command. The command immediately follows the attribute and is repeatedly executed with each target and each prerequisite as its arguments; if its exit status is non-zero, they are considered out of date and the recipe is executed. Consider the mkfile
foo.ref:Pcmp -s: foo cp $prereq $target
cmp -s foo.ref foo
When possible, mk executes recipes in parallel. The variable $NPROC specifies the maximum number of simultaneously executing recipes. Normally it is imported from the environment, where the system has set it to the number of available processors. It can be decreased by assigning a new value and can be set to 1 to force single-threaded recipe execution. This is necessary when several targets access a common resource such as a status file or data base. When there is no dependency between targets, mk assumes the recipes can be executed concurrently. Normally, this allows multiple prerequisites to be built simultaneously; for example, the object file prerequisites of a load rule can be produced by compiling the source files in parallel. Mk does not define the order of execution of independent recipes. When the prerequisites of a rule are not independent, the dependencies between them should be specified in a rule or the mkfile should be single-threaded. For example, the archive update rules
$LIB(%):N: % $LIB: ${OBJS:%=$LIB(%)} ar rv $LIB `{membername $newprereq}
$LIB(%): % ar rv $LIB $stem
The $nproc environment variable contains a number associated with the processor executing a recipe. It can be used to create unique names when the recipe may be executing simultaneously on several processors. Other maintenance tools provide mechanisms to control recipe scheduling explicitly [Cmel86], but mk's general rules are sufficient for all but the most unusual cases.
The D attribute causes mk to remove the target file when a recipe terminates prematurely. The error message describing the termination condition warns of the deletion. A partially built file is doubly dangerous: it is not only wrong, but is also considered to be up to date so a subsequent mk will not rebuild it. For example,
pic.out:D: mk.ms pic $prereq | tbl | troff -ms > $target
mk: pic mk.ms | ... : exit status=rc 685: deleting 'pic.out'
The -w command line flag forces the files following the flag to be treated as if they were just modified. We can use this flag with a command that selects files to force a build based on the selection criterion. For example, if the declaration of a global variable named var is changed in a header file, all source files that reference it can be rebuilt with the command
$ mk -w`{grep -l var *.[cyl]}
There are many programs related to make, each choosing a different balance between specialization and generality. Mk emphasizes generality but allows customization through its pattern specifications and include facilities.
Plan 9 presents a difficult maintenance environment with its heterogeneous architectures and languages. Mk's flexible specification language and simple interaction with rc work well in this environment. As a result, Plan 9 relies on mk to automate almost all maintenance. Tasks as diverse as updating the network data base, producing the manual, or building a release are expressed as mk procedures.
The differences between mk and make are:
It is usually easy to convert a makefile to or from an equivalent mkfile.