# GYP (Generate Your Projects) Tests -- Status: Draft (as of 2009-08-18) Steven Knight _et al._ Modified: 2009-08-18 [TOC] ## Introduction This document describes the GYP testing infrastructure, as provided by the `TestGyp.py` module. These tests emphasize testing the _behavior_ of the various GYP-generated build configurations: Visual Studio, Xcode, SCons, Make, etc. The goal is _not_ to test the output of the GYP generators by, for example, comparing a GYP-generated Makefile against a set of known "golden" Makefiles (although the testing infrastructure could be used to write those kinds of tests). The idea is that the generated build configuration files could be completely written to add a feature or fix a bug so long as they continue to support the functional behaviors defined by the tests: building programs, shared libraries, etc. ## "Hello, world!" GYP test configuration Here is an actual test configuration, a simple build of a C program to print `"Hello, world!"`. ``` $ ls -l test/hello total 20 -rw-r--r-- 1 knight knight 312 Jul 30 20:22 gyptest-all.py -rw-r--r-- 1 knight knight 307 Jul 30 20:22 gyptest-default.py -rwxr-xr-x 1 knight knight 326 Jul 30 20:22 gyptest-target.py -rw-r--r-- 1 knight knight 98 Jul 30 20:22 hello.c -rw-r--r-- 1 knight knight 142 Jul 30 20:22 hello.gyp $ ``` The `gyptest-*.py` files are three separate tests (test scripts) that use this configuration. The first one, `gyptest-all.py`, looks like this: ``` #!/usr/bin/env python """ Verifies simplest-possible build of a "Hello, world!" program using an explicit build target of 'all'. """ import TestGyp test = TestGyp.TestGyp() test.run_gyp('hello.gyp') test.build_all('hello.gyp') test.run_built_executable('hello', stdout="Hello, world!\n") test.pass_test() ``` The test script above runs GYP against the specified input file (`hello.gyp`) to generate a build configuration. It then tries to build the `'all'` target (or its equivalent) using the generated build configuration. Last, it verifies that the build worked as expected by running the executable program (`hello`) that was just presumably built by the generated configuration, and verifies that the output from the program matches the expected `stdout` string (`"Hello, world!\n"`). Which configuration is generated (i.e., which build tool to test) is specified when the test is run; see the next section. Surrounding the functional parts of the test described above are the header, which should be basically the same for each test (modulo a different description in the docstring): ``` #!/usr/bin/env python """ Verifies simplest-possible build of a "Hello, world!" program using an explicit build target of 'all'. """ import TestGyp test = TestGyp.TestGyp() ``` Similarly, the footer should be the same in every test: ``` test.pass_test() ``` ## Running tests Test scripts are run by the `gyptest.py` script. You can specify (an) explicit test script(s) to run: ``` $ python gyptest.py test/hello/gyptest-all.py PYTHONPATH=/home/knight/src/gyp/trunk/test/lib TESTGYP_FORMAT=scons /usr/bin/python test/hello/gyptest-all.py PASSED $ ``` If you specify a directory, all test scripts (scripts prefixed with `gyptest-`) underneath the directory will be run: ``` $ python gyptest.py test/hello PYTHONPATH=/home/knight/src/gyp/trunk/test/lib TESTGYP_FORMAT=scons /usr/bin/python test/hello/gyptest-all.py PASSED /usr/bin/python test/hello/gyptest-default.py PASSED /usr/bin/python test/hello/gyptest-target.py PASSED $ ``` Or you can specify the `-a` option to run all scripts in the tree: ``` $ python gyptest.py -a PYTHONPATH=/home/knight/src/gyp/trunk/test/lib TESTGYP_FORMAT=scons /usr/bin/python test/configurations/gyptest-configurations.py PASSED /usr/bin/python test/defines/gyptest-defines.py PASSED . . . . /usr/bin/python test/variables/gyptest-commands.py PASSED $ ``` If any tests fail during the run, the `gyptest.py` script will report them in a summary at the end. ## Debugging tests Tests that create intermediate output do so under the gyp/out/testworkarea directory. On test completion, intermediate output is cleaned up. To preserve this output, set the environment variable PRESERVE=1. This can be handy to inspect intermediate data when debugging a test. You can also set PRESERVE\_PASS=1, PRESERVE\_FAIL=1 or PRESERVE\_NO\_RESULT=1 to preserve output for tests that fall into one of those categories. # Specifying the format (build tool) to use By default, the `gyptest.py` script will generate configurations for the "primary" supported build tool for the platform you're on: Visual Studio on Windows, Xcode on Mac, and (currently) SCons on Linux. An alternate format (build tool) may be specified using the `-f` option: ``` $ python gyptest.py -f make test/hello/gyptest-all.py PYTHONPATH=/home/knight/src/gyp/trunk/test/lib TESTGYP_FORMAT=make /usr/bin/python test/hello/gyptest-all.py PASSED $ ``` Multiple tools may be specified in a single pass as a comma-separated list: ``` $ python gyptest.py -f make,scons test/hello/gyptest-all.py PYTHONPATH=/home/knight/src/gyp/trunk/test/lib TESTGYP_FORMAT=make /usr/bin/python test/hello/gyptest-all.py PASSED TESTGYP_FORMAT=scons /usr/bin/python test/hello/gyptest-all.py PASSED $ ``` ## Test script functions and methods The `TestGyp` class contains a lot of functionality intended to make it easy to write tests. This section describes the most useful pieces for GYP testing. (The `TestGyp` class is actually a subclass of more generic `TestCommon` and `TestCmd` base classes that contain even more functionality than is described here.) ### Initialization The standard initialization formula is: ``` import TestGyp test = TestGyp.TestGyp() ``` This copies the contents of the directory tree in which the test script lives to a temporary directory for execution, and arranges for the temporary directory's removal on exit. By default, any comparisons of output or file contents must be exact matches for the test to pass. If you need to use regular expressions for matches, a useful alternative initialization is: ``` import TestGyp test = TestGyp.TestGyp(match = TestGyp.match_re, diff = TestGyp.diff_re)` ``` ### Running GYP The canonical invocation is to simply specify the `.gyp` file to be executed: ``` test.run_gyp('file.gyp') ``` Additional GYP arguments may be specified: ``` test.run_gyp('file.gyp', arguments=['arg1', 'arg2', ...]) ``` To execute GYP from a subdirectory (where, presumably, the specified file lives): ``` test.run_gyp('file.gyp', chdir='subdir') ``` ### Running the build tool Running the build tool requires passing in a `.gyp` file, which may be used to calculate the name of a specific build configuration file (such as a MSVS solution file corresponding to the `.gyp` file). There are several different `.build_*()` methods for invoking different types of builds. To invoke a build tool with an explicit `all` target (or equivalent): ``` test.build_all('file.gyp') ``` To invoke a build tool with its default behavior (for example, executing `make` with no targets specified): ``` test.build_default('file.gyp') ``` To invoke a build tool with an explicit specified target: ``` test.build_target('file.gyp', 'target') ``` ### Running executables The most useful method executes a program built by the GYP-generated configuration: ``` test.run_built_executable('program') ``` The `.run_built_executable()` method will account for the actual built target output location for the build tool being tested, as well as tack on any necessary executable file suffix for the platform (for example `.exe` on Windows). `stdout=` and `stderr=` keyword arguments specify expected standard output and error output, respectively. Failure to match these (if specified) will cause the test to fail. An explicit `None` value will suppress that verification: ``` test.run_built_executable('program', stdout="expect this output\n", stderr=None) ``` Note that the default values are `stdout=None` and `stderr=''` (that is, no check for standard output, and error output must be empty). Arbitrary executables (not necessarily those built by GYP) can be executed with the lower-level `.run()` method: ``` test.run('program') ``` The program must be in the local directory (that is, the temporary directory for test execution) or be an absolute path name. ### Fetching command output ``` test.stdout() ``` Returns the standard output from the most recent executed command (including `.run_gyp()`, `.build_*()`, or `.run*()` methods). ``` test.stderr() ``` Returns the error output from the most recent executed command (including `.run_gyp()`, `.build_*()`, or `.run*()` methods). ### Verifying existence or non-existence of files or directories ``` test.must_exist('file_or_dir') ``` Verifies that the specified file or directory exists, and fails the test if it doesn't. ``` test.must_not_exist('file_or_dir') ``` Verifies that the specified file or directory does not exist, and fails the test if it does. ### Verifying file contents ``` test.must_match('file', 'expected content\n') ``` Verifies that the content of the specified file match the expected string, and fails the test if it does not. By default, the match must be exact, but line-by-line regular expressions may be used if the `TestGyp` object was initialized with `TestGyp.match_re`. ``` test.must_not_match('file', 'expected content\n') ``` Verifies that the content of the specified file does _not_ match the expected string, and fails the test if it does. By default, the match must be exact, but line-by-line regular expressions may be used if the `TestGyp` object was initialized with `TestGyp.match_re`. ``` test.must_contain('file', 'substring') ``` Verifies that the specified file contains the specified substring, and fails the test if it does not. ``` test.must_not_contain('file', 'substring') ``` Verifies that the specified file does not contain the specified substring, and fails the test if it does. ``` test.must_contain_all_lines(output, lines) ``` Verifies that the output string contains all of the "lines" in the specified list of lines. In practice, the lines can be any substring and need not be `\n`-terminaed lines per se. If any line is missing, the test fails. ``` test.must_not_contain_any_lines(output, lines) ``` Verifies that the output string does _not_ contain any of the "lines" in the specified list of lines. In practice, the lines can be any substring and need not be `\n`-terminaed lines per se. If any line exists in the output string, the test fails. ``` test.must_contain_any_line(output, lines) ``` Verifies that the output string contains at least one of the "lines" in the specified list of lines. In practice, the lines can be any substring and need not be `\n`-terminaed lines per se. If none of the specified lines is present, the test fails. ### Reading file contents ``` test.read('file') ``` Returns the contents of the specified file. Directory elements contained in a list will be joined: ``` test.read(['subdir', 'file']) ``` ### Test success or failure ``` test.fail_test() ``` Fails the test, reporting `FAILED` on standard output and exiting with an exit status of `1`. ``` test.pass_test() ``` Passes the test, reporting `PASSED` on standard output and exiting with an exit status of `0`. ``` test.no_result() ``` Indicates the test had no valid result (i.e., the conditions could not be tested because of an external factor like a full file system). Reports `NO RESULT` on standard output and exits with a status of `2`.