|
@@ -1,44 +1,46 @@
|
|
|
+Guidelines for test developers
|
|
|
+==============================
|
|
|
+
|
|
|
How to add recipes
|
|
|
-==================
|
|
|
+------------------
|
|
|
|
|
|
For any test that you want to perform, you write a script located in
|
|
|
-test/recipes/, named {nn}-test_{name}.t, where {nn} is a two digit number and
|
|
|
-{name} is a unique name of your choice.
|
|
|
+`test/recipes/`, named `{nn}-test_{name}.t`,
|
|
|
+where `{nn}` is a two digit number and
|
|
|
+`{name}` is a unique name of your choice.
|
|
|
|
|
|
Please note that if a test involves a new testing executable, you will need to
|
|
|
-do some additions in test/build.info. Please refer to the section "Changes to
|
|
|
-test/build.info" below.
|
|
|
-
|
|
|
+do some additions in test/build.info. Please refer to the section
|
|
|
+["Changes to test/build.info"](README.md#changes-to-testbuildinfo) below.
|
|
|
|
|
|
Naming conventions
|
|
|
-=================
|
|
|
-
|
|
|
-A test executable is named test/{name}test.c
|
|
|
-
|
|
|
-A test recipe is named test/recipes/{nn}-test_{name}.t, where {nn} is a two
|
|
|
-digit number and {name} is a unique name of your choice.
|
|
|
-
|
|
|
-The number {nn} is (somewhat loosely) grouped as follows:
|
|
|
-
|
|
|
-00-04 sanity, internal and essential API tests
|
|
|
-05-09 individual symmetric cipher algorithms
|
|
|
-10-14 math (bignum)
|
|
|
-15-19 individual asymmetric cipher algorithms
|
|
|
-20-24 openssl commands (some otherwise not tested)
|
|
|
-25-29 certificate forms, generation and verification
|
|
|
-30-35 engine and evp
|
|
|
-60-79 APIs:
|
|
|
- 60 X509 subsystem
|
|
|
- 61 BIO subsystem
|
|
|
- 65 CMP subsystem
|
|
|
- 70 PACKET layer
|
|
|
-80-89 "larger" protocols (CA, CMS, OCSP, SSL, TSA)
|
|
|
-90-98 misc
|
|
|
-99 most time consuming tests [such as test_fuzz]
|
|
|
-
|
|
|
+------------------
|
|
|
+
|
|
|
+A test executable is named `test/{name}test.c`
|
|
|
+
|
|
|
+A test recipe is named `test/recipes/{nn}-test_{name}.t`, where `{nn}` is a two
|
|
|
+digit number and `{name}` is a unique name of your choice.
|
|
|
+
|
|
|
+The number `{nn}` is (somewhat loosely) grouped as follows:
|
|
|
+
|
|
|
+ 00-04 sanity, internal and essential API tests
|
|
|
+ 05-09 individual symmetric cipher algorithms
|
|
|
+ 10-14 math (bignum)
|
|
|
+ 15-19 individual asymmetric cipher algorithms
|
|
|
+ 20-24 openssl commands (some otherwise not tested)
|
|
|
+ 25-29 certificate forms, generation and verification
|
|
|
+ 30-35 engine and evp
|
|
|
+ 60-79 APIs:
|
|
|
+ 60 X509 subsystem
|
|
|
+ 61 BIO subsystem
|
|
|
+ 65 CMP subsystem
|
|
|
+ 70 PACKET layer
|
|
|
+ 80-89 "larger" protocols (CA, CMS, OCSP, SSL, TSA)
|
|
|
+ 90-98 misc
|
|
|
+ 99 most time consuming tests [such as test_fuzz]
|
|
|
|
|
|
A recipe that just runs a test executable
|
|
|
-=========================================
|
|
|
+-----------------------------------------
|
|
|
|
|
|
A script that just runs a program looks like this:
|
|
|
|
|
@@ -48,21 +50,20 @@ A script that just runs a program looks like this:
|
|
|
|
|
|
simple_test("test_{name}", "{name}test", "{name}");
|
|
|
|
|
|
-{name} is the unique name you have chosen for your test.
|
|
|
-
|
|
|
-The second argument to `simple_test' is the test executable, and `simple_test'
|
|
|
-expects it to be located in test/
|
|
|
+`{name}` is the unique name you have chosen for your test.
|
|
|
|
|
|
-For documentation on OpenSSL::Test::Simple, do
|
|
|
-`perldoc util/perl/OpenSSL/Test/Simple.pm'.
|
|
|
+The second argument to `simple_test` is the test executable, and `simple_test`
|
|
|
+expects it to be located in `test/`
|
|
|
|
|
|
+For documentation on `OpenSSL::Test::Simple`,
|
|
|
+do `perldoc util/perl/OpenSSL/Test/Simple.pm`.
|
|
|
|
|
|
A recipe that runs a more complex test
|
|
|
-======================================
|
|
|
+--------------------------------------
|
|
|
|
|
|
For more complex tests, you will need to read up on Test::More and
|
|
|
-OpenSSL::Test. Test::More is normally preinstalled, do `man Test::More' for
|
|
|
-documentation. For OpenSSL::Test, do `perldoc util/perl/OpenSSL/Test.pm'.
|
|
|
+OpenSSL::Test. Test::More is normally preinstalled, do `man Test::More` for
|
|
|
+documentation. For OpenSSL::Test, do `perldoc util/perl/OpenSSL/Test.pm`.
|
|
|
|
|
|
A script to start from could be this:
|
|
|
|
|
@@ -89,26 +90,25 @@ A script to start from could be this:
|
|
|
# test feature 2
|
|
|
}
|
|
|
|
|
|
-
|
|
|
Changes to test/build.info
|
|
|
-==========================
|
|
|
+--------------------------
|
|
|
|
|
|
Whenever a new test involves a new test executable you need to do the
|
|
|
following (at all times, replace {NAME} and {name} with the name of your
|
|
|
test):
|
|
|
|
|
|
-* add {name} to the list of programs under PROGRAMS_NO_INST
|
|
|
+ * add `{name}` to the list of programs under `PROGRAMS_NO_INST`
|
|
|
|
|
|
-* create a three line description of how to build the test, you will have
|
|
|
-to modify the include paths and source files if you don't want to use the
|
|
|
-basic test framework:
|
|
|
+ * create a three line description of how to build the test, you will have
|
|
|
+ to modify the include paths and source files if you don't want to use the
|
|
|
+ basic test framework:
|
|
|
|
|
|
- SOURCE[{name}]={name}.c
|
|
|
- INCLUDE[{name}]=.. ../include ../apps/include
|
|
|
- DEPEND[{name}]=../libcrypto libtestutil.a
|
|
|
+ SOURCE[{name}]={name}.c
|
|
|
+ INCLUDE[{name}]=.. ../include ../apps/include
|
|
|
+ DEPEND[{name}]=../libcrypto libtestutil.a
|
|
|
|
|
|
Generic form of C test executables
|
|
|
-==================================
|
|
|
+----------------------------------
|
|
|
|
|
|
#include "testutil.h"
|
|
|
|
|
@@ -133,12 +133,12 @@ Generic form of C test executables
|
|
|
return 1; /* Indicate success */
|
|
|
}
|
|
|
|
|
|
-You should use the TEST_xxx macros provided by testutil.h to test all failure
|
|
|
+You should use the `TEST_xxx` macros provided by `testutil.h` to test all failure
|
|
|
conditions. These macros produce an error message in a standard format if the
|
|
|
condition is not met (and nothing if the condition is met). Additional
|
|
|
-information can be presented with the TEST_info macro that takes a printf
|
|
|
-format string and arguments. TEST_error is useful for complicated conditions,
|
|
|
-it also takes a printf format string and argument. In all cases the TEST_xxx
|
|
|
+information can be presented with the `TEST_info` macro that takes a `printf`
|
|
|
+format string and arguments. `TEST_error` is useful for complicated conditions,
|
|
|
+it also takes a `printf` format string and argument. In all cases the `TEST_xxx`
|
|
|
macros are guaranteed to evaluate their arguments exactly once. This means
|
|
|
that expressions with side effects are allowed as parameters. Thus,
|
|
|
|
|
@@ -152,6 +152,5 @@ works fine and can be used in place of:
|
|
|
The former produces a more meaningful message on failure than the latter.
|
|
|
|
|
|
Note that the test infrastructure automatically sets up all required environment
|
|
|
-variables (such as OPENSSL_MODULES, OPENSSL_CONF etc) for the tests. Individual
|
|
|
-tests may choose to override the default settings as required.
|
|
|
-
|
|
|
+variables (such as `OPENSSL_MODULES`, `OPENSSL_CONF`, etc.) for the tests.
|
|
|
+Individual tests may choose to override the default settings as required.
|