2
0

README.external 4.6 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164
  1. Running external test suites with OpenSSL
  2. =========================================
  3. It is possible to integrate external test suites into OpenSSL's "make test".
  4. This capability is considered a developer option and does not work on all
  5. platforms.
  6. The BoringSSL test suite
  7. ========================
  8. In order to run the BoringSSL tests with OpenSSL, first checkout the BoringSSL
  9. source code into an appropriate directory. This can be done in two ways:
  10. 1) Separately from the OpenSSL checkout using:
  11. $ git clone https://boringssl.googlesource.com/boringssl boringssl
  12. The BoringSSL tests are only confirmed to work at a specific commit in the
  13. BoringSSL repository. Later commits may or may not pass the test suite:
  14. $ cd boringssl
  15. $ git checkout 490469f850e
  16. 2) Using the already configured submodule settings in OpenSSL:
  17. $ git submodule update --init
  18. Configure the OpenSSL source code to enable the external tests:
  19. $ cd ../openssl
  20. $ ./config enable-ssl3 enable-ssl3-method enable-weak-ssl-ciphers \
  21. enable-external-tests
  22. Note that using other config options than those given above may cause the tests
  23. to fail.
  24. Run the OpenSSL tests by providing the path to the BoringSSL test runner in the
  25. BORING_RUNNER_DIR environment variable:
  26. $ BORING_RUNNER_DIR=/path/to/boringssl/ssl/test/runner make test
  27. Note that the test suite may change directory while running so the path provided
  28. should be absolute and not relative to the current working directory.
  29. To see more detailed output you can run just the BoringSSL tests with the
  30. verbose option:
  31. $ VERBOSE=1 BORING_RUNNER_DIR=/path/to/boringssl/ssl/test/runner make \
  32. TESTS="test_external_boringssl" test
  33. Test failures and suppressions
  34. ------------------------------
  35. A large number of the BoringSSL tests are known to fail. A test could fail
  36. because of many possible reasons. For example:
  37. - A bug in OpenSSL
  38. - Different interpretations of standards
  39. - Assumptions about the way BoringSSL works that do not apply to OpenSSL
  40. - The test uses APIs added to BoringSSL that are not present in OpenSSL
  41. - etc
  42. In order to provide a "clean" baseline run with all the tests passing a config
  43. file has been provided that suppresses the running of tests that are known to
  44. fail. These suppressions are held in the file "test/ossl_shim/ossl_config.json"
  45. within the OpenSSL source code.
  46. The community is encouraged to contribute patches which reduce the number of
  47. suppressions that are currently present.
  48. Python PYCA/Cryptography test suite
  49. ===================================
  50. This python test suite runs cryptographic tests with a local OpenSSL build as
  51. the implementation.
  52. First checkout the PYCA/Cryptography module into ./pyca-cryptography using:
  53. $ git submodule update --init
  54. Then configure/build OpenSSL compatible with the python module:
  55. $ ./config shared enable-external-tests
  56. $ make
  57. The tests will run in a python virtual environment which requires virtualenv
  58. to be installed.
  59. $ make test VERBOSE=1 TESTS=test_external_pyca
  60. Test failures and suppressions
  61. ------------------------------
  62. Some tests target older (<=1.0.2) versions so will not run. Other tests target
  63. other crypto implementations so are not relevant. Currently no tests fail.
  64. krb5 test suite
  65. ===============
  66. Much like the PYCA/Cryptography test suite, this builds and runs the krb5
  67. tests against the local OpenSSL build.
  68. You will need a git checkout of krb5 at the top level:
  69. $ git clone https://github.com/krb5/krb5
  70. krb5's master has to pass this same CI, but a known-good version is
  71. krb5-1.15.1-final if you want to be sure.
  72. $ cd krb5
  73. $ git checkout krb5-1.15.1-final
  74. $ cd ..
  75. OpenSSL must be built with external tests enabled:
  76. $ ./config enable-external-tests
  77. $ make
  78. krb5's tests will then be run as part of the rest of the suite, or can be
  79. explicitly run (with more debugging):
  80. $ VERBOSE=1 make TESTS=test_external_krb5 test
  81. Test-failures suppressions
  82. --------------------------
  83. krb5 will automatically adapt its test suite to account for the configuration
  84. of your system. Certain tests may require more installed packages to run. No
  85. tests are expected to fail.
  86. Updating test suites
  87. ====================
  88. To update the commit for any of the above test suites:
  89. - Make sure the submodules are cloned locally:
  90. $ git submodule update --init --recursive
  91. - Enter subdirectory and pull from the repository (use a specific branch/tag if required):
  92. $ cd <submodule-dir>
  93. $ git pull origin master
  94. - Go to root directory, there should be a new git status:
  95. $ cd ../
  96. $ git status
  97. ...
  98. # modified: <submodule-dir> (new commits)
  99. ...
  100. - Add/commit/push the update
  101. git add <submodule-dir>
  102. git commit -m "Updated <submodule> to latest commit"
  103. git push