runtests.1 9.2 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196
  1. .\" **************************************************************************
  2. .\" * _ _ ____ _
  3. .\" * Project ___| | | | _ \| |
  4. .\" * / __| | | | |_) | |
  5. .\" * | (__| |_| | _ <| |___
  6. .\" * \___|\___/|_| \_\_____|
  7. .\" *
  8. .\" * Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
  9. .\" *
  10. .\" * This software is licensed as described in the file COPYING, which
  11. .\" * you should have received as part of this distribution. The terms
  12. .\" * are also available at https://curl.se/docs/copyright.html.
  13. .\" *
  14. .\" * You may opt to use, copy, modify, merge, publish, distribute and/or sell
  15. .\" * copies of the Software, and permit persons to whom the Software is
  16. .\" * furnished to do so, under the terms of the COPYING file.
  17. .\" *
  18. .\" * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
  19. .\" * KIND, either express or implied.
  20. .\" *
  21. .\" * SPDX-License-Identifier: curl
  22. .\" *
  23. .\" **************************************************************************
  24. .\"
  25. .TH runtests.pl 1 "06 Jun 2023" runtests runtests
  26. .SH NAME
  27. runtests.pl \- run one or more test cases
  28. .SH SYNOPSIS
  29. .B runtests.pl [options] [tests]
  30. .SH DESCRIPTION
  31. \fIruntests.pl\fP runs one, several or all the existing test cases in curl's
  32. test suite. It is often called from the root Makefile of the curl package with
  33. \&'make test'.
  34. .SH "TESTS"
  35. Specify which test(s) to run by specifying test numbers or keywords.
  36. If no test number or keyword is given, all existing tests that the script can
  37. find will be considered for running. You can specify single test cases to run
  38. by specifying test numbers space-separated, like "1 3 5 7 11", and you can
  39. specify a range of tests like "45 to 67".
  40. Specify tests to not run with a leading exclamation point, like "!66", which
  41. runs all available tests except number 66.
  42. Prefix a test number with a tilde (~) to still run it, but ignore the results.
  43. It is also possible to specify tests based on a keyword describing the test(s)
  44. to run, like "FTPS". The keywords are strings used in the individual tests.
  45. You can also specify keywords with a leading exclamation point and the keyword
  46. or phrase, like "!HTTP NTLM auth" to run all tests \fBexcept\fP those using
  47. this keyword. Remember that the exclamation marks and spaces will need to be
  48. quoted somehow when entered at many command shells.
  49. Prefix a keyword with a tilde (~) to still run it, but ignore the results.
  50. .SH OPTIONS
  51. .IP "-a"
  52. Continue running the rest of the test cases even if one test fails. By
  53. default, the test script stops as soon as an error is detected.
  54. .IP "-ac <curl>"
  55. Provide a path to a curl binary to talk to APIs (currently only CI test APIs).
  56. .IP "-am"
  57. Display test results in automake style output (PASS/FAIL: [number] [name]).
  58. .IP "-c <curl>"
  59. Provide a path to a custom curl binary to run the tests with. Default is the
  60. curl executable in the build tree.
  61. .IP "-d"
  62. Enable protocol debug: have the servers display protocol output. If used in
  63. conjunction with parallel testing, it will be difficult to associate the logs
  64. with the test being run.
  65. .IP "-E <exclude_file>"
  66. Load the \fBexclude_file\fP with additional reasons why certain tests
  67. should be skipped. Useful when testing with external HTTP proxies in
  68. which case some of the tests aren't appropriate.
  69. The file contains colon-delimited lines. The first field contains the
  70. type of exclusion, the second field contains a pattern and the final
  71. field contains the reason why matching tests should be skipped.
  72. The exclusion types are \fkeyword\fP, \ftest\fP, and \ftool\fP.
  73. .IP "-e"
  74. Run the test event-based (if possible). This will make runtests invoke curl
  75. with --test-event option. This option only works if both curl and libcurl were
  76. built debug-enabled.
  77. .IP "-f"
  78. Force the test to run even if mentioned in DISABLED.
  79. .IP "-g"
  80. Run the given test(s) with gdb. This is best used on a single test case and
  81. curl built --disable-shared. This then fires up gdb with command line set to
  82. run the specified test case. Simply (set a break-point and) type 'run' to
  83. start.
  84. .IP "-gw"
  85. Run the given test(s) with gdb as a windowed application.
  86. .IP "-h, --help"
  87. Displays a help text about this program's command line options.
  88. .IP "-k"
  89. Keep output and log files in log/ after a test run, even if no error was
  90. detected. Useful for debugging.
  91. .IP "-j[num]"
  92. Spawn num processes to run tests. This defaults to 0 to run tests serially
  93. within a single process. Using a number greater than one allows multiple tests
  94. to run in parallel, speeding up a test run. The optimum number is dependent on
  95. the system and set of tests to run, but 7*number of CPU cores is a good figure
  96. to start with, or 1.3*number of CPU cores if Valgrind is in use. Enabling
  97. parallel tests is not recommended in conjunction with the \-g option.
  98. .IP "-L <file>"
  99. Load and execute the specified file which should contain perl code.
  100. This option allows one to change \fIruntests.pl\fP behaviour by overwriting
  101. functions and variables and is useful when testing external proxies
  102. using curl's regression test suite.
  103. .IP "-l"
  104. Lists all test case names.
  105. .IP "-n"
  106. Disable the check for and use of valgrind.
  107. .IP "--no-debuginfod"
  108. Delete the DEBUGINFOD_URLS variable if that is defined. Makes valgrind, gdb
  109. etc not able to use this functionality.
  110. .IP "-o <variablename=value>"
  111. Overwrite the specified internal \fBvariable\fP with \fBvalue\fP.
  112. Useful to change variables that didn't get a dedicated flag to change them.
  113. Check the source to see which variables are available.
  114. .IP "-P <proxy>"
  115. Use the specified HTTP proxy when executing tests, even if the tests
  116. themselves don't specify a proxy. This option allows one to test external
  117. proxies using curl's regression test suite.
  118. .IP "-p"
  119. Prints out all files in "log/" to stdout when a test case fails. Very
  120. practical when used in the automated and distributed tests since then the
  121. people checking the failures and the reasons for them might not have physical
  122. access to the machine and logs.
  123. .IP "-R"
  124. Run the tests in a scrambled, or randomized, order instead of sequentially.
  125. The random seed initially set for this is fixed per month and can be set with
  126. \fI--seed\fP.
  127. .IP "-r"
  128. Display run time statistics. (Requires Perl Time::HiRes module)
  129. .IP "-rf"
  130. Display full run time statistics. (Requires Perl Time::HiRes module)
  131. .IP "-rm"
  132. Force removal of files by killing locking processes. (Windows only,
  133. requires Sysinternals handle[64].exe to be on PATH)
  134. .IP "--repeat=[num]"
  135. This will repeat the given set of test numbers this many times. If no test
  136. numbers are given, it will repeat ALL tests this many times. It iteratively
  137. adds the new sequence at the end of the initially given one.
  138. If \fB-R\fP is also used, the scrambling is done after the repeats have
  139. extended the test sequence.
  140. .IP "-s"
  141. Shorter output. Speaks less than default.
  142. .IP "--seed=[num]"
  143. When using \fI--shallow\fP or \fI-R\fP that randomize certain aspects of the
  144. behavior, this option can set the initial seed. If not set, the random seed
  145. will be set based on the currently set local year and month and the first line
  146. of the "curl -V" output.
  147. .IP "--shallow=[num]"
  148. Used together with \fB-t\fP. This limits the number of tests to fail in
  149. torture mode to no more than 'num' per test case. If this reduces the amount,
  150. the script will randomly discard entries to fail until the amount is 'num'.
  151. The random seed initially set for this is fixed per month and can be set with
  152. \fI--seed\fP.
  153. .IP "-t[num]"
  154. Selects a \fBtorture\fP test for the given tests. This makes runtests.pl first
  155. run the tests once and count the number of memory allocations made. It then
  156. reruns the test that number of times, each time forcing one of the allocations
  157. to fail until all allocs have been tested. By setting \fInum\fP you can force
  158. the allocation with that number to be set to fail at once instead of looping
  159. through everyone, which is very handy when debugging and then often in
  160. combination with \fI-g\fP.
  161. .IP "-u"
  162. Error instead of warning on server unexpectedly alive.
  163. .IP "-v"
  164. Enable verbose output. Speaks more than by default. If used in conjunction with
  165. parallel testing, it will be difficult to associate the logs with the test
  166. being run.
  167. .IP "-vc <curl>"
  168. Provide a path to a custom curl binary to run when verifying that the servers
  169. running are indeed our test servers. Default is the curl executable in the
  170. build tree.
  171. .SH "RUNNING TESTS"
  172. Many tests have conditions that must be met before the test case can run
  173. fine. They could depend on built-in features in libcurl or features present in
  174. the operating system or even in third-party libraries that curl may or may not
  175. use.
  176. .P
  177. The test script checks most of these by itself to determine when it is
  178. safe to attempt to run each test. Those which cannot be run due to
  179. failed requirements will simply be skipped and listed at the completion
  180. of all test cases. In some unusual configurations, the test script
  181. cannot make the correct determination for all tests. In these cases,
  182. the problematic tests can be skipped using the "!keyword" skip feature
  183. documented earlier.
  184. .SH "WRITING TESTS"
  185. The simplest way to write test cases is to start with a similar existing test,
  186. save it with a new number and then adjust it to fit. There's an attempt to
  187. document the test case file format in the tests/FILEFORMAT.md.