README 6.2 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153
  1. _ _ ____ _
  2. ___| | | | _ \| |
  3. / __| | | | |_) | |
  4. | (__| |_| | _ <| |___
  5. \___|\___/|_| \_\_____|
  6. The cURL Test Suite
  7. Requires:
  8. perl (and a unix-style shell)
  9. diff (when a test fails, a diff is shown)
  10. stunnel (for HTTPS and FTPS tests)
  11. OpenSSH or SunSSH (for SCP, SFTP and SOCKS4/5 tests)
  12. TCP ports used by default:
  13. - 8990 on localhost for HTTP tests
  14. - 8991 on localhost for HTTPS tests
  15. - 8994 on localhost for HTTP IPv6 tests
  16. - 8992 on localhost for FTP tests
  17. - 8995 on localhost for FTP (2) tests
  18. - 8993 on localhost for FTPS tests
  19. - 8996 on localhost for FTP IPv6 tests
  20. - 8997 on localhost for TFTP tests
  21. - 8999 on localhost for SCP/SFTP tests
  22. - 9000 on localhost for SOCKS tests
  23. The test suite runs simple FTP, HTTP and TFTP servers on these ports to
  24. which it makes requests. For SSL tests, it runs stunnel to handle
  25. encryption to the regular servers. For SSH, it runs a standard OpenSSH
  26. server. For SOCKS4/5 tests SSH is used to perform the SOCKS functionality
  27. and requires a SSH client and server.
  28. The base port number shown above can be changed using runtests' -b option
  29. to allow running more than one instance of the test suite simultaneously
  30. on one machine.
  31. Run:
  32. 'make test'. This builds the test suite support code and invokes the
  33. 'runtests.pl' perl script to run all the tests. Edit the top variables
  34. of that script in case you have some specific needs, or run the script
  35. manually (after the support code has been built).
  36. The script breaks on the first test that doesn't do OK. Use -a to prevent
  37. the script from abort on the first error. Run the script with -v for more
  38. verbose output. Use -d to run the test servers with debug output enabled as
  39. well. Specifying -k keeps all the log files generated by the test intact.
  40. Use -s for shorter output, or pass test numbers to run specific tests only
  41. (like "./runtests.pl 3 4" to test 3 and 4 only). It also supports test case
  42. ranges with 'to', as in "./runtests 3 to 9" which runs the seven tests from
  43. 3 to 9. Any test numbers starting with ! are disabled, as are any test
  44. numbers found in the file data/DISABLED (one per line).
  45. Shell startup scripts:
  46. Tests which use the ssh test server, SCP/SFTP/SOCKS tests, might be badly
  47. influenced by the output of system wide or user specific shell startup scripts,
  48. .bashrc, .profile, /etc/csh.cshrc, .login, /etc/bashrc, etc. which output text
  49. messages or escape sequences on user login. When these shell startup messages
  50. or escape sequences are output they might corrupt the expected stream of data
  51. which flows to the sftp-server or from the ssh client which can result in bad
  52. test behaviour or even prevent the test server from running.
  53. If the test suite ssh or sftp server fails to start up and logs the message
  54. 'Received message too long' then you are certainly suffering the unwanted
  55. output of a shell startup script. Locate, cleanup or adjust the shell script.
  56. Memory:
  57. The test script will check that all allocated memory is freed properly IF
  58. curl has been built with the CURLDEBUG define set. The script will
  59. automatically detect if that is the case, and it will use the ../memanalyze
  60. script to analyze the memory debugging output.
  61. The -t option will enable torture testing mode, which runs each test
  62. many times but causes a different memory allocation to fail on each
  63. successive run. This tests the out of memory error handling code to
  64. ensure that memory leaks do not occur even in those situations.
  65. Debug:
  66. If a test case fails, you can conveniently get the script to invoke the
  67. debugger (gdb) for you with the server running and the exact same command
  68. line parameters that failed. Just invoke 'runtests.pl <test number> -g' and
  69. then just type 'run' in the debugger to perform the command through the
  70. debugger.
  71. If a test case causes a core dump, analyze it by running gdb like:
  72. # gdb ../curl/src core
  73. ... and get a stack trace with the gdb command:
  74. (gdb) where
  75. Logs:
  76. All logs are generated in the logs/ subdirectory (it is emptied first
  77. in the runtests.pl script). Use runtests.pl -k to keep the temporary files
  78. after the test run.
  79. Data:
  80. All test cases are put in the data/ subdirectory. Each test is stored in the
  81. file named according to the test number.
  82. See FILEFORMAT for the description of the test case files.
  83. Code coverage:
  84. gcc provides a tool that can determine the code coverage figures for
  85. the test suite. To use it, configure curl with
  86. CFLAGS='-fprofile-arcs -ftest-coverage -g -O0'. Make sure you run the normal
  87. and torture tests to get more full coverage, i.e. do:
  88. make test
  89. make test-torture
  90. The graphical tool ggcov can be used to browse the source and create
  91. coverage reports on *NIX hosts:
  92. ggcov -r lib src
  93. The text mode tool gcov may also be used, but it doesn't handle object files
  94. in more than one directory very well.
  95. Remote testing:
  96. The runtests.pl script provides some hooks to allow curl to be tested on a
  97. machine where perl can not be run. The test framework in this case runs on
  98. a workstation where perl is available, while curl itself is run on a remote
  99. system using ssh or some other remote execution method. See the comments at
  100. the beginning of runtests.pl for details.
  101. TEST CASE NUMBERS
  102. So far, I've used this system:
  103. 1 - 99 HTTP
  104. 100 - 199 FTP*
  105. 200 - 299 FILE*
  106. 300 - 399 HTTPS
  107. 400 - 499 FTPS
  108. 500 - 599 libcurl source code tests, not using the curl command tool
  109. 600 - 699 SCP/SFTP
  110. 700 - 799 SOCKS4 (even numbers) and SOCK5 (odd numbers)
  111. 1000 - 1999 miscellaneous*
  112. 2000 - x multiple sequential protocols per test case*
  113. Since 30-apr-2003, there's nothing in the system that requires us to keep
  114. within these number series, and those sections marked with * actually
  115. contain tests for a variety of protocols. Each test case now specifies
  116. its own server requirements, independent of test number.
  117. TODO:
  118. * Add tests for TELNET, LDAP, DICT...
  119. * SOCKS4/5 test deficiencies - no proxy authentication tests as SSH (the
  120. test mechanism) doesn't support them