Next: , Previous: Writing Testsuites, Up: Using Autotest

19.3 Running testsuite Scripts

Autotest test suites support the following arguments:

--help
-h
Display the list of options and exit successfully.
--version
-V
Display the version of the test suite and exit successfully.
--directory=dir
-C dir
Change the current directory to dir before creating any files. Useful for running the testsuite in a subdirectory from a top-level Makefile.
--jobs[=n]
-j[n]
Run n tests in parallel, if possible. If n is not given, run all given tests in parallel. Note that there should be no space before the argument to -j, as -j number denotes the separate arguments -j and number, see below.

In parallel mode, the standard input device of the testsuite script is not available to commands inside a test group. Furthermore, banner lines are not printed, and the summary line for each test group is output after the test group completes. Summary lines may appear unordered. If verbose and trace output are enabled (see below), they may appear intermixed from concurrently running tests.

Parallel mode requires the mkfifo command to work, and will be silently disabled otherwise.

--clean
-c
Remove all the files the test suite might have created and exit. Meant for clean Make targets.
--list
-l
List all the tests (or only the selection), including their possible keywords.
By default all tests are performed (or described with --list) in the default environment first silently, then verbosely, but the environment, set of tests, and verbosity level can be tuned:
‘variable=value’
Set the environment variable to value. Use this rather than ‘FOO=foo ./testsuite’ as debugging scripts would then run in a different environment.

The variable AUTOTEST_PATH specifies the testing path to prepend to PATH. Relative directory names (not starting with ‘/’) are considered to be relative to the top level of the package being built. All directories are made absolute, first starting from the top level build tree, then from the source tree. For instance ‘./testsuite AUTOTEST_PATH=tests:bin’ for a /src/foo-1.0 source package built in /tmp/foo results in ‘/tmp/foo/tests:/tmp/foo/bin’ and then ‘/src/foo-1.0/tests:/src/foo-1.0/bin’ being prepended to PATH.

‘number’
‘number-number’
‘number-’
‘-number’
Add the corresponding test groups, with obvious semantics, to the selection.
‘--keywords=keywords’
‘-k keywords’
Add to the selection the test groups with title or keywords (arguments to AT_SETUP or AT_KEYWORDS) that match all keywords of the comma separated list keywords, case-insensitively. Use ‘!’ immediately before the keyword to invert the selection for this keyword. By default, the keywords match whole words; enclose them in ‘.*’ to also match parts of words.

For example, running 

          ./testsuite -k 'autoupdate,.*FUNC.*'

selects all tests tagged ‘autoupdate’ and with tags containing ‘FUNC’ (as in ‘AC_CHECK_FUNC’, ‘AC_FUNC_ALLOCA’, etc.), while 

          ./testsuite -k '!autoupdate' -k '.*FUNC.*'

selects all tests not tagged ‘autoupdate’ or with tags containing ‘FUNC’.

‘--errexit’
‘-e’
If any test fails, immediately abort testing. It implies --debug: post test group clean up, and top-level logging are inhibited. This option is meant for the full test suite, it is not really useful for generated debugging scripts. If the testsuite is run in parallel mode using --jobs, then concurrently running tests will finish before exiting.
‘--verbose’
‘-v’
Force more verbosity in the detailed output of what is being done. This is the default for debugging scripts.
‘--debug’
‘-d’
Do not remove the files after a test group was performed —but they are still removed before, therefore using this option is sane when running several test groups. Create debugging scripts. Do not overwrite the top-level log (in order to preserve supposedly existing full log file). This is the default for debugging scripts, but it can also be useful to debug the testsuite itself.
‘--trace’
‘-x’
Trigger shell tracing of the test groups.