If the variable TESTS is defined, its value is taken to be a
list of programs or scripts to run in order to do the testing.
Programs needing data files should look for them in srcdir
(which is both an environment variable and a make variable) so they
work when building in a separate directory (see Build Directories in The Autoconf Manual), and in
particular for the distcheck rule (see Checking the Distribution).
For each of the TESTS, the result of execution is printed along
with the test name, where PASS denotes a successful test,
FAIL denotes a failed test, XFAIL an expected failure,
XPASS an unexpected pass for a test that is supposed to fail,
and SKIP denotes a skipped test.
The number of failures will be printed at the end of the run. If a given test program exits with a status of 77, then its result is ignored in the final count. This feature allows non-portable tests to be ignored in environments where they don’t make sense.
If the Automake option color-tests is used (see Changing Automake’s Behavior)
and standard output is connected to a capable terminal, then the test
results and the summary are colored appropriately. The user can disable
colored output by setting the make variable
‘AM_COLOR_TESTS=no’, or force colored output even without a connecting
terminal with ‘AM_COLOR_TESTS=always’.
Note that the semantics of some make implementations when used
in parallel mode (see Parallel make in The Autoconf Manual)
can cause the automatic detection of a connection to a capable terminal
to fail. In that case, you can still resort to the use of
‘AM_COLOR_TESTS=always’.
The variable TESTS_ENVIRONMENT can be used to set environment
variables for the test run; the environment variable srcdir is
set in the rule. If all your test programs are scripts, you can also
set TESTS_ENVIRONMENT to an invocation of the shell (e.g.
‘$(SHELL) -x’ can be useful for debugging the tests), or any other
interpreter. For instance, the following setup may be used to run tests
with Perl:
TESTS_ENVIRONMENT = $(PERL) -Mstrict -w TESTS = foo.pl bar.pl baz.pl
Note that the parallel-tests driver provides a more elegant
way to achieve the same effect, freeing the TESTS_ENVIRONMENT
variable for the user to override (see Simple Tests using ‘parallel-tests’).
You may define the variable XFAIL_TESTS to a list of tests
(usually a subset of TESTS) that are expected to fail. This will
reverse the result of those tests.
Automake ensures that each file listed in TESTS is built before
any tests are run; you can list both source and derived programs (or
scripts) in TESTS; the generated rule will look both in
srcdir and .. For instance, you might want to run a C
program as a test. To do this you would list its name in TESTS
and also in check_PROGRAMS, and then specify it as you would
any other program.
Programs listed in check_PROGRAMS (and check_LIBRARIES,
check_LTLIBRARIES...) are only built during make check,
not during make all. You should list there any program needed
by your tests that does not need to be built by make all. Note
that check_PROGRAMS are not automatically added to
TESTS because check_PROGRAMS usually lists programs used
by the tests, not the tests themselves. Of course you can set
TESTS = $(check_PROGRAMS) if all your programs are test cases.