Starting from Automake version 1.12, the parallel test harness allows the package authors to use third-party custom test drivers, in case the default ones are inadequate for their purposes, or do not support their testing protocol of choice.
A custom test driver is expected to properly run the test programs passed to it (including the command-line arguments passed to those programs, if any), to analyze their execution and outcome, to create the .log and .trs files associated to these test runs, and to display the test results on the console. It is responsibility of the author of the test driver to ensure that it implements all the above steps meaningfully and correctly; Automake isn’t and can’t be of any help here. On the other hand, the Automake-provided code for testsuite summary generation offers support for test drivers allowing several test results per test script, if they take care to register such results properly (see Log files generation and test results recording).
The exact details of how test scripts’ results are to be determined and analyzed is left to the individual drivers. Some drivers might only consider the test script exit status (this is done for example by the default test driver used by the parallel test harness, described in the previous section). Other drivers might implement more complex and advanced test protocols, which might require them to parse and interpreter the output emitted by the test script they’re running (examples of such protocols are TAP and SubUnit).
It’s very important to note that, even when using custom test drivers, most of the infrastructure described in the previous section about the parallel harness remains in place; this includes:
TESTS, and overridable at runtime through the redefinition of
make’s option -j;
RECHECK_LOGSvariable, and lazy reruns of tests;
VERBOSEenvironment variable to get verbose output on testsuite failures;
On the other hand, the exact semantics of how (and if)
XFAIL_TESTS, and hard errors are supported
and handled is left to the individual test drivers.