Next: , Previous: , Up: Using Autotest   [Contents][Index]


19.2 Writing testsuite.at

The testsuite.at is a Bourne shell script making use of special Autotest M4 macros. It often contains a call to AT_INIT near its beginning followed by one call to m4_include per source file for tests. Each such included file, or the remainder of testsuite.at if include files are not used, contain a sequence of test groups. Each test group begins with a call to AT_SETUP, then an arbitrary number of shell commands or calls to AT_CHECK, and then completes with a call to AT_CLEANUP. Multiple test groups can be categorized by a call to AT_BANNER.

All of the public Autotest macros have all-uppercase names in the namespace ‘^AT_’ to prevent them from accidentally conflicting with other text; Autoconf also reserves the namespace ‘^_AT_’ for internal macros. All shell variables used in the testsuite for internal purposes have mostly-lowercase names starting with ‘at_’. Autotest also uses here-document delimiters in the namespace ‘^_AT[A-Z]’, and makes use of the file system namespace ‘^at-’.

Since Autoconf is built on top of M4sugar (see Programming in M4sugar) and M4sh (see Programming in M4sh), you must also be aware of those namespaces (‘^_?\(m4\|AS\)_’). In general, you should not use the namespace of a package that does not own the macro or shell code you are writing.

Macro: AT_INIT ([name])

Initialize Autotest. Giving a name to the test suite is encouraged if your package includes several test suites. Before this macro is called, AT_PACKAGE_STRING and AT_PACKAGE_BUGREPORT must be defined, which are used to display information about the testsuite to the user. Typically, these macros are provided by a file package.m4 built by make (see Making testsuite Scripts), in order to inherit the package name, version, and bug reporting address from configure.ac.

Macro: AT_COPYRIGHT (copyright-notice)

State that, in addition to the Free Software Foundation’s copyright on the Autotest macros, parts of your test suite are covered by copyright-notice.

The copyright-notice shows up in both the head of testsuite and in ‘testsuite --version’.

Macro: AT_ARG_OPTION (options, help-text, [action-if-given], [action-if-not-given])

Accept options from the space-separated list options, a list that has leading dashes removed from the options. Long options will be prefixed with ‘--’, single-character options with ‘-’. The first word in this list is the primary option, any others are assumed to be short-hand aliases. The variable associated with it is at_arg_option, with any dashes in option replaced with underscores.

If the user passes --option to the testsuite, the variable will be set to ‘:’. If the user does not pass the option, or passes --no-option, then the variable will be set to ‘false’.

action-if-given is run each time the option is encountered; here, the variable at_optarg will be set to ‘:’ or ‘false’ as appropriate. at_optarg is actually just a copy of at_arg_option.

action-if-not-given will be run once after option parsing is complete and if no option from options was used.

help-text is added to the end of the list of options shown in testsuite --help (see AS_HELP_STRING).

It is recommended that you use a package-specific prefix to options names in order to avoid clashes with future Autotest built-in options.

Macro: AT_ARG_OPTION_ARG (options, help-text, [action-if-given], [action-if-not-given])

Accept options with arguments from the space-separated list options, a list that has leading dashes removed from the options. Long options will be prefixed with ‘--’, single-character options with ‘-’. The first word in this list is the primary option, any others are assumed to be short-hand aliases. The variable associated with it is at_arg_option, with any dashes in option replaced with underscores.

If the user passes --option=arg or --option arg to the testsuite, the variable will be set to ‘arg’.

action-if-given is run each time the option is encountered; here, the variable at_optarg will be set to ‘arg’. at_optarg is actually just a copy of at_arg_option.

action-if-not-given will be run once after option parsing is complete and if no option from options was used.

help-text is added to the end of the list of options shown in testsuite --help (see AS_HELP_STRING).

It is recommended that you use a package-specific prefix to options names in order to avoid clashes with future Autotest built-in options.

Macro: AT_COLOR_TESTS

Enable colored test results by default when the output is connected to a terminal.

Macro: AT_TESTED (executables)

Log the file name and answer to --version of each program in space-separated list executables. Several invocations register new executables, in other words, don’t fear registering one program several times.

Autotest test suites rely on PATH to find the tested program. This avoids the need to generate absolute names of the various tools, and makes it possible to test installed programs. Therefore, knowing which programs are being exercised is crucial to understanding problems in the test suite itself, or its occasional misuses. It is a good idea to also subscribe foreign programs you depend upon, to avoid incompatible diagnostics.

executables is implicitly wrapped in shell double quotes, but it will still use shell variable expansion (‘$’), command substitution (‘`’), and backslash escaping (‘\’). In particular, the EXEEXT variable is available if it is passed to the testsuite via atlocal or atconfig.

Macro: AT_PREPARE_TESTS (shell-code)

Execute shell-code in the main testsuite process, after initializing the test suite and processing command-line options, but before running any tests. If this macro is used several times, all of the shell-codes will be executed, in the order they appeared in testsuite.at.

One reason to use AT_PREPARE_TESTS is when the programs under test are sensitive to environment variables: you can unset all these variables or reset them to safe values in shell-code.

shell-code is only executed if at least one test is going to be run. In particular, it will not be executed if any of the --help, --version, --list, or --clean options are given to testsuite (see testsuite Invocation).

Macro: AT_PREPARE_EACH_TEST (shell-code)

Execute shell-code in each test group’s subshell, at the point of the AT_SETUP that starts the test group.

Macro: AT_TEST_HELPER_FN (name, args, description, code)

Define a shell function that will be available to the code for each test group. Its name will be ath_fn_name, and its body will be code. (The prefix prevents name conflicts with shell functions defined by M4sh and Autotest.)

args should describe the function’s arguments and description what it does; these are used only for documentation comments in the generated testsuite script.


Macro: AT_BANNER (test-category-name)

This macro identifies the start of a category of related test groups. When the resulting testsuite is invoked with more than one test group to run, its output will include a banner containing test-category-name prior to any tests run from that category. The banner should be no more than about 40 or 50 characters. A blank banner indicates uncategorized tests; an empty line will be inserted after tests from an earlier category, effectively ending that category.

Macro: AT_SETUP (test-group-name)

This macro starts a group of related tests, all to be executed in the same subshell. It accepts a single argument, which holds a few words (no more than about 30 or 40 characters) quickly describing the purpose of the test group being started. test-group-name must not expand to unbalanced quotes, although quadrigraphs can be used.

Macro: AT_KEYWORDS (keywords)

Associate the space-separated list of keywords to the enclosing test group. This makes it possible to run “slices” of the test suite. For instance, if some of your test groups exercise some ‘foo’ feature, then using ‘AT_KEYWORDS(foo)’ lets you run ‘./testsuite -k foo’ to run exclusively these test groups. The test-group-name of the test group is automatically recorded to AT_KEYWORDS.

Several invocations within a test group accumulate new keywords. In other words, don’t fear registering the same keyword several times in a test group.

Macro: AT_CAPTURE_FILE (file)

If the current test group fails, log the contents of file. Several identical calls within one test group have no additional effect.

Macro: AT_FAIL_IF (shell-condition)

Make the test group fail and skip the rest of its execution, if shell-condition is true. shell-condition is a shell expression such as a test command. Tests before AT_FAIL_IF will be executed and may still cause the test group to be skipped. You can instantiate this macro many times from within the same test group.

You should use this macro only for very simple failure conditions. If the shell-condition could emit any kind of output you should instead use AT_CHECK like

AT_CHECK([if shell-condition; then exit 99; fi])

so that such output is properly recorded in the testsuite.log file.

Macro: AT_SKIP_IF (shell-condition)

Determine whether the test should be skipped because it requires features that are unsupported on the machine under test. shell-condition is a shell expression such as a test command. Tests before AT_SKIP_IF will be executed and may still cause the test group to fail. You can instantiate this macro many times from within the same test group.

You should use this macro only for very simple skip conditions. If the shell-condition could emit any kind of output you should instead use AT_CHECK like

AT_CHECK([if shell-condition; then exit 77; fi])

so that such output is properly recorded in the testsuite.log file.

Macro: AT_XFAIL_IF (shell-condition)

Determine whether the test is expected to fail because it is a known bug (for unsupported features, you should skip the test). shell-condition is a shell expression such as a test command; you can instantiate this macro many times from within the same test group, and one of the conditions is enough to turn the test into an expected failure.

Macro: AT_CLEANUP

End the current test group.


Macro: AT_DATA (file, contents)
Macro: AT_DATA_UNQUOTED (file, contents)

Initialize an input data file with given contents. Of course, the contents have to be properly quoted between square brackets to protect against included commas or spurious M4 expansion. contents must be empty or end with a newline. file must be a single shell word that expands into a single file name.

The difference between AT_DATA and AT_DATA_UNQUOTED is that only the latter performs shell variable expansion (‘$’), command substitution (‘`’), and backslash escaping (‘\’) on contents.

Macro: AT_CHECK (commands, [status = ‘0], [stdout], [stderr], [run-if-fail], [run-if-pass])
Macro: AT_CHECK_UNQUOTED (commands, [status = ‘0], [stdout], [stderr], [run-if-fail], [run-if-pass])

Perform a test, by running the shell commands in a subshell. commands is output as-is, so shell expansions are honored. These commands are expected to have a final exit status of status, and to produce output as described by stdout and stderr (see below).

This macro must be invoked in between AT_SETUP and AT_CLEANUP.

If commands exit with unexpected status 77, then the rest of the test group is skipped. If commands exit with unexpected status 99, then the test group is immediately failed; this is called a hard failure. Otherwise, the test is considered to have succeeeded if all of the status, stdout, and stderr expectations were met.

If run-if-fail is nonempty, it provides extra shell commands to run when the test fails; if run-if-pass is nonempty, it provides extra shell commands to run when the test succeeds. These commands are not run in a subshell, and they are not run when the test group is skipped (exit code 77) or hard-failed (exit code 99). They may change whether the test group is considered to have succeeded, by modifying the shell variable at_failed; set it to : to indicate that the test group has failed, or false to indicate that it has succeeded.

The exit status of commands is available to run-if-fail and run-if-pass commands in the at_status shell variable. The output from commands is also available, in the files named by the at_stdout and at_stderr variables.

If status is the literal ‘ignore’, then the exit status of commands is not checked, except for the special cases of 77 (skip) and 99 (hard failure). The existence of hard failures allows one to mark a test as an expected failure with AT_XFAIL_IF because a feature has not yet been implemented, but to still distinguish between gracefully handling the missing feature and dumping core.

If the value of the stdout or stderr parameter is one of the literals in the following table, then the test treats the output according to the rules of that literal.

ignore

The content of the output is ignored, but still captured in the test group log (if the testsuite is run with the -v option, the test group log is displayed as the test is run; if the test group later fails, the test group log is also copied into the overall testsuite log). This action is valid for both stdout and stderr.

ignore-nolog

The content of the output is ignored, and nothing is captured in the log files. If commands are likely to produce binary output (including long lines) or large amounts of output, then logging the output can make it harder to locate details related to subsequent tests within the group, and could potentially corrupt terminal display of a user running testsuite -v. This action is valid for both stdout and stderr.

stdout

Only valid as the stdout parameter. Capture the content of standard output in both a file named stdout and the test group log. Subsequent commands in the test group can then post-process the file. This action is often used when it is desired to use grep to look for a substring in the output, or when the output must be post-processed to normalize error messages into a common form.

stderr

Only valid as the stderr parameter. Capture the content of standard error in both a file named stderr and the test group log.

stdout-nolog
stderr-nolog

Like ‘stdout’ or ‘stderr’, except that the captured output is not duplicated into the test group log. This action is particularly useful for an intermediate check that produces large amounts of data, which will be followed by another check that filters down to the relevant data, as it makes it easier to locate details in the log.

expout

Only valid as the stdout parameter. Compare standard output with the previously created file expout, and list any differences in the testsuite log.

experr

Only valid as the stderr parameter. Compare standard error with the previously created file experr, and list any differences in the testsuite log.

Otherwise, the values of the stdout and stderr parameters are treated as text that must exactly match the output given by commands on standard output and standard error (including an empty parameter for no output); any differences are captured in the testsuite log and the test is failed (unless an unexpected exit status of 77 skipped the test instead).

AT_CHECK_UNQUOTED performs shell variable expansion (‘$’), command substitution (‘`’), and backslash escaping (‘\’) on comparison text given in the stdout and stderr parameters; AT_CHECK does not. There is no difference in the interpretation of commands.

Macro: AT_CHECK_EUNIT (module, test-spec, [erlflags], [run-if-fail], [run-if-pass])

Initialize and execute an Erlang module named module that performs tests following the test-spec EUnit test specification. test-spec must be a valid EUnit test specification, as defined in the EUnit Reference Manual. erlflags are optional command-line options passed to the Erlang interpreter to execute the test Erlang module. Typically, erlflags defines at least the paths to directories containing the compiled Erlang modules under test, as ‘-pa path1 path2 ...’.

For example, the unit tests associated with Erlang module ‘testme’, which compiled code is in subdirectory src, can be performed with:

AT_CHECK_EUNIT([testme_testsuite], [{module, testme}],
               [-pa "${abs_top_builddir}/src"])

This macro must be invoked in between AT_SETUP and AT_CLEANUP.

Variables ERL, ERLC, and (optionally) ERLCFLAGS must be defined as the path of the Erlang interpreter, the path of the Erlang compiler, and the command-line flags to pass to the compiler, respectively. Those variables should be configured in configure.ac using the AC_ERLANG_PATH_ERL and AC_ERLANG_PATH_ERLC macros, and the configured values of those variables are automatically defined in the testsuite. If ERL or ERLC is not defined, the test group is skipped.

If the EUnit library cannot be found, i.e. if module eunit cannot be loaded, the test group is skipped. Otherwise, if test-spec is an invalid EUnit test specification, the test group fails. Otherwise, if the EUnit test passes, shell commands run-if-pass are executed or, if the EUnit test fails, shell commands run-if-fail are executed and the test group fails.

Only the generated test Erlang module is automatically compiled and executed. If test-spec involves testing other Erlang modules, e.g. module ‘testme’ in the example above, those modules must be already compiled.

If the testsuite is run in verbose mode and with the --verbose option, EUnit is also run in verbose mode to output more details about individual unit tests.


Next: , Previous: , Up: Using Autotest   [Contents][Index]