19.4 Making testsuite Scripts

For putting Autotest into movement, you need some configuration and makefile machinery. We recommend, at least if your package uses deep or shallow hierarchies, that you use tests/ as the name of the directory holding all your tests and their makefile. Here is a check list of things to do, followed by an example, taking into consideration whether you are also using Automake.

The following example demonstrates the above checklist, first by assuming that you are using Automake (see below for tweaks to make to get the same results without Automake). Begin by adding the following lines to your configure.ac:

# Initialize the test suite.
AC_CONFIG_TESTDIR([tests])
AC_CONFIG_FILES([tests/Makefile tests/atlocal])
AM_MISSING_PROG([AUTOM4TE], [autom4te])

Next, add the following lines to your tests/Makefile.am, in order to link ‘make check’ with a validation suite.

# The ':;' works around a Bash 3.2 bug when the output is not writable.
$(srcdir)/package.m4: $(top_srcdir)/configure.ac
        :;{ \
          echo '# Signature of the current package.' && \
          echo 'm4_define([AT_PACKAGE_NAME],' && \
          echo '  [$(PACKAGE_NAME)])' && \
          echo 'm4_define([AT_PACKAGE_TARNAME],' && \
          echo '  [$(PACKAGE_TARNAME)])' && \
          echo 'm4_define([AT_PACKAGE_VERSION],' && \
          echo '  [$(PACKAGE_VERSION)])' && \
          echo 'm4_define([AT_PACKAGE_STRING],' && \
          echo '  [$(PACKAGE_STRING)])' && \
          echo 'm4_define([AT_PACKAGE_BUGREPORT],' && \
          echo '  [$(PACKAGE_BUGREPORT)])'; \
          echo 'm4_define([AT_PACKAGE_URL],' && \
          echo '  [$(PACKAGE_URL)])'; \
        } >'$(srcdir)/package.m4'

EXTRA_DIST = testsuite.at $(srcdir)/package.m4 $(TESTSUITE) atlocal.in
TESTSUITE = $(srcdir)/testsuite

check-local: atconfig atlocal $(TESTSUITE)
        $(SHELL) '$(TESTSUITE)' $(TESTSUITEFLAGS)

installcheck-local: atconfig atlocal $(TESTSUITE)
        $(SHELL) '$(TESTSUITE)' AUTOTEST_PATH='$(bindir)' \
          $(TESTSUITEFLAGS)

clean-local:
        test ! -f '$(TESTSUITE)' || \
         $(SHELL) '$(TESTSUITE)' --clean

AUTOTEST = $(AUTOM4TE) --language=autotest
$(TESTSUITE): $(srcdir)/testsuite.at $(srcdir)/package.m4
        $(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at
        mv $@.tmp $@

Note that the built testsuite is distributed; this is necessary because users might not have Autoconf installed, and thus would not be able to rebuild it. Likewise, the use of Automake’s AM_MISSING_PROG will arrange for the definition of $AUTOM4TE within the Makefile to provide the user with a nicer error message if they modify a source file to the testsuite, and accidentally trigger the rebuild rules.

You might want to list explicitly the dependencies, i.e., the list of the files testsuite.at includes.

If you don’t use Automake, you should make the following tweaks. In your configure.ac, replace the AM_MISSING_PROG line above with AC_PATH_PROG([AUTOM4TE], [autom4te], [false]). You are welcome to also try using the missing script from the Automake project instead of false, to try to get a nicer error message when the user modifies prerequisites but did not have Autoconf installed, but at that point you may be better off using Automake. Then, take the code suggested above for tests/Makefile.am and place it in your tests/Makefile.in instead. Add code to your tests/Makefile.in to ensure that $(EXTRA_DIST) files are distributed, as well as adding the following additional lines to prepare the set of needed Makefile variables:

subdir = tests
PACKAGE_NAME = @PACKAGE_NAME@
PACKAGE_TARNAME = @PACKAGE_TARNAME@
PACKAGE_VERSION = @PACKAGE_VERSION@
PACKAGE_STRING = @PACKAGE_STRING@
PACKAGE_BUGREPORT = @PACKAGE_BUGREPORT@
PACKAGE_URL = @PACKAGE_URL@
AUTOM4TE = @AUTOM4TE@

atconfig: $(top_builddir)/config.status
        cd $(top_builddir) && \
           $(SHELL) ./config.status $(subdir)/$@

atlocal: $(srcdir)/atlocal.in $(top_builddir)/config.status
        cd $(top_builddir) && \
           $(SHELL) ./config.status $(subdir)/$@

Using the above example (with or without Automake), and assuming you were careful to not initialize ‘TESTSUITEFLAGS’ within your makefile, you can now fine-tune test suite execution at runtime by altering this variable, for example:

make check TESTSUITEFLAGS='-v -d -x 75 -k AC_PROG_CC CFLAGS=-g'