5. Customizing DejaGnu

The site configuration file, site.exp, captures configuration-dependent values and propagates them to the DejaGnu test environment using Tcl variables. This ties the DejaGnu test scripts into the configure and make programs. If this file is setup correctly, it is possible to execute a testsuite merely by typing runtest.

DejaGnu supports two site.exp files. The multiple instances of site.exp are loaded in a fixed order built into DejaGnu. The first file loaded is the local file site.exp, and then the optional global site.exp file as pointed to by the DEJAGNU environment variable.

There is an optional master site.exp, capturing configuration values that apply to DejaGnu across the board, in each configuration-specific subdirectory of the DejaGnu library directory. runtest loads these values first. The master site.exp contains the default values for all targets and hosts supported by DejaGnu. This master file is identified by setting the environment variable DEJAGNU to the name of the file. This is also referred to as the ``global'' config file.

Any directory containing a configured testsuite also has a local site.exp, capturing configuration values specific to the tool under test. Since runtest loads these values last, the individual test configuration can either rely on and use, or override, any of the global values from the global site.exp file.

You can usually generate or update the testsuite's local site.exp by typing make site.exp in the testsuite directory, after the test suite is configured.

You can also have a file in your home directory called .dejagnurc. This gets loaded first before the other config files. Usually this is used for personal stuff, like setting the all_flag so all the output gets printed, or your own verbosity levels. This file is usually restricted to setting command line options.

You can further override the default values in a user-editable section of any site.exp, or by setting variables on the runtest command line.

5.1. Local Config File

It is usually more convenient to keep these manual overrides in the site.exp local to each test directory, rather than in the global site.exp in the installed DejaGnu library. This file is mostly for supplying tool specific info that is required by the testsuite.

All local site.exp files have two sections, separated by comment text. The first section is the part that is generated by make. It is essentially a collection of Tcl variable definitions based on Makefile environment variables. Since they are generated by make, they contain the values as specified by configure. (You can also customize these values by using the --site option to configure.) In particular, this section contains the Makefile variables for host and target configuration data. Do not edit this first section; if you do, your changes are replaced next time you run make.

Example 23. The first section starts with

	## these variables are automatically generated by make ##
	# Do not edit here. If you wish to override these values
	# add them to the last section

In the second section, you can override any default values (locally to DejaGnu) for all the variables. The second section can also contain your preferred defaults for all the command line options to runtest. This allows you to easily customize runtest for your preferences in each configured test-suite tree, so that you need not type options repeatedly on the command line. (The second section may also be empty, if you do not wish to override any defaults.)

Example 24. The first section ends with this line

	## All variables above are generated by configure. Do Not Edit ##

You can make any changes under this line. If you wish to redefine a variable in the top section, then just put a duplicate value in this second section. Usually the values defined in this config file are related to the configuration of the test run. This is the ideal place to set the variables host_triplet, build_triplet, target_triplet. All other variables are tool dependent, i.e., for testing a compiler, the value for CC might be set to a freshly built binary, as opposed to one in the user's path.

Here's an example local site.exp file, as used for GCC/G++ testing.

Example 25. Local Config File

      ## these variables are automatically generated by make ##
      # Do not edit here. If you wish to override these values
      # add them to the last section
      set rootme "/build/devo-builds/i586-pc-linux-gnulibc1/gcc"
      set host_triplet i586-pc-linux-gnulibc1
      set build_triplet i586-pc-linux-gnulibc1
      set target_triplet i586-pc-linux-gnulibc1
      set target_alias i586-pc-linux-gnulibc1
      set CFLAGS ""
      set CXXFLAGS "-isystem /build/devo-builds/i586-pc-linux-gnulibc1/gcc/../libio -isystem $srcdir/../libg++/src -isystem $srcdir/../libio -isystem $srcdir/../libstdc++ -isystem $srcdir/../libstdc++/stl -L/build/devo-builds/i586-pc-linux-gnulibc1/gcc/../libg++ -L/build/devo-builds/i586-pc-linux-gnulibc1/gcc/../libstdc++"
      append LDFLAGS " -L/build/devo-builds/i586-pc-linux-gnulibc1/gcc/../ld"
      set tmpdir /build/devo-builds/i586-pc-linux-gnulibc1/gcc/testsuite
      set srcdir "${srcdir}/testsuite"
      ## All variables above are generated by configure. Do Not Edit ##


This file defines the required fields for a local config file, namely the three config triplets, and the srcdir. It also defines several other Tcl variables that are used exclusively by the GCC testsuite. For most test cases, the CXXFLAGS and LDFLAGS are supplied by DejaGnu itself for cross testing, but to test a compiler, GCC needs to manipulate these itself.

5.2. Global Config File

The master config file is where all the target specific config variables for a whole site get set. The idea is that for a centralized testing lab where people have to share a target between multiple developers. There are settings for both remote targets and remote hosts. Here's an example of a Master Config File (also called the Global config file) for a canadian cross. A canadian cross is when you build and test a cross compiler on a machine other than the one it's to be hosted on.

Here we have the config settings for our California office. Note that all config values are site dependent. Here we have two sets of values that we use for testing m68k-aout cross compilers. As both of these target boards has a different debugging protocol, we test on both of them in sequence.

Example 26. Global Config file

      # Make sure we look in the right place for the board description files.
      if ![info exists boards_dir] {
          set boards_dir {}
      lappend boards_dir "/nfs/cygint/s1/cygnus/dejagnu/boards"

      verbose "Global Config File: target_triplet is $target_triplet" 2
      global target_list

      case "$target_triplet" in {
          { "native" } {
              set target_list "unix"
          { "sparc64-*elf" } {
              set target_list "sparc64-sim"
          { "mips-*elf" } {
              set target_list "mips-sim wilma barney"
          { "mips-lsi-elf" } {
              set target_list "mips-lsi-sim{,soft-float,el}"
          { "sh-*hms" } {
              set target_list { "sh-hms-sim" "bloozy" }

In this case, we have support for several cross compilers, that all run on this host. For testing on operating systems that don't support Expect, DejaGnu can be run on the local build machine, and it can connect to the remote host and run all the tests for this cross compiler on that host. All the remote OS requires is a working telnetd.

As you can see, all one does is set the variable target_list to the list of targets and options to test. The simple settings, like for sparc64-elf only require setting the name of the single board config file. The mips-elf target is more complicated. Here it sets the list to three target boards. One is the default mips target, and both wilma barney are symbolic names for other mips boards. Symbolic names are covered in the Adding A New Board chapter. The more complicated example is the one for mips-lsi-elf. This one runs the tests with multiple iterations using all possible combinations of the --soft-float and the --el (little endian) option. Needless to say, this last feature is mostly compiler specific.

5.3. Board Config File

The board config file is where board specific config data is stored. A board config file contains all the higher-level configuration settings. There is a rough inheritance scheme, where it is possible to base a new board description file on an existing one. There are also collections of custom procedures for common environments. For more information on adding a new board config file, go to the Adding A New Board chapter.

An example board config file for a GNU simulator is as follows. set_board_info is a procedure that sets the field name to the specified value. The procedures in square brackets [] are helper procedures. These are used to find parts of a tool chain required to build an executable image that may reside in various locations. This is mostly of use for when the startup code, the standard C libraries, or the tool chain itself is part of your build tree.

Example 27. Board Config File

      # This is a list of toolchains that are supported on this board.
      set_board_info target_install {sparc64-elf}

      # Load the generic configuration for this board. This will define any
      # routines needed by the tool to communicate with the board.
      load_generic_config "sim"

      # We need this for find_gcc and *_include_flags/*_link_flags.
      load_base_board_description "basic-sim"

      # Use long64 by default.
      process_multilib_options "long64"

      setup_sim sparc64

      # We only support newlib on this target. We assume that all multilib
      # options have been specified before we get here.
      set_board_info compiler  "[find_gcc]"
      set_board_info cflags  "[libgloss_include_flags] [newlib_include_flags]"
      set_board_info ldflags  "[libgloss_link_flags] [newlib_link_flags]"
      # No linker script.
      set_board_info ldscript "";

      # Used by a few gcc.c-torture testcases to delimit how large the
      # stack can be.
      set_board_info gcc,stack_size 16384
      # The simulator doesn't return exit statuses and we need to indicate this
      # the standard GCC wrapper will work with this target.
      set_board_info needs_status_wrapper 1
      # We can't pass arguments to programs.
      set_board_info noargs 1

There are five helper procedures used in this example. The first one, find gcc looks for a copy of the GNU compiler in your build tree, or it uses the one in your path. This will also return the proper transformed name for a cross compiler if you whole build tree is configured for one. The next helper procedures are libgloss_include_flags & libgloss_link_flags. These return the proper flags to compiler and link an executable image using Libgloss, the GNU BSP (Board Support Package). The final procedures are newlib_include_flag & newlib_include_flag. These find the Newlib C library, which is a reentrant standard C library for embedded systems comprising of non GPL'd code.

5.4. Remote Host Testing


Thanks to Dj Delorie for the original paper that this section is based on.

DejaGnu also supports running the tests on a remote host. To set this up, the remote host needs an ftp server, and a telnet server. Currently foreign operating systems used as remote hosts are VxWorks, VRTX, DOS/Windows 3.1, MacOS and Windows.

The recommended source for a Windows-based FTP server is to get IIS (either IIS 1 or Personal Web Server) from http://www.microsoft.com. When you install it, make sure you install the FTP server - it's not selected by default. Go into the IIS manager and change the FTP server so that it does not allow anonymous FTP. Set the home directory to the root directory (i.e. c:\) of a suitable drive. Allow writing via FTP.

It will create an account like IUSR_FOOBAR where foobar is the name of your machine. Go into the user editor and give that account a password that you don't mind hanging around in the clear (i.e. not the same as your admin or personal passwords). Also, add it to all the various permission groups.

You'll also need a telnet server. For Windows, go to the Ataman web site, pick up the Ataman Remote Logon Services for Windows, and install it. You can get started on the eval period anyway. Add IUSR_FOOBAR to the list of allowed users, set the HOME directory to be the same as the FTP default directory. Change the Mode prompt to simple.

Ok, now you need to pick a directory name to do all the testing in. For the sake of this example, we'll call it piggy (i.e. c:\piggy). Create this directory.

You'll need a unix machine. Create a directory for the scripts you'll need. For this example, we'll use /usr/local/swamp/testing. You'll need to have a source tree somewhere, say /usr/src/devo. Now, copy some files from releng's area in SV to your machine:

Example 28. Remote host setup

      cd /usr/local/swamp/testing
      mkdir boards
      scp darkstar.welcomehome.org:/dejagnu/cst/bin/MkTestDir .
      scp darkstar.welcomehome.org:/dejagnu/site.exp .
      scp darkstar.welcomehome.org:/dejagnu/boards/useless98r2.exp boards/foobar.exp
      export DEJAGNU=/usr/local/swamp/testing/site.exp


You must edit the boards/foobar.exp file to reflect your machine; change the hostname (foobar.com), username (iusr_foobar), password, and ftp_directory (c:/piggy) to match what you selected.

Edit the global site.exp to reflect your boards directory:

Example 29. Add The Board Directory

	lappend boards_dir "/usr/local/swamp/testing/boards"

Now run MkTestDir, which is in the contrib directory. The first parameter is the toolchain prefix, the second is the location of your devo tree. If you are testing a cross compiler (ex: you have sh-hms-gcc.exe in your PATH on the PC), do something like this:

Example 30. Setup Cross Remote Testing

	./MkTestDir sh-hms /usr/dejagnu/src/devo

If you are testing a native PC compiler (ex: you have gcc.exe in your PATH on the PC), do this:

Example 31. Setup Native Remote Testing

	./MkTestDir '' /usr/dejagnu/src/devo

To test the setup, ftp to your PC using the username (iusr_foobar) and password you selected. CD to the test directory. Upload a file to the PC. Now telnet to your PC using the same username and password. CD to the test directory. Make sure the file is there. Type "set" and/or "gcc -v" (or sh-hms-gcc -v) and make sure the default PATH contains the installation you want to test.

Example 32. Run Test Remotely

	cd /usr/local/swamp/testing
	make  -k -w check RUNTESTFLAGS="--host_board foobar --target_board foobar -v -v" > check.out 2>&1

To run a specific test, use a command like this (for this example, you'd run this from the gcc directory that MkTestDir created):

Example 33. Run a Test Remotely

	make check RUNTESTFLAGS="--host_board sloth --target_board sloth -v compile.exp=921202-1.c"

Note: if you are testing a cross-compiler, put in the correct target board. You'll also have to download more .exp files and modify them for your local configuration. The -v's are optional.

5.5. Config File Values

DejaGnu uses a named array in Tcl to hold all the info for each machine. In the case of a canadian cross, this means host information as well as target information. The named array is called target_info, and it has two indices. The following fields are part of the array.

5.5.1. Command Line Option Variables

In the user editable second section of the Personal Config File you can not only override the configuration variables captured in the first section, but also specify default values for all on the runtest command line options. Save for --debug, --help, and --version, each command line option has an associated Tcl variable. Use the Tcl set command to specify a new default value (as for the configuration variables). The following table describes the correspondence between command line options and variables you can set in site.exp. Invoking Runtest, for explanations of the command-line options.

Table 1. Tcl Variables For Command Line Options

--allall_flagdisplay all test results if set  
--baudbaudset the default baud rate to something other than 9600.  
--connectconnectmoderlogin, telnet, rsh, kermit, tip, or mondfe  
--outdiroutdirdirectory for tool.sum and tool.log.  
--objdirobjdirdirectory for pre-compiled binaries  
--rebootrebootreboot the target if set to "1"; do not reboot if set to "0" (the default).  
--srcdirsrcdirdirectory of test subdirectories  
--stracetracelevela number: Tcl trace depth  
--tooltoolname of tool to test; identifies init, test subdir  
--verboseverboseverbosity level. As option, use multiple times; as variable, set a number, 0 or greater.  
--targettarget_tripletThe canonical configuration string for the target.  
--hosthost_tripletThe canonical configuration string for the host.  
--buildbuild_tripletThe canonical configuration string for the build host.  
--mailaddressEmail the output log to the specified address.  

5.5.2. Personal Config File

The personal config file is used to customize runtest's behaviour for each person. It is typically used to set the user preferred setting for verbosity, and any experimental Tcl procedures. My personal ~/.dejagnurc file looks like:

Example 34. Personal Config File

	set all_flag 1
	set RLOGIN /usr/ucb/rlogin
	set RSH /usr/local/sbin/ssh

Here I set all_flag so I see all the test cases that PASS along with the ones that FAIL. I also set RLOGIN to the BSD version. I have Kerberos installed, and when I rlogin to a target board, it usually isn't supported. So I use the non secure version rather than the default that's in my path. I also set RSH to the SSH secure shell, as rsh is mostly used to test unix machines within a local network here.