6. Extending DejaGnu

6.1. Adding A New Testsuite

The testsuite for a new tool should always be located in that tools source directory. DejaGnu require the directory be named testsuite. Under this directory, the test cases go in a subdirectory whose name begins with the tool name. For example, for a tool named flubber, each subdirectory containing testsuites must start with "flubber.".

6.2. Adding A New Tool

In general, the best way to learn how to write (code or even prose) is to read something similar. This principle applies to test cases and to testsuites. Unfortunately, well-established testsuites have a way of developing their own conventions: as test writers become more experienced with DejaGnu and with Tcl, they accumulate more utilities, and take advantage of more and more features of Expect and Tcl in general.

Inspecting such established testsuites may make the prospect of creating an entirely new testsuite appear overwhelming. Nevertheless, it is quite straightforward to get a new testsuite going.

There is one testsuite that is guaranteed not to grow more elaborate over time: both it and the tool it tests were created expressly to illustrate what it takes to get started with DejaGnu. The example/ directory of the DejaGnu distribution contains both an interactive tool called calc, and a testsuite for it. Reading this testsuite, and experimenting with it, is a good way to supplement the information in this section. (Thanks to Robert Lupton for creating calc and its testsuite---and also the first version of this section of the manual!)

To help orient you further in this task, here is an outline of the steps to begin building a testsuite for a program example.

6.3. Adding A New Target

DejaGnu has some additional requirements for target support, beyond the general-purpose provisions of configure. DejaGnu must actively communicate with the target, rather than simply generating or managing code for the target architecture. Therefore, each tool requires an initialization module for each target. For new targets, you must supply a few Tcl procedures to adapt DejaGnu to the target. This permits DejaGnu itself to remain target independent.

Usually the best way to write a new initialization module is to edit an existing initialization module; some trial and error will be required. If necessary, you can use the @samp{--debug} option to see what is really going on.

When you code an initialization module, be generous in printing information controlled by the verbose procedure.

For cross targets, most of the work is in getting the communications right. Communications code (for several situations involving IP networks or serial lines) is available in a DejaGnu library file.

If you suspect a communication problem, try running the connection interactively from Expect. (There are three ways of running Expect as an interactive interpreter. You can run Expect with no arguments, and control it completely interactively; or you can use expect -i together with other command-line options and arguments; or you can run the command interpreter from any Expect procedure. Use return to get back to the calling procedure (if any), or return -tcl to make the calling procedure itself return to its caller; use exit or end-of-file to leave Expect altogether.) Run the program whose name is recorded in $connectmode, with the arguments in $targetname, to establish a connection. You should at least be able to get a prompt from any target that is physically connected.

6.4. Adding A New Board

Adding a new board consists of creating a new board config file. Examples are in dejagnu/baseboards. Usually to make a new board file, it's easiest to copy an existing one. It is also possible to have your file be based on a baseboard file with only one or two changes needed. Typically, this can be as simple as just changing the linker script. Once the new baseboard file is done, add it to the boards_DATA list in the dejagnu/baseboards/Makefile.am, and regenerate the Makefile.in using automake. Then just rebuild and install DejaGnu. You can test it by:

There is a crude inheritance scheme going on with board files, so you can include one board file into another, The two main procedures used to do this are load_generic_config and load_base_board_description. The generic config file contains other procedures used for a certain class of target. The board description file is where the board specific settings go. Commonly there are similar target environments with just different processors.

Example 40. Testing a New Board Config File


      make check RUNTESTFLAGS="--target_board=newboardfile".
      

Here's an example of a board config file. There are several helper procedures used in this example. A helper procedure is one that look for a tool of files in commonly installed locations. These are mostly used when testing in the build tree, because the executables to be tested are in the same tree as the new dejagnu files. The helper procedures are the ones in square braces [], which is the Tcl execution characters.

Example 41. Example Board Config File


      # Load the generic configuration for this board. This will define a basic
      # set of routines needed by the tool to communicate with the board.
      load_generic_config "sim"

      # basic-sim.exp is a basic description for the standard Cygnus simulator.
      load_base_board_description "basic-sim"

      # The compiler used to build for this board. This has *nothing* to do
      # with what compiler is tested if we're testing gcc.
      set_board_info compiler "[find_gcc]"

      # We only support newlib on this target.
      # However, we include libgloss so we can find the linker scripts.
      set_board_info cflags "[newlib_include_flags] [libgloss_include_flags]"
      set_board_info ldflags "[newlib_link_flags]"

      # No linker script for this board.
      set_board_info ldscript "-Tsim.ld";

      # The simulator doesn't return exit statuses and we need to indicate this.
      set_board_info needs_status_wrapper 1

      # Can't pass arguments to this target.
      set_board_info noargs 1

      # No signals.
      set_board_info gdb,nosignals 1

      # And it can't call functions.
      set_board_info gdb,cannot_call_functions 1

      

6.5. Board Config File Values

These fields are all in the board_info These are all set by using the set_board_info procedure. The parameters are the field name, followed by the value to set the field to.

Table 2. Common Board Info Fields

FieldSample ValueDescription
compiler"[find_gcc]"The path to the compiler to use.
cflags"-mca"Compilation flags for the compiler.
ldflags"[libgloss_link_flags] [newlib_link_flags]"Linking flags for the compiler.
ldscript"-Wl,-Tidt.ld"The linker script to use when cross compiling.
libs"-lgcc"Any additional libraries to link in.
shell_prompt"cygmon>"The command prompt of the remote shell.
hex_startaddr"0xa0020000"The Starting address as a string.
start_addr0xa0008000The starting address as a value.
startaddr"a0020000" 
exit_statuses_bad1Whether there is an accurate exit status.
reboot_delay10The delay between power off and power on.
unreliable1Whether communication with the board is unreliable.
sim[find_sim]The path to the simulator to use.
objcopy$tempfilThe path to the objcopy program.
support_libs"${prefix_dir}/i386-coff/"Support libraries needed for cross compiling.
addl_link_flags"-N"Additional link flags, rarely used.

These fields are used by the GCC and GDB tests, and are mostly only useful to somewhat trying to debug a new board file for one of these tools. Many of these are used only by a few testcases, and their purpose is esoteric. These are listed with sample values as a guide to better guessing if you need to change any of these.

Table 3. Board Info Fields For GCC & GDB

FieldSample ValueDescription
strip$tempfileStrip the executable of symbols.
gdb_load_offset"0x40050000" 
gdb_protocol"remote"The GDB debugging protocol to use.
gdb_sect_offset"0x41000000"; 
gdb_stub_ldscript"-Wl,-Teva-stub.ld"The linker script to use with a GDB stub.
gdb_init_command"set mipsfpu none" 
gdb,cannot_call_functions1Whether GDB can call functions on the target,
gdb,noargs1Whether the target can take command line arguments.
gdb,nosignals1Whether there are signals on the target.
gdb,short_int1 
gdb,start_symbol"_start";The starting symbol in the executable.
gdb,target_sim_options"-sparclite"Special options to pass to the simulator.
gdb,timeout540Timeout value to use for remote communication.
gdb_init_command"print/x \$fsr = 0x0" 
gdb_load_offset"0x12020000" 
gdb_opts"--command gdbinit" 
gdb_prompt"\\(gdb960\\)"The prompt GDB is using.
gdb_run_command"jump start" 
gdb_stub_offset"0x12010000" 
use_gdb_stub1Whether to use a GDB stub.
use_vma_offset1 
wrap_m68k_aout1 
gcc,no_label_values1 
gcc,no_trampolines1 
gcc,no_varargs1 
gcc,stack_size16384Stack size to use with some GCC testcases.
ieee_multilib_flags"-mieee"; 
is_simulator1 
needs_status_wrapper1 
no_double1 
no_long_long1 
noargs1 
nullstone,lib"mips-clock.c" 
nullstone,ticks_per_sec3782018 
sys_speed_value200 
target_install{sh-hms} 

6.6. Writing A Test Case

The easiest way to prepare a new test case is to base it on an existing one for a similar situation. There are two major categories of tests: batch or interactive. Batch oriented tests are usually easier to write.

The GCC tests are a good example of batch oriented tests. All GCC tests consist primarily of a call to a single common procedure, since all the tests either have no output, or only have a few warning messages when successfully compiled. Any non-warning output is a test failure. All the C code needed is kept in the test directory. The test driver, written in Tcl, need only get a listing of all the C files in the directory, and compile them all using a generic procedure. This procedure and a few others supporting for these tests are kept in the library module lib/c-torture.exp in the GCC test suite. Most tests of this kind use very few expect features, and are coded almost purely in Tcl.

Writing the complete suite of C tests, then, consisted of these steps:

Testing interactive programs is intrinsically more complex. Tests for most interactive programs require some trial and error before they are complete.

However, some interactive programs can be tested in a simple fashion reminiscent of batch tests. For example, prior to the creation of DejaGnu, the GDB distribution already included a wide-ranging testing procedure. This procedure was very robust, and had already undergone much more debugging and error checking than many recent DejaGnu test cases. Accordingly, the best approach was simply to encapsulate the existing GDB tests, for reporting purposes. Thereafter, new GDB tests built up a family of Tcl procedures specialized for GDB testing.

6.7. Debugging A Test Case

These are the kinds of debugging information available from DejaGnu:

6.8. Adding A Test Case To A Testsuite.

There are two slightly different ways to add a test case. One is to add the test case to an existing directory. The other is to create a new directory to hold your test. The existing test directories represent several styles of testing, all of which are slightly different; examine the directories for the tool of interest to see which (if any) is most suitable.

Adding a GCC test can be very simple: just add the C code to any directory beginning with gcc. and it runs on the next

runtest --tool
      gcc
.

To add a test to GDB, first add any source code you will need to the test directory. Then you can either create a new expect file, or add your test to an existing one (any file with a .exp suffix). Creating a new .exp file is probably a better idea if the test is significantly different from existing tests. Adding it as a separate file also makes upgrading easier. If the C code has to be already compiled before the test will run, then you'll have to add it to the Makefile.in file for that test directory, then run configure and make.

Adding a test by creating a new directory is very similar:

6.9. Hints On Writing A Test Case

It is safest to write patterns that match all the output generated by the tested program; this is called closure. If a pattern does not match the entire output, any output that remains will be examined by the next expect command. In this situation, the precise boundary that determines which expect command sees what is very sensitive to timing between the Expect task and the task running the tested tool. As a result, the test may sometimes appear to work, but is likely to have unpredictable results. (This problem is particularly likely for interactive tools, but can also affect batch tools---especially for tests that take a long time to finish.) The best way to ensure closure is to use the -re option for the expect command to write the pattern as a full regular expressions; then you can match the end of output using a $. It is also a good idea to write patterns that match all available output by using .*\ after the text of interest; this will also match any intervening blank lines. Sometimes an alternative is to match end of line using \r or \n, but this is usually too dependent on terminal settings.

Always escape punctuation, such as ( or ", in your patterns; for example, write \(. If you forget to escape punctuation, you will usually see an error message like

extra
      characters after close-quote.

If you have trouble understanding why a pattern does not match the program output, try using the --debug option to runtest, and examine the debug log carefully.

Be careful not to neglect output generated by setup rather than by the interesting parts of a test case. For example, while testing GDB, I issue a send set height 0\n command. The purpose is simply to make sure GDB never calls a paging program. The set height command in GDB does not generate any output; but running any command makes GDB issue a new (gdb) prompt. If there were no expect command to match this prompt, the output (gdb) begins the text seen by the next expect command---which might make that pattern fail to match.

To preserve basic sanity, I also recommended that no test ever pass if there was any kind of problem in the test case. To take an extreme case, tests that pass even when the tool will not spawn are misleading. Ideally, a test in this sort of situation should not fail either. Instead, print an error message by calling one of the DejaGnu procedures error or warning.

6.10. Special variables used by test cases.

There are special variables used by test cases. These contain other information from DejaGnu. Your test cases can use these variables, with conventional meanings (as well as the variables saved in site.exp. You can use the value of these variables, but they should never be changed.

$prms_id

The tracking system (e.g. GNATS) number identifying a corresponding bugreport. (0} if you do not specify it in the test script.)

$item bug_id

An optional bug id; may reflect a bug identification from another organization. (0 if you do not specify it.)

$subdir

The subdirectory for the current test case.

$expect_out(buffer)

The output from the last command. This is an internal variable set by Expect. More information can be found in the Expect manual.

$exec_output

This is the output from a ${tool}_load command. This only applies to tools like GCC and GAS which produce an object file that must in turn be executed to complete a test.

$comp_output

This is the output from a ${tool}_start command. This is conventionally used for batch oriented programs, like GCC and GAS, that may produce interesting output (warnings, errors) without further interaction.