A POSIX Conforming Test Framework (DejaGnu)

1.4 A POSIX compliant test framework

DejaGnu conforms to the POSIX 1003.3 standard for test frameworks. Rob Savoye was a member of that committee.

POSIX standard 1003.3 defines what a testing framework needs to provide to create a POSIX compliant testsuite. This standard is primarily oriented to checking POSIX conformance, but its requirements also support testing of features not related to POSIX conformance. POSIX 1003.3 does not specify a particular testing framework, but at this time there is only one other POSIX conforming test framework. TET was created by Unisoft for a consortium comprised of X/Open, Unix International and the Open Software Foundation.

The POSIX documentation refers to assertions. An assertion is a description of behavior. For example, if a standard says “The sun shall shine”, a corresponding assertion might be “The sun is shining.” A test based on this assertion would pass or fail depending on whether it is day or night. It is important to note that the standard being tested is never 1003.3; the standard being tested is some other standard, for which the assertions were written.

As there is no testsuite to verify that testing frameworks are POSIX 1003.3 compliant, this is done by repeatedly reading the standard and experimenting. One of the main things POSIX 1003.3 does specify is the set of allowed output messages and their definitions. Four messages are supported for a required feature of POSIX conforming systems and a fifth for a conditional feature. DejaGnu supports all five output messages. In this sense a testsuite that uses exactly these messages can be considered POSIX compliant. These definitions specify the output of a test case:

PASS

A test has succeeded. That is, it demonstrated that the assertion is true.

FAIL

A test has not succeeded – the assertion is false. The FAIL message is based on this test case only. Other messages are used to indicate a failure of the framework. As with PASS, POSIX tests must return FAIL rather than XFAIL even if a failure was expected.

XFAIL

POSIX 1003.3 does not incorporate the notion of expected failures, so PASS, instead of XPASS, must also be returned for test cases which were expected to fail and did not. This means that PASS is in some sense more ambiguous than if XPASS is also used.

UNRESOLVED

A test produced indeterminate results. Usually, this means the test executed in an unexpected fashion. This outcome requires a human to go over results to determine if the test should have passed or failed. This message is also used for any test that requires human intervention because it is beyond the abilities of the testing framework. Any unresolved test should resolved to PASS or FAIL before a test run can be considered finished.

Note that for POSIX, each assertion must produce a test result code. If the test isn’t actually run, it must produce UNRESOLVED rather than just leaving that test out of the output. This means that you have to be careful when writing tests to not carelessly use Tcl commands like return—if you alter the flow of control of the Tcl code you must insure that every test still produces some result code.

Here are some of the ways a test may wind up UNRESOLVED:

Execution of a test is interrupted.
A test does not produce a clear result. This is usually because there was an ERROR from DejaGnu while processing the test, or because there were three or more WARNING messages. Any WARNING or ERROR messages can invalidate the output of the test. This usually requires a human to examine the output to determine what really happened – and to improve the test case.
A test depends on a previous test, which has failed.
The test was set up incorrectly.
A test script aborts due to a Tcl error. In this case, the DejaGnu framework inserts an UNRESOLVED result as a placeholder for an unknown number of tests that were not run because the script crashed.

UNTESTED: A test was not run. This is a placeholder used when there is no real test case yet.

UNSUPPORTED: There is no support for the tested case. This may mean that a conditional feature of an operating system, or of a compiler, is not implemented. DejaGnu also uses this message when a testing environment (often a “bare board” target) lacks basic support for compiling or running the test case. For example, a test for the system subroutine gethostname would never work on a target board running only a boot monitor.

DejaGnu uses the same output procedures to produce these messages for all testsuites and these procedures are already known to conform to POSIX 1003.3. For a DejaGnu testsuite to conform to POSIX 1003.3, you must avoid the setup_xfail procedure as described in the PASS section above and you must be careful to return UNRESOLVED where appropriate, as described in the UNRESOLVED section above.