Next: , Up: Runtest  


2.2.1 Output States

runtest flags the outcome of each test as one of these cases. See A POSIX Conforming Test Framework for a discussion of how POSIX specifies the meanings of these cases.

PASS

The most desirable outcome: the test was expected to succeed and did succeed.

XPASS

A pleasant kind of failure: a test was expected to fail, but succeeded. This may indicate progress; inspect the test case to determine whether you should amend it to stop expecting failure.

FAIL

A test failed, although it was expected to succeed. This may indicate regress; inspect the test case and the failing software to locate the bug.

XFAIL

A test failed, but it was expected to fail. This result indicates no change in a known bug. If a test fails because the operating system where the test runs lacks some facility required by the test, the outcome is UNSUPPORTED instead.

UNRESOLVED

Output from a test requires manual inspection; the testsuite could not automatically determine the outcome. For example, your tests can report this outcome is when a test does not complete as expected.

UNTESTED

A test case is not yet complete, and in particular cannot yet produce a PASS or FAIL. You can also use this outcome in dummy “tests” that note explicitly the absence of a real test case for a particular property.

UNSUPPORTED

A test depends on a conditionally available feature that does not exist (in the configured testing environment). For example, you can use this outcome to report on a test case that does not work on a particular target because its operating system support does not include a required subroutine.

runtest may also display the following messages:

ERROR

Indicates a major problem (detected by the test case itself) in running the test. This is usually an unrecoverable error, such as a missing file or loss of communication to the target. (POSIX testsuites should not emit this message; use UNSUPPORTED, UNTESTED, or UNRESOLVED instead, as appropriate.)

WARNING

Indicates a possible problem in running the test. Usually warnings correspond to recoverable errors, or display an important message about the following tests.

NOTE

An informational message about the test case.


Next: , Up: Runtest