Next: Invoking runtest, Up: Running runtest [Contents][Index]
runtest flags the outcome of each test as one of these cases. See A POSIX compliant test framework for a discussion of how POSIX specifies the meanings of these cases.
The most desirable outcome: the test was expected to succeed and did succeed.
A pleasant kind of failure: a test was expected to fail, but succeeded. This may indicate progress; inspect the test case to determine whether you should amend it to stop expecting failure.
A test failed, although it was expected to succeed. This may indicate regress; inspect the test case and the failing software to locate the bug.
A test failed, but it was expected to fail. This result indicates no change in a known bug. If a test fails because the operating system where the test runs lacks some facility required by the test, the outcome is UNSUPPORTED instead.
Output from a test requires manual inspection; the testsuite could not automatically determine the outcome. For example, your tests can report this outcome is when a test does not complete as expected.
A test case is not yet complete, and in particular cannot yet produce a PASS or FAIL. You can also use this outcome in dummy “tests” that note explicitly the absence of a real test case for a particular property.
A test depends on a conditionally available feature that does not exist (in the configured testing environment). For example, you can use this outcome to report on a test case that does not work on a particular target because its operating system support does not include a required subroutine.
runtest may also display the following messages:
Indicates a major problem (detected by the test case itself) in running the test. This is usually an unrecoverable error, such as a missing file or loss of communication to the target. POSIX testsuites should not emit this message; use UNSUPPORTED, UNTESTED, or UNRESOLVED instead, as appropriate.
Indicates a possible problem in running the test. Usually warnings correspond to recoverable errors, or display an important message about the following tests.
An informational message about the test case.
Next: Invoking runtest, Up: Running runtest [Contents][Index]