Next: SRFI-64 Conditonal Test Suites and Other Advanced Features, Previous: SRFI-64 Rationale, Up: SRFI-64: A Scheme API for Test Suites [Contents][Index]
Let’s start with a simple example. This is a complete self-contained test-suite.
;; Initialize and give a name to a simple testsuite. (test-begin "vec-test") (define v (make-vector 5 99)) ;; Require that an expression evaluate to true. (test-assert (vector? v)) ;; Test that an expression is eqv? to some other expression. (test-eqv 99 (vector-ref v 2)) (vector-set! v 2 7) (test-eqv 7 (vector-ref v 2)) ;; Finish the testsuite, and report results. (test-end "vec-test")
This testsuite could be saved in its own source file. Nothing else is needed: We do not require any top-level forms, so it is easy to wrap an existing program or test to this form, without adding indentation. It is also easy to add new tests, without having to name individual tests (though that is optional).
Test cases are executed in the context of a test runner, which is
an object that accumulates and reports test results. This specification
defines how to create and use custom test runners, but implementations
should also provide a default test runner. It is suggested (but not
required) that loading the above file in a top-level environment will
cause the tests to be executed using an implementation-specified default
test runner, and test-end will cause a summary to be displayed in
an implementation-specified manner. The SRFI 64 implementation used in
Guile provides such a default test runner; running the above snippet at
the REPL prints:
*** Entering test group: vec-test *** $1 = #t * PASS: $2 = ((pass . 1)) * PASS: $3 = ((pass . 2)) * PASS: $4 = ((pass . 3)) *** Leaving test group: vec-test *** *** Test suite finished. *** *** # of expected passes : 3
It also returns the <test-runner> object.
Primitive test cases test that a given condition is true. They may have
a name. The core test case form is test-assert:
This evaluates the expression. The test passes if the result is
true; if the result is false, a test failure is reported. The test also
fails if an exception is raised, assuming the implementation has a way
to catch exceptions. How the failure is reported depends on the test
runner environment. The test-name is a string that names the test
case. (Though the test-name is a string literal in the examples,
it is an expression. It is evaluated only once.) It is used when
reporting errors, and also when skipping tests, as described below. It
is an error to invoke test-assertif there is no current test
runner.
The following forms may be more convenient than using test-assert
directly:
This is equivalent to:
(test-assert [test-name] (eqv? expected test-expr))
Similarly test-equal and test-eq are shorthand for
test-assert combined with equal? or eq?,
respectively:
Here is a simple example:
(define (mean x y) (/ (+ x y) 2.0)) (test-eqv 4 (mean 3 5))
For testing approximate equality of inexact reals
we can use test-approximate:
This is equivalent to (except that each argument is only evaluated once):
(test-assert [test-name]
(and (>= test-expr (- expected error))
(<= test-expr (+ expected error))))
Here’s an example:
(test-approximate "is 22/7 within 1% of π?" 3.1415926535 22/7 1/100)
We need a way to specify that evaluation should fail. This verifies that errors are detected when required.
Evaluating test-expr is expected to signal an error. The kind of error is indicated by error-type.
If the error-type is left out, or it is #t, it means "some
kind of unspecified error should be signaled". For example:
(test-error #t (vector-ref '#(1 2) 9))
This specification leaves it implementation-defined (or for a future
specification) what form test-error may take, though all
implementations must allow #t. Some implementations may support
SRFI-35’s
conditions, but these are only standardized for
SRFI-36’s I/O
conditions, which are seldom useful in test suites. An implementation
may also allow implementation-specific “exception types”. For example
Java-based implementations may allow the names of Java exception
classes:
;; Kawa-specific example (test-error <java.lang.IndexOutOfBoundsException> (vector-ref '#(1 2) 9))
An implementation that cannot catch exceptions should skip
test-error forms.
The SRFI-64 implementation in Guile supports specifying error-type as either:
#f, meaning the test is not expected to produce an error
#t, meaning the test is expected to produce an error, of any type
make-exception-type or
make-condition-type from SRFI-35
make-exception-from-throw style exceptions.
Below are some examples valid in Guile:
(test-error "expect old-style exception kind" 'numerical-overflow (/ 1 0))
(use-modules (ice-9 exceptions)) ;for standard exception types (test-error "expect a native exception type" &warning (raise-exception (make-warning))) (test-error "expect a native exception, using predicate" warning? (raise-exception (make-warning)))
(use-modules (srfi srfi-35)) (test-error "expect a serious SRFI 35 condition type" &serious (raise-exception (condition (&serious)))) (test-error "expect a serious SRFI 35 condition type, using predicate" serious-condition? (raise-exception (condition (&serious))))
Testing syntax is tricky, especially if we want to check that invalid syntax is causing an error. The following utility function can help:
This function parses string (using read) and evaluates the
result. The result of evaluation is returned from
test-read-eval-string. An error is signalled if there are unread
characters after the read is done. For example:
(test-read-eval-string "(+ 3 4)") evaluates to 7.
(test-read-eval-string "(+ 3 4") signals an error.
(test-read-eval-string "(+ 3 4) ") signals an error, because
there is extra “junk” (i.e. a space) after the list is read.
The test-read-eval-string used in tests:
(test-equal 7 (test-read-eval-string "(+ 3 4)")) (test-error (test-read-eval-string "(+ 3")) (test-equal #\newline (test-read-eval-string "#\\newline")) (test-error (test-read-eval-string "#\\newlin")) ;; Skip the next 2 tests unless srfi-62 is available. (test-skip (cond-expand (srfi-62 0) (else 2))) (test-equal 5 (test-read-eval-string "(+ 1 #;(* 2 3) 4)")) (test-equal '(x z) (test-read-string "(list 'x #;'y 'z)"))
A test group is a named sequence of forms containing testcases, expressions, and definitions. Entering a group sets the test group name; leaving a group restores the previous group name. These are dynamic (run-time) operations, and a group has no other effect or identity. Test groups are informal groupings: they are neither Scheme values, nor are they syntactic forms. A test group may contain nested inner test groups. The test group path is a list of the currently-active (entered) test group names, oldest (outermost) first.
A test-begin enters a new test group. The suite-name
becomes the current test group name, and is added to the end of the test
group path. Portable test suites should use a string literal for
suite-name; the effect of expressions or other kinds of literals
is unspecified.
Rationale: In some ways using symbols would be preferable. However, we want human-readable names, and standard Scheme does not provide a way to include spaces or mixed-case text in literal symbols.
The optional count must match the number of test-cases executed by this group. (Nested test groups count as a single test case for this count.) This extra test may be useful to catch cases where a test doesn’t get executed because of some unexpected error.
Additionally, if there is no currently executing test runner, one is installed in an implementation-defined manner.
A test-end leaves the current test group.
An error is reported if the suite-name does not
match the current test group name.
Additionally, if the matching test-begininstalled a new
test-runner, then the test-end will uninstall it, after reporting
the accumulated test results in an implementation-defined manner.
Equivalent to:
(if (not (test-to-skip% (var suite-name)))
(dynamic-wind
(lambda () (test-begin (var suite-name)))
(lambda () (var decl-or-expr) ...)
(lambda () (test-end (var suite-name)))))
This is usually equivalent to executing the decl-or-exprs
within the named test group. However, the entire group is skipped
if it matched an active test-skip (see later).
Also, the test-end is executed in case of an exception.
Execute each of the decl-or-expr forms in order (as in a <body>), and then execute the cleanup-form. The latter should be executed even if one of a decl-or-expr forms raises an exception (assuming the implementation has a way to catch exceptions).
For example:
(let ((f (open-output-file "log")))
(test-group-with-cleanup "test-file"
(do-a-bunch-of-tests f)
(close-output-port f)))