Next: , Up: Test and Demo Functions   [Contents][Index]


B.1 Test Functions

Command: test name
Command: test name quiet|normal|verbose
Function File: test ("name", "quiet|normal|verbose", fid)
Function File: test ([], "explain", fid)
Function File: success = test (…)
Function File: [n, max] = test (…)
Function File: [code, idx] = test ("name", "grabdemo")

Perform tests from the first file in the loadpath matching name. test can be called as a command or as a function. Called with a single argument name, the tests are run interactively and stop after the first error is encountered.

With a second argument the tests which are performed and the amount of output is selected.

"quiet"

Don’t report all the tests as they happen, just the errors.

"normal"

Report all tests as they happen, but don’t do tests which require user interaction.

"verbose"

Do tests which require user interaction.

The argument fid can be used to allow batch processing. Errors can be written to the already open file defined by fid, and hopefully when Octave crashes this file will tell you what was happening when it did. You can use stdout if you want to see the results as they happen. You can also give a file name rather than an fid, in which case the contents of the file will be replaced with the log from the current test.

Called with a single output argument success, test returns true if all of the tests were successful. Called with two output arguments n and max, the number of successful tests and the total number of tests in the file name are returned.

If the second argument is the string "grabdemo", the contents of the demo blocks are extracted but not executed. Code for all code blocks is concatenated and returned as code with idx being a vector of positions of the ends of the demo blocks.

If the second argument is "explain", then name is ignored and an explanation of the line markers used is written to the file fid.

See also: assert, fail, error, demo, example.

test scans the named script file looking for lines which start with the identifier ‘%!’. The prefix is stripped off and the rest of the line is processed through the Octave interpreter. If the code generates an error, then the test is said to fail.

Since eval() will stop at the first error it encounters, you must divide your tests up into blocks, with anything in a separate block evaluated separately. Blocks are introduced by valid keywords like test, function, or assert immediately following ‘%!’. A block is defined by indentation as in Python. Lines beginning with ‘%!<whitespace>’ are part of the preceeding block.

For example:

%!test error ("this test fails!");
%!test "test doesn't fail. it doesn't generate an error";

When a test fails, you will see something like:

  ***** test error ("this test fails!")
!!!!! test failed
this test fails!

Generally, to test if something works, you want to assert that it produces a correct value. A real test might look something like

%!test
%! a = [1, 2, 3; 4, 5, 6]; B = [1; 2];
%! expect = [ a ; 2*a ];
%! get = kron (b, a);
%! if (any (size (expect) != size (get)))
%!   error ("wrong size: expected %d,%d but got %d,%d",
%!          size (expect), size (get));
%! elseif (any (any (expect != get)))
%!   error ("didn't get what was expected.");
%! endif

To make the process easier, use the assert function. For example, with assert the previous test is reduced to:

%!test
%! a = [1, 2, 3; 4, 5, 6]; b = [1; 2];
%! assert (kron (b, a), [ a; 2*a ]);

assert can accept a tolerance so that you can compare results absolutely or relatively. For example, the following all succeed:

%!test assert (1+eps, 1, 2*eps)          # absolute error
%!test assert (100+100*eps, 100, -2*eps) # relative error

You can also do the comparison yourself, but still have assert generate the error:

%!test assert (isempty ([]))
%!test assert ([1, 2; 3, 4] > 0)

Because assert is so frequently used alone in a test block, there is a shorthand form:

%!assert (…)

which is equivalent to:

%!test assert (…)

Occasionally a block of tests will depend on having optional functionality in Octave. Before testing such blocks the availability of the required functionality must be checked. A %!testif HAVE_XXX block will only be run if Octave was compiled with functionality ‘HAVE_XXX’. For example, the sparse single value decomposition, svds(), depends on having the ARPACK library. All of the tests for svds begin with

%!testif HAVE_ARPACK

Review config.h or octave_config_info ("features") to see some of the possible values to check.

Sometimes during development there is a test that should work but is known to fail. You still want to leave the test in because when the final code is ready the test should pass, but you may not be able to fix it immediately. To avoid unnecessary bug reports for these known failures, mark the block with xtest rather than test:

%!xtest assert (1==0)
%!xtest fail ("success=1", "error")

In this case, the test will run and any failure will be reported. However, testing is not aborted and subsequent test blocks will be processed normally. Another use of xtest is for statistical tests which should pass most of the time but are known to fail occasionally.

Each block is evaluated in its own function environment, which means that variables defined in one block are not automatically shared with other blocks. If you do want to share variables, then you must declare them as shared before you use them. For example, the following declares the variable a, gives it an initial value (default is empty), and then uses it in several subsequent tests.

%!shared a
%! a = [1, 2, 3; 4, 5, 6];
%!assert (kron ([1; 2], a), [ a; 2*a ]);
%!assert (kron ([1, 2], a), [ a, 2*a ]);
%!assert (kron ([1,2; 3,4], a), [ a,2*a; 3*a,4*a ]);

You can share several variables at the same time:

%!shared a, b

You can also share test functions:

%!function a = fn (b)
%!  a = 2*b;
%!endfunction
%!assert (fn(2), 4);

Note that all previous variables and values are lost when a new shared block is declared.

Remember that %!function begins a new block and that %!endfunction ends this block. Be aware that until a new block is started, lines starting with ‘%!<space>’ will be discarded as comments. The following is nearly identical to the example above, but does nothing.

%!function a = fn (b)
%!  a = 2*b;
%!endfunction
%! assert (fn(2), 4);

Because there is a space after ‘%!’ the assert statement does not begin a new block and this line is treated as a comment.

Error and warning blocks are like test blocks, but they only succeed if the code generates an error. You can check the text of the error is correct using an optional regular expression <pattern>. For example:

%!error <passes!> error ("this test passes!");

If the code doesn’t generate an error, the test fails. For example:

%!error "this is an error because it succeeds.";

produces

  ***** error "this is an error because it succeeds.";
!!!!! test failed: no error

It is important to automate the tests as much as possible, however some tests require user interaction. These can be isolated into demo blocks, which if you are in batch mode, are only run when called with demo or the verbose option to test. The code is displayed before it is executed. For example,

%!demo
%! t = [0:0.01:2*pi]; x = sin (t);
%! plot (t, x);
%! # you should now see a sine wave in your figure window

produces

funcname example 1:
 t = [0:0.01:2*pi]; x = sin (t);
 plot (t, x);
 # you should now see a sine wave in your figure window

Press <enter> to continue: 

Note that demo blocks cannot use any shared variables. This is so that they can be executed by themselves, ignoring all other tests.

If you want to temporarily disable a test block, put # in place of the block type. This creates a comment block which is echoed in the log file but not executed. For example:

%!#demo
%! t = [0:0.01:2*pi]; x = sin (t);
%! plot (t, x);
%! # you should now see a sine wave in your figure window

The following trivial code snippet provides examples for the use of fail, assert, error and xtest:

function output = must_be_zero (input)
  if (input != 0)
    error ("Non-zero input!")
  endif
  output = input;
endfunction

%!fail ("must_be_zero (1)");
%!assert (must_be_zero (0), 0);
%!error <Non-zero> must_be_zero (1);
%!xtest error ("This code generates an error");

When putting this a file must_be_zero.m, and running the test, we see

test must_be_zero verbose

⇒
>>>>> /path/to/must_be_zero.m
  ***** fail ("must_be_zero (1)");
  ***** assert (must_be_zero (0), 0);
  ***** error <Non-zero> must_be_zero (1);
  ***** xtest error ("This code generates an error");
!!!!! known failure
This code generates an error
PASSES 4 out of 4 tests (1 expected failures)

Block type summary:

%!test

check that entire block is correct

%!testif HAVE_XXX

check block only if Octave was compiled with feature HAVE_XXX.

%!xtest

check block, report a test failure but do not abort testing.

%!error

check for correct error message

%!warning

check for correct warning message

%!demo

demo only executes in interactive mode

%!#

comment: ignore everything within the block

%!shared x,y,z

declare variables for use in multiple tests

%!function

define a function for use in multiple tests

%!endfunction

close a function definition

%!assert (x, y, tol)

shorthand for %!test assert (x, y, tol)

You can also create test scripts for builtins and your own C++ functions. To do so put a file with the bare function name (no .m extension) in a directory in the load path and it will be discovered by the test function. Alternatively, you can embed tests directly in your C++ code:

/*
%!test disp ("this is a test")
*/

or

#if 0
%!test disp ("this is a test")
#endif

However, in this case the raw source code will need to be on the load path and the user will have to remember to type test ("funcname.cc").

Function File: assert (cond)
Function File: assert (cond, errmsg, …)
Function File: assert (cond, msg_id, errmsg, …)
Function File: assert (observed, expected)
Function File: assert (observed, expected, tol)

Produce an error if the specified condition is not met. assert can be called in three different ways.

assert (cond)
assert (cond, errmsg, …)
assert (cond, msg_id, errmsg, …)

Called with a single argument cond, assert produces an error if cond is zero. When called with more than one argument the additional arguments are passed to the error function.

assert (observed, expected)

Produce an error if observed is not the same as expected. Note that observed and expected can be scalars, vectors, matrices, strings, cell arrays, or structures.

assert (observed, expected, tol)

Produce an error if observed is not the same as expected but equality comparison for numeric data uses a tolerance tol. If tol is positive then it is an absolute tolerance which will produce an error if abs (observed - expected) > abs (tol). If tol is negative then it is a relative tolerance which will produce an error if abs (observed - expected) > abs (tol * expected). If expected is zero tol will always be interpreted as an absolute tolerance. If tol is not scalar its dimensions must agree with those of observed and expected and tests are performed on an element-wise basis.

See also: test, fail, error.

Function File: fail (code)
Function File: fail (code, pattern)
Function File: fail (code, "warning", pattern)

Return true if code fails with an error message matching pattern, otherwise produce an error. Note that code is a string and if code runs successfully, the error produced is:

          expected error <.> but got none

Code must be in the form of a string that may be passed by fail to the Octave interpreter via the evalin function, that is, a (quoted) string constant or a string variable.

If called with two arguments, the behavior is similar to fail (code), except the return value will only be true if code fails with an error message containing pattern (case sensitive). If the code fails with a different error to that given in pattern, the message produced is:

          expected <pattern>
          but got <text of actual error>

The angle brackets are not part of the output.

Called with three arguments, the behavior is similar to fail (code, pattern), but produces an error if no warning is given during code execution or if the code fails.

See also: assert.


Next: , Up: Test and Demo Functions   [Contents][Index]