Unit testing of BDM

Unit tests are especially useful for C++, where the simplest change can introduce (or uncover) a difficult-to-find bug. They also facilitate changing the internal structure of classes without modifying their external behavior - unit tests can be run before and after such a change to make sure the behavior had indeed stayed the same. More ambitiously, tests can be written before implementing new functionality, serving as its machine-checkable specification. Some authorities go so far as to demand that "not a single line of code" be written before having a failing unit test for it first, but BDM unit testing isn't that demanding - it does facilitate the style for developers who prefer it, but most BDM code is tested retroactively, with new tests suggested by various considerations (see below).

Setup

Existing unit tests are collected into a testsuite executable (so that they can all easily be run together), whose sources are in library/tests. Apart from the top-level testsuite.cpp, testsuite sources are generally named <classname>_test.cpp, where <classname> is the class tested by the file (or a base class of classes tested by the file, when these derived classes are similar enough to be tested in the same way). BDM doesn't test abstract classes using mock-ups (although it does test some template instantiations on test-only classes) - all its abstract classes already have concrete children and methods of the base class can be tested on them.

When run, testsuite prints to standard output the names of performed tests and error messages when they fail, which means, among other things, that it should generally be run from the command line - even on Windows. Note that while UnitTest++ does have support for more elaborate GUIs, this support hasn't been extended for BDM-specific use cases. Testsuite can also be run under debugging tools - not only under a debugger, but also valgrind for checking memory management, coverage tools etc.

Specific tests can be run individually, by passing their name (or names) as a command-line argument to testsuite. Note that a test which behaves differently depending on which tests run before it is a bug in testsuite (this can happen e.g. when using RVs, which are global - it's probably best to include the tested class name in their names, although not all existing tests do that yet). Tests of sampling and other functionality depending on random inputs can occasionally fail. Whether that's due to skewed inputs, BDM bugs or too tight error limits is not clear. Testsuite aims for every test passing with at least 95% probability, but the quantification of error limits isn't finished yet.

Not all tests depending on UnitTest++ are in testsuite. Performance experiments have their own executables - currently just square_mat_stress, for testing BDM matrix wrappers. square_mat_stress is more configurable than testsuite: it doesn't generate its own random data, but reads them from a configuration file (named agenda.cfg by default) generated by another executable, square_mat_prep. This allows generation (and checking) of test inputs for various numerically demanding scenarios, which are encapsulated in square_mat_prep's generators - that is, classes implementing its generator interface. generators themselves are configurable using the same scripting framework as other BDM classes, and both square_mat_prep and square_mat_stress have command-line parameters - run them with '-?' to see the list. Analogical support can be created for other classes which would benefit from it. Focus of stress tests is rather different from unit tests as generally understood - that a class has stress tests doesn't mean it shouldn't be tested separately inside testsuite.

Adding new unit tests

Unit tests are implemented by adding a block of code (it actually isn't just a function body, but it looks like one) starting with the TEST macro, whose argument is the test name:

TEST ( test_egiw ) {
        epdf_harness::test_config ( "egiw.cfg" );
}

"UnitTest++.h" should be included in every test file to have the macro defined.

Tests are usually named test_<classname>, or test_<classname>_postfix when a class has more than one test:

TEST ( test_egiw_1_2 ) {

This is useful when a class should be thoroughly tested from multiple aspects, e.g. for different dimensions. Exception-throwing execution paths are also generally run by extra tests, whose names end with _error. Note that each test in testsuite must have a unique name, which must be a legal C++ identifier.

Testsuite has explicit support for testing classes derived from bdm::epdf and bdm::pdf: bdm::epdf_harness and bdm::pdf_harness. These classes run a list of tests on objects created from the specified configuration file (normally called <classname>.cfg) and check them against expected values also specified in that file, so that a test using harness is just a single-line call to its test_config method (as above).

Unit tests can also be written explicitly. The test body is normal C++ code, implementing the checks of actual against expected values using various CHECK_* macros defined by UnitTest++:

        double summ = 0.0;
        for ( int k = 0; k < n; k++ ) { // ALL b
...
        }
        CHECK_CLOSE ( 1.0, summ, 0.1 );

Note that when checking vectors or matrices, you should also include "mat_checks.h", which defines their comparison. Unit tests shouldn't require any user interaction and probably shouldn't output anything other than error reports via the UnitTest++ facilities - among other things, UnitTest++ uses C functions for its output, and therefore a test printing via std::cout and std::cerr may report in rather confusing mixed order.

Developers wishing to add new tests can use various heuristics to come up with the specific functionality to test - a selection is described below, roughly in the order of usefulness.

Eliciting existing bugs

Testing the code known to be tricky

Documenting and stabilizing unclear behavior

Increasing test coverage

Generated on 2 Dec 2013 for mixpp by  1.4.7