root/library/doc/tutorial/unit_testing.dox @ 1377

/*!
\page unit_testing Unit testing of BDM

Unit tests are simple, fast and easy to run tests of small pieces of
code (the "units") designed to reassure developers that individual
parts of the developped functionality behave as they're meant to. BDM
being an object-oriented library, the independent units are classes,
tested by calling their individual methods and comparing actual to
expected results.

Unit tests are especially useful for C++, where the simplest change
can introduce (or uncover) a difficult-to-find bug. They also
facilitate changing the internal structure of classes without
modifying their external behavior - unit tests can be run before and
after such a change to make sure the behavior had indeed stayed the
same. More ambitiously, tests can be written before implementing new
functionality, serving as its machine-checkable specification. Some
authorities go so far as to demand that "not a single line of code" be
written before having a failing unit test for it first, but BDM unit
testing isn't that demanding - it does facilitate the style for
developers who prefer it, but most BDM code is tested retroactively,
with new tests suggested by various considerations (see below).

\section Setup

Unit tests of BDM use a 3rd-party unit testing framework, UnitTest++
(from http://unittest-cpp.sourceforge.net/ ), which is designed
specifically for C++, reasonably popular and maintained (for a C++
unit testing framework), works on Unix, Windows and Mac, allows simple
test addition and proved to be readily extensible. Sources of
UnitTest++ are included in directory library/tests/unittest-cpp, so
that they can be built as part of the normal BDM build, without the
need for explicit installation and setup, and are lightly modified for
specific requirements of BDM.

Existing unit tests are collected into a testsuite executable (so that
they can all easily be run together), whose sources are in
library/tests. Apart from the top-level testsuite.cpp, testsuite
sources are generally named <classname>_test.cpp, where
<classname> is the class tested by the file (or a base class of
classes tested by the file, when these derived classes are similar
enough to be tested in the same way). BDM doesn't test abstract
classes using mock-ups (although it does test some template
instantiations on test-only classes) - all its abstract classes
already have concrete children and methods of the base class can be
tested on them.

When run, testsuite prints to standard output the names of performed
tests and error messages when they fail, which means, among other
things, that it should generally be run from the command line - even
on Windows. Note that while UnitTest++ does have support for more
elaborate GUIs, this support hasn't been extended for BDM-specific use
cases. Testsuite can also be run under debugging tools - not only
under a debugger, but also valgrind for checking memory management,
coverage tools etc.

Specific tests can be run individually, by passing their name (or
names) as a command-line argument to testsuite. Note that a test which
behaves differently depending on which tests run before it is a bug in
testsuite (this can happen e.g. when using RVs, which are global -
it's probably best to include the tested class name in their names,
although not all existing tests do that yet). Tests of sampling and
other functionality depending on random inputs can occasionally fail.
Whether that's due to skewed inputs, BDM bugs or too tight error
limits is not clear. Testsuite aims for every test passing with at
least 95% probability, but the quantification of error limits isn't
finished yet.

Not all tests depending on UnitTest++ are in testsuite. Performance
experiments have their own executables - currently just
square_mat_stress, for testing BDM matrix wrappers. square_mat_stress
is more configurable than testsuite: it doesn't generate its own
random data, but reads them from a configuration file (named
agenda.cfg by default) generated by another executable,
square_mat_prep. This allows generation (and checking) of test inputs
for various numerically demanding scenarios, which are encapsulated in
square_mat_prep's generators - that is, classes implementing its
generator interface. generators themselves are configurable using the
same scripting framework as other BDM classes, and both
square_mat_prep and square_mat_stress have command-line parameters -
run them with '-?' to see the list. Analogical support can be created
for other classes which would benefit from it. Focus of stress tests
is rather different from unit tests as generally understood - that a
class has stress tests doesn't mean it shouldn't be tested separately
inside testsuite.

\section new_unit_tests Adding new unit tests

Adding new unit tests is encouraged - it's a good way to get
acquainted with the library, and there's always space for more
tests.

Unit tests are implemented by adding a block of code (it actually
isn't just a function body, but it looks like one) starting with the
TEST macro, whose argument is the test name:

\code
TEST ( test_egiw ) {
        epdf_harness::test_config ( "egiw.cfg" );
}
\endcode

"UnitTest++.h" should be included in every test file to have the macro
defined.

Tests are usually named test_<classname>, or
test_<classname>_postfix when a class has more than one test:

\code
TEST ( test_egiw_1_2 ) {
\endcode

This is useful when a class should be thoroughly tested from multiple
aspects, e.g. for different dimensions. Exception-throwing execution
paths are also generally run by extra tests, whose names end with
_error. Note that each test in testsuite must have a unique name,
which must be a legal C++ identifier.

Testsuite has explicit support for testing classes derived from
bdm::epdf and bdm::pdf: bdm::epdf_harness and
bdm::pdf_harness. These classes run a list of tests on objects
created from the specified configuration file (normally called
<classname>.cfg) and check them against expected values also
specified in that file, so that a test using harness is just a
single-line call to its test_config method (as above).

Unit tests can also be written explicitly. The test body is normal C++
code, implementing the checks of actual against expected values using
various CHECK_* macros defined by UnitTest++:

\code
        double summ = 0.0;
        for ( int k = 0; k < n; k++ ) { // ALL b
...
        }

        CHECK_CLOSE ( 1.0, summ, 0.1 );
\endcode

Note that when checking vectors or matrices, you should also include
"mat_checks.h", which defines their comparison. Unit tests shouldn't
require any user interaction and probably shouldn't output anything
other than error reports via the UnitTest++ facilities - among other
things, UnitTest++ uses C functions for its output, and therefore a
test printing via std::cout and std::cerr may report in rather
confusing mixed order.

Developers wishing to add new tests can use various heuristics to come
up with the specific functionality to test - a selection is described
below, roughly in the order of usefulness.

\subsection existing_bugs Eliciting existing bugs

When the library has a bug, it may be useful to isolate the smallest
possible code snippet which exhibits it - if not always, then at
least as reliably as possible. Such a snippet is a good candidate for
a unit test - it can be added either before the bug is fixed, or
afterwards.

\subsection tricky_parts Testing the code known to be tricky

Every body of code has its dark corners - copy&pasted legacy code,
highly optimized algorithms, complicated interactions. These should be
tested first.

\subsection unclear_behavior Documenting and stabilizing unclear behavior

The library can behave in ways surprising its users without being
necessarily "wrong". In such cases, it's most important that all
maintainers agree on what the correct behavior should be (so that
attempts to fix it don't run in circles). Unit tests constraining the
contested functionality communicate very efficiently what their author
thinks the behavior should (or shouldn't) be.

\subsection coverage Increasing test coverage

Ideally, unit tests should test every method of every class (there are
higher ideals, but they're even less realistic). Failing that, at
least some methods of every class should be tested. Coverage tools
(e.g. gcov) can show which parts of the library are not exercised by
testsuite.

*/