[543] | 1 | /*! |
---|
| 2 | \page unit_testing Unit testing of BDM |
---|
| 3 | |
---|
| 4 | Unit tests are simple, fast and easy to run tests of small pieces of |
---|
| 5 | code (the "units") designed to reassure developers that individual |
---|
| 6 | parts of the developped functionality behave as they're meant to. BDM |
---|
| 7 | being an object-oriented library, the independent units are classes, |
---|
| 8 | tested by calling their individual methods and comparing actual to |
---|
| 9 | expected results. |
---|
| 10 | |
---|
| 11 | Unit tests are especially useful for C++, where the simplest change |
---|
| 12 | can introduce (or uncover) a difficult-to-find bug. They also |
---|
| 13 | facilitate changing the internal structure of classes without |
---|
| 14 | modifying their external behavior - unit tests can be run before and |
---|
| 15 | after such a change to make sure the behavior had indeed stayed the |
---|
| 16 | same. More ambitiously, tests can be written before implementing new |
---|
| 17 | functionality, serving as its machine-checkable specification. Some |
---|
| 18 | authorities go so far as to demand that "not a single line of code" be |
---|
| 19 | written before having a failing unit test for it first, but BDM unit |
---|
| 20 | testing isn't that demanding - it does facilitate the style for |
---|
| 21 | developers who prefer it, but most BDM code is tested retroactively, |
---|
| 22 | with new tests suggested by various considerations (see below). |
---|
| 23 | |
---|
| 24 | \section Setup |
---|
| 25 | |
---|
| 26 | Unit tests of BDM use a 3rd-party unit testing framework, UnitTest++ |
---|
| 27 | (from http://unittest-cpp.sourceforge.net/ ), which is designed |
---|
| 28 | specifically for C++, reasonably popular and maintained (for a C++ |
---|
| 29 | unit testing framework), works on Unix, Windows and Mac, allows simple |
---|
| 30 | test addition and proved to be readily extensible. Sources of |
---|
| 31 | UnitTest++ are included in directory library/tests/unittest-cpp, so |
---|
| 32 | that they can be built as part of the normal BDM build, without the |
---|
| 33 | need for explicit installation and setup, and are lightly modified for |
---|
| 34 | specific requirements of BDM. |
---|
| 35 | |
---|
| 36 | Existing unit tests are collected into a testsuite executable (so that |
---|
| 37 | they can all easily be run together), whose sources are in |
---|
| 38 | library/tests. Apart from the top-level testsuite.cpp, testsuite |
---|
| 39 | sources are generally named <classname>_test.cpp, where |
---|
| 40 | <classname> is the class tested by the file (or a base class of |
---|
| 41 | classes tested by the file, when these derived classes are similar |
---|
| 42 | enough to be tested in the same way). BDM doesn't test abstract |
---|
| 43 | classes using mock-ups (although it does test some template |
---|
| 44 | instantiations on test-only classes) - all its abstract classes |
---|
| 45 | already have concrete children and methods of the base class can be |
---|
| 46 | tested on them. |
---|
| 47 | |
---|
| 48 | When run, testsuite prints to standard output the names of performed |
---|
| 49 | tests and error messages when they fail, which means, among other |
---|
| 50 | things, that it should generally be run from the command line - even |
---|
| 51 | on Windows. Note that while UnitTest++ does have support for more |
---|
| 52 | elaborate GUIs, this support hasn't been extended for BDM-specific use |
---|
| 53 | cases. Testsuite can also be run under debugging tools - not only |
---|
| 54 | under a debugger, but also valgrind for checking memory management, |
---|
| 55 | coverage tools etc. |
---|
| 56 | |
---|
| 57 | Specific tests can be run individually, by passing their name (or |
---|
| 58 | names) as a command-line argument to testsuite. Note that a test which |
---|
| 59 | behaves differently depending on which tests run before it is a bug in |
---|
| 60 | testsuite (this can happen e.g. when using RVs, which are global - |
---|
| 61 | it's probably best to include the tested class name in their names, |
---|
| 62 | although not all existing tests do that yet). Tests of sampling and |
---|
| 63 | other functionality depending on random inputs can occasionally fail. |
---|
| 64 | Whether that's due to skewed inputs, BDM bugs or too tight error |
---|
| 65 | limits is not clear. Testsuite aims for every test passing with at |
---|
| 66 | least 95% probability, but the quantification of error limits isn't |
---|
| 67 | finished yet. |
---|
| 68 | |
---|
| 69 | Not all tests depending on UnitTest++ are in testsuite. Performance |
---|
| 70 | experiments have their own executables - currently just |
---|
| 71 | square_mat_stress, for testing BDM matrix wrappers. square_mat_stress |
---|
| 72 | is more configurable than testsuite: it doesn't generate its own |
---|
| 73 | random data, but reads them from a configuration file (named |
---|
| 74 | agenda.cfg by default) generated by another executable, |
---|
| 75 | square_mat_prep. This allows generation (and checking) of test inputs |
---|
| 76 | for various numerically demanding scenarios, which are encapsulated in |
---|
| 77 | square_mat_prep's generators - that is, classes implementing its |
---|
| 78 | generator interface. generators themselves are configurable using the |
---|
| 79 | same scripting framework as other BDM classes, and both |
---|
| 80 | square_mat_prep and square_mat_stress have command-line parameters - |
---|
| 81 | run them with '-?' to see the list. Analogical support can be created |
---|
| 82 | for other classes which would benefit from it. Focus of stress tests |
---|
| 83 | is rather different from unit tests as generally understood - that a |
---|
| 84 | class has stress tests doesn't mean it shouldn't be tested separately |
---|
| 85 | inside testsuite. |
---|
| 86 | |
---|
| 87 | \section new_unit_tests Adding new unit tests |
---|
| 88 | |
---|
| 89 | Adding new unit tests is encouraged - it's a good way to get |
---|
| 90 | acquainted with the library, and there's always space for more |
---|
| 91 | tests. |
---|
| 92 | |
---|
| 93 | Unit tests are implemented by adding a block of code (it actually |
---|
| 94 | isn't just a function body, but it looks like one) starting with the |
---|
| 95 | TEST macro, whose argument is the test name: |
---|
| 96 | |
---|
| 97 | \code |
---|
| 98 | TEST ( test_egiw ) { |
---|
| 99 | epdf_harness::test_config ( "egiw.cfg" ); |
---|
| 100 | } |
---|
| 101 | \endcode |
---|
| 102 | |
---|
| 103 | "UnitTest++.h" should be included in every test file to have the macro |
---|
| 104 | defined. |
---|
| 105 | |
---|
| 106 | Tests are usually named test_<classname>, or |
---|
| 107 | test_<classname>_postfix when a class has more than one test: |
---|
| 108 | |
---|
| 109 | \code |
---|
| 110 | TEST ( test_egiw_1_2 ) { |
---|
| 111 | \endcode |
---|
| 112 | |
---|
| 113 | This is useful when a class should be thoroughly tested from multiple |
---|
[560] | 114 | aspects, e.g. for different dimensions. Exception-throwing execution |
---|
| 115 | paths are also generally run by extra tests, whose names end with |
---|
| 116 | _error. Note that each test in testsuite must have a unique name, |
---|
| 117 | which must be a legal C++ identifier. |
---|
[543] | 118 | |
---|
| 119 | Testsuite has explicit support for testing classes derived from |
---|
[693] | 120 | bdm::epdf and bdm::pdf: bdm::epdf_harness and |
---|
| 121 | bdm::pdf_harness. These classes run a list of tests on objects |
---|
[543] | 122 | created from the specified configuration file (normally called |
---|
| 123 | <classname>.cfg) and check them against expected values also |
---|
| 124 | specified in that file, so that a test using harness is just a |
---|
| 125 | single-line call to its test_config method (as above). |
---|
| 126 | |
---|
| 127 | Unit tests can also be written explicitly. The test body is normal C++ |
---|
| 128 | code, implementing the checks of actual against expected values using |
---|
| 129 | various CHECK_* macros defined by UnitTest++: |
---|
| 130 | |
---|
| 131 | \code |
---|
| 132 | double summ = 0.0; |
---|
| 133 | for ( int k = 0; k < n; k++ ) { // ALL b |
---|
| 134 | ... |
---|
| 135 | } |
---|
| 136 | |
---|
| 137 | CHECK_CLOSE ( 1.0, summ, 0.1 ); |
---|
| 138 | \endcode |
---|
| 139 | |
---|
| 140 | Note that when checking vectors or matrices, you should also include |
---|
| 141 | "mat_checks.h", which defines their comparison. Unit tests shouldn't |
---|
| 142 | require any user interaction and probably shouldn't output anything |
---|
| 143 | other than error reports via the UnitTest++ facilities - among other |
---|
| 144 | things, UnitTest++ uses C functions for its output, and therefore a |
---|
| 145 | test printing via std::cout and std::cerr may report in rather |
---|
| 146 | confusing mixed order. |
---|
| 147 | |
---|
| 148 | Developers wishing to add new tests can use various heuristics to come |
---|
| 149 | up with the specific functionality to test - a selection is described |
---|
| 150 | below, roughly in the order of usefulness. |
---|
| 151 | |
---|
| 152 | \subsection existing_bugs Eliciting existing bugs |
---|
| 153 | |
---|
| 154 | When the library has a bug, it may be useful to isolate the smallest |
---|
| 155 | possible code snippet which exhibits it - if not always, then at |
---|
| 156 | least as reliably as possible. Such a snippet is a good candidate for |
---|
| 157 | a unit test - it can be added either before the bug is fixed, or |
---|
| 158 | afterwards. |
---|
| 159 | |
---|
| 160 | \subsection tricky_parts Testing the code known to be tricky |
---|
| 161 | |
---|
| 162 | Every body of code has its dark corners - copy&pasted legacy code, |
---|
| 163 | highly optimized algorithms, complicated interactions. These should be |
---|
| 164 | tested first. |
---|
| 165 | |
---|
| 166 | \subsection unclear_behavior Documenting and stabilizing unclear behavior |
---|
| 167 | |
---|
| 168 | The library can behave in ways surprising its users without being |
---|
| 169 | necessarily "wrong". In such cases, it's most important that all |
---|
| 170 | maintainers agree on what the correct behavior should be (so that |
---|
| 171 | attempts to fix it don't run in circles). Unit tests constraining the |
---|
| 172 | contested functionality communicate very efficiently what their author |
---|
| 173 | thinks the behavior should (or shouldn't) be. |
---|
| 174 | |
---|
| 175 | \subsection coverage Increasing test coverage |
---|
| 176 | |
---|
| 177 | Ideally, unit tests should test every method of every class (there are |
---|
| 178 | higher ideals, but they're even less realistic). Failing that, at |
---|
| 179 | least some methods of every class should be tested. Coverage tools |
---|
| 180 | (e.g. gcov) can show which parts of the library are not exercised by |
---|
| 181 | testsuite. |
---|
| 182 | |
---|
| 183 | */ |
---|