Cppunit effort

From FlightGear wiki
Jump to navigation Jump to search
cppunit-foo
Started in 07/2017
Description foo
Contributor(s) Bugman
Status Under active development as of 04/2018
Folders foo
Changelog foo
This article is a stub. You can help the wiki by expanding it.

Status

For related postings, see: https://sourceforge.net/p/flightgear/mailman/search/?q=cppunit

Objective

The aim is to make it very easy for people to write unit or system/functional tests for fgfs (rather than simgear), which should make it much easier for new developers to dive into the flightgear sources. A long term goal might be to have good test coverage of the whole binary so that fgfs is harder to break and is more stable.[1]

Background

Edward developed this as part of his FGPythonSys experiments to have full system test coverage. The test suite is based on the venerable, yet old CppUnit framework. Edward has spun this out into its own standalone branches rebased to 2016.4.0 and made independent of embedded Python.

The test suite is an almost direct Python->C++ translation my own comprehensive test suite for one of my own software projects [4]. The benefits of such a framework, assuming a good test coverage, include:

  • Enabling quick and brutal refactorisation of the entire code base - ensuring that the end result works perfectly well.
  • Interest in independence and modularisation of all FlightGear components to simplify tests.
  • If all subsystems are tested for standalone creation, destruction, init(), reinit(), shutdown(), etc., then repetitive resetting of FlightGear should work flawlessly.
  • Allow valgrind and other heavy tools to be run on the absolute minimal code paths, by running individual tests.

The functionality of the test suite can be very much expanded in the future. For example there are CppUnit classes for XML output, rather than text, so this test suite is ideal for automated testing and presenting the results through web pages (e.g. Jenkins).

As a demonstration there are now 4 initial categories of tests:

  1. System/functional tests (currently 0 tests).
  2. Unit tests (currently 2 tests).
  3. GUI tests (currently 0 tests).
  4. Simgear unit tests (currently 2 test).

There are an addition 242 unit tests in the FGPythonSys branches. One of the unit tests and one of the simgear unit tests are the old tests/test-mktime.cxx and tests/test-up.cxx programs respectively (the rest of the tests/ directory might be better suited for FGMeta). A list of the test suite features are:

  • Run 'make test_suite' to build the test suite as the $BUILD_DIR/test_suite/run_test_suite standalone binary (with full independence from fgfs) and to run the test suite.
  • The absence of CppUnit causes 'CppUnit not found' to be printed by CMake, then nothing happens.
  • No CMake installation target, so it remains in the build directory.
  • A setUp() function called before every test that can be customised for each grouping or suite of tests (for FGPythonSys, set up a new subsystem instance, and inject an empty property tree).
  • A tearDown() function pre suite unconditionally called after every test (for FGPythonSys, delete the subsystem).
  • Minimal output of '.', 'F', and 'E' per test for a pass, fail, or error state [6].
  • Comprehensive simgear logstream and STDOUT+STDERR capture and reporting only for failed tests [7] (printf() function calls cannot be captured).

A final synopsis of all tests.

For full details, see the branches. The framework is powerful enough to bring in all types of tests under one roof, including all those with main() functions and currently scattered all over the place.

[2]

Challenges

The hardest one would be to set up and run only a small subset of subsystems and other infrastructure. But it is possible to write unit tests now. Writing a system/functional test is much more difficult but, with a bit of work, it should be possible.[3]


Roadmap

The idea is to make it possible to set up a single subsystem and all its dependencies in a few lines of test code. Then it should be easy to write a test. This is the dependency arrays, subsystem manager/factory fusion, and dynamic dependency resolution algorithm discussed in another thread. Edward is currently working on some ideas, he is also working on subsystem robustness testing for the test suite. This will make it much easier to work with isolated subsystems or sets of subsystems, as many subsystems are quite brittle and if you run them in a way unintended compared to how they are run in fgfs, you see segfaults all over the place.[4]

Structure

Another point is that one new test category curently being considered is adding a FGData Nasal script category, where the FGNasalSys subsystem is repetitively initialised and the FGData code is then extensively tested, these tests should be probably not be in $FG_ROOT/Nasal/. [5]

Related

References
  1. Edward d'Auvergne  (Apr 1st, 2018).  Re: [Flightgear-devel] Questions about TestSuite .
  2. Edward d'Auvergne  (Feb 11th, 2017).  [Flightgear-devel] A FlightGear test suite using CppUnit. .
  3. Edward d'Auvergne  (Apr 1st, 2018).  Re: [Flightgear-devel] Questions about TestSuite .
  4. Edward d'Auvergne  (Apr 1st, 2018).  Re: [Flightgear-devel] Questions about TestSuite .
  5. Edward d'Auvergne  (Mar 18th, 2017).  Re: [Flightgear-devel] A FlightGear test suite using CppUnit. .