Hi fellow wiki editors!

To help newly registered users get more familiar with the wiki (and maybe older users too) there is now a {{Welcome to the wiki}} template. Have a look at it and feel free to add it to new users discussion pages (and perhaps your own).

I have tried to keep the template short, but meaningful. /Johan G

Difference between revisions of "FlightGear Headless"

From FlightGear wiki
Jump to: navigation, search
(Milestones)
(FlightGear Headless)
 
(61 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Template:RFC Navigation}}
+
{{Non-stable|version=2018.x|progress=30}}
 +
 
 +
{{Portability Navbar}}
  
 
= FlightGear Headless =
 
= FlightGear Headless =
 +
{{Main article|Supporting multiple renderers}}
  
== Status (10/2012) ==
+
there is no document listing the minimum set of stable features. If someone wants to write one, they can do it, but I guess they won’t volunteer to test release candidates against it; indeed getting people to even run them has been a struggle.<ref>{{cite web
 +
  |url    = https://sourceforge.net/p/flightgear/mailman/message/35990557/
 +
  |title  = <nowiki> Re: [Flightgear-devel] carrier features: is there any modification ,
 +
your policy related ? </nowiki>
 +
  |author = <nowiki> James Turner </nowiki>
 +
  |date  = Aug 10th, 2017
 +
  |added  =  Aug 10th, 2017
 +
  |script_version = 0.40
 +
  }}</ref>
  
As of 10/2012, this is being worked on (also see [[FlightGear Run Levels]]):
+
with the new test framework it should be possible to add tests for carrier start, as I already did to ensure runway selection with METAR. Indeed the position init code is a very good candidate for testing this way since we broken different aspects (parking position startup, WXR runway selection, the carrier, probably some more things<ref>{{cite web
 +
  |url    =  https://sourceforge.net/p/flightgear/mailman/message/35990557/
 +
  |title  =  <nowiki> Re: [Flightgear-devel] carrier features: is there any modification ,
 +
your policy related ? </nowiki>
 +
  |author =  <nowiki> James Turner </nowiki>
 +
  |date  =  Aug 10th, 2017
 +
  |added  =  Aug 10th, 2017
 +
  |script_version = 0.40
 +
  }}</ref>
  
  
{{cquote|I'm working on a prototype of this - with no X11 / VLC required at all. It will load scenery tiles (for ground-intersections) but no views or rendering. But don't be in a rush for something to appear in Git, it's just an experiment.<ref>{{cite web |url=https://code.google.com/p/flightgear-bugs/issues/detail?id=882 |title=Issues 882 (Improved CI / regression testing support by providing options to disable interactive features (sound/window))|author=James Turner |date=Sep 26, 2012}}</ref>|James Turner}}
+
The original FG scene graph design is based on rigid assumptions of the ordering of the sequence of events for start up and shut down. Branch management is highly inconsistent, and is scattered all over the flightgear and simgear codebases. But it worked perfectly due to the fixed event sequences. The new subsystem design however brings in a much more flexible design, whereby systems can be dynamically turned on, turned off, restarted, initialised, reinitialised, etc. This is essential for enabling rebooting, and allows for custom FlightGear usage of only a small subset of systems and for '''a headless mode'''. The OSG+subsystem breakage occurred once the scene manager subsystem was fully converted to the new design. The OSG scene graph root is created at the start of FGScenery::init(). With the subsystem design, the scene graph root may not be present when other flightgear/simgear code paths are called to create, manage, or destroy the various scene graph branches.<ref>{{cite web
 +
  | url   = http://sourceforge.net/p/flightgear/mailman/message/35048478/
 +
  | title = <nowiki>[Flightgear-devel] Integration of the new subsystem design and the
 +
OSG scene graph management.</nowiki>
 +
  | author = <nowiki>Edward d'Auvergne</nowiki>
 +
  | date   = Apr 28th, 2016
 +
  | added  = Apr 28th, 2016
 +
  | script_version = 0.26
 +
  }}
 +
</ref>
  
{{cquote|Make subsystems create-able and removable from commands. Only some subsystems are supported so far, since many have non-default constructors or other complexities.
+
Another option is to make [[The FlightGear Rendering Pipeline|FGRenderer]] the optional piece - James Turner started doing some work in this direction in December 2015. I.e without FGRenderer there is no rendering at all, and hence no scene-graph.<ref>{{cite web
 +
  |url    = https://sourceforge.net/p/flightgear/mailman/message/35064249/
 +
  |title  = <nowiki>Re: [Flightgear-devel] Integration of the new subsystem design and
 +
the OSG scene graph management.</nowiki>
 +
  |author = <nowiki>James Turner</nowiki>
 +
  |date  = May 4th, 2016
 +
  |added  = May 4th, 2016
 +
  |script_version = 0.31
 +
  }}
 +
</ref>
  
With this, change, it's possible to dynamically add and remove the traffic-manager at runtime, for example: fgcommand("add-subsystem", props.Node.new({ "subsystem": "traffic-manager", "name":"traffic-manager", "do-bind-init":1}));<ref>{{cite web |url=https://gitorious.org/fg/flightgear/commit/8608a480736651999c5ea31a489343ee097ee915 |title=Initial work on dynamic subsystem creation.|author=James Turner |date=Oct 01, 2012}}</ref>|James Turner}}
+
we should really do with with subclassing, making an SGDisplaySubsystem with:  
 +
* virtual void initVisuals();
 +
* virtual void updateVisuals();
 +
* virtual void shutdownVisuals();  
  
 +
With the expectation that the manager layer (FGScenery or FGRenderer) calls initVisuals / updateVisuals / shutdownVisuals if it exists. This also forces the implementation of each subsystem to distinguish simulation pieces (modifies properties) from display pieces (reads but hopefully does not modify properties, modifies OSG/rendering state, i.e. visuals) <ref>{{cite web
 +
  |url    = https://sourceforge.net/p/flightgear/mailman/message/35064249/
 +
  |title  = <nowiki>Re: [Flightgear-devel] Integration of the new subsystem design and
 +
the OSG scene graph management.</nowiki>
 +
  |author = <nowiki>James Turner</nowiki>
 +
  |date  = May 4th, 2016
 +
  |added  = May 4th, 2016
 +
  |script_version = 0.31
 +
  }}
 +
</ref>
 +
 +
<references/>
 +
 +
== Motivation ==
 +
{{FGCquote
 +
|1= these are exactly the same types of changes I was thinking of for fully decoupling all the various parts of FlightGear for use in a CppUnit-based test suite
 +
|2= {{cite web
 +
  | url    = http://forum.flightgear.org/viewtopic.php?p=277299#p277299
 +
  | title  = <nowiki>Re: FGPython an propose for Python as an nasal alternative</nowiki>
 +
  | author = <nowiki>bugman</nowiki>
 +
  | date  = Feb 23rd, 2016
 +
  | added  = Feb 23rd, 2016
 +
  | script_version = 0.25
 +
  }}
 +
}}
 +
 +
{{FGCquote
 +
|1= what remains true is the that overwhelming majority of the FlightGear main loop time is spent rendering the graphics.
 +
|2= {{cite web
 +
  | url    = http://sourceforge.net/p/flightgear/mailman/message/34811441/
 +
  | title  = <nowiki>Re: [Flightgear-devel] Designing a thread-safe property tree API</nowiki>
 +
  | author = <nowiki>Curtis Olson</nowiki>
 +
  | date  = Feb 1st, 2016
 +
  }}
 +
}}
 +
{{FGCquote
 +
|1= That's not to say there aren't some bottlenecks in our rendering pipeline that need to eventually be dealt with, but these are more tied to our use of properties and nasal in graphics system call backs (much of this we inherited from the original design which was build around plib's SG (scene graph) library.)
 +
|2= {{cite web
 +
  | url    = http://sourceforge.net/p/flightgear/mailman/message/34811441/
 +
  | title  = <nowiki>Re: [Flightgear-devel] Designing a thread-safe property tree API</nowiki>
 +
  | author = <nowiki>Curtis Olson</nowiki>
 +
  | date  = Feb 1st, 2016
 +
  }}
 +
}}
 +
 +
[[File:Patching-fg-3.2-to-make-more-subsystems-optional.png|450px|thumb|Screen shot showing a the performance monitor in a patched version of FlightGear 3.2  where subsystem initialization is made better configurable and increasingly optional by allowing subsystems to be explicitly disabled/enabled during startup. Decoupling internal subsystem dependencies means that we can more easily provide support for [[FlightGear Benchmark|benchmarking]], but also [[FlightGear Headless|headless]] regression testing - and eventually, also a standalone [[FGCanvas]] startup mode.]]
 +
 +
{{Note|Also see [[Testing]]}}
 +
 +
{{FGCquote|1=If we want to remove an subsystem from the flight gear source code, without leaving any trace of it.. What are the essential steps ?(for example, if the sound update has to be removed, what are steps to be followed).|2= {{cite web  | url    = http://forum.flightgear.org/viewtopic.php?p=255555#p255555  | title  = <nowiki>Steps for Removing subsystems in flightgear source code</nowiki>  | author = <nowiki>Divi</nowiki>  | date  = Aug 31st, 2015  }}}}
 +
 +
{{FGCquote
 +
  |the route manager has a fairly nasty bug that results in memory consumption increasing throughout a flight. The rate at which it increases appears to increase with the number of waypoints, with a flight of 20 waypoints experiencing an increase of over 2GB per hour in my tests. The longer the flight, the more waypoints it tends to have and the more unsustainable the growth becomes.
 +
  |{{cite web |url=http://sourceforge.net/p/flightgear/mailman/message/33493896/
 +
    |title=<nowiki>Memory leak most likely from route manager in 3.4.0 and next</nowiki>
 +
    |author=<nowiki>Richard Senior</nowiki>
 +
    |date=<nowiki>2015-02-25 22:00:53</nowiki>
 +
  }}
 +
}}
 +
 +
{{FGCquote
 +
  |I am wishing I could easily add tests for the route-path / GPS code, but anything that needs ‘globals’ is hard to test in isolation.
 +
  |{{cite web |url=http://sourceforge.net/p/flightgear/mailman/message/33226290/
 +
    |title=<nowiki>Re: [Flightgear-devel] ATIS and negative temperatures (+ very fast
 +
speech)</nowiki>
 +
    |author=<nowiki>James Turner</nowiki>
 +
    |date=<nowiki>2015-01-12</nowiki>
 +
  }}
 +
}}
 +
 +
{{FGCquote
 +
  |I appreciate that would require us to have a unit-testing infrastructure, and way to bring up a subset of globals and the nav-cache, in a test environment. I have made a few abortive attempts at exactly that but the sprawling nature of FGGlobals keeps defeating me.<br/>
 +
<br/>
 +
(The ability to run many tests efficiently was actually one of the smaller reasons for introducing the nav-cache, since without it any automated testing system would be horribly slow)
 +
  |{{cite web |url=http://sourceforge.net/p/flightgear/mailman/message/34113076/
 +
    |title=<nowiki>Re: [Flightgear-devel] Major problems with the AI traffic system</nowiki>
 +
    |author=<nowiki>James Turner</nowiki>
 +
    |date=<nowiki>2015-05-14</nowiki>
 +
  }}
 +
}}
  
  
Line 22: Line 142:
 
Obviously supporting a standalone 'fgcanvas' would be quite a small extension from those. I'm not worrying about dynamic dependencies or automatic subsystem creation for the moment - I expect the user / defaults to have defined a set of subsystems that work without crashing  You're correct of course that Nasal has many assumptions about subsystems, but I think that can be improved incrementally on the Nasal side.
 
Obviously supporting a standalone 'fgcanvas' would be quite a small extension from those. I'm not worrying about dynamic dependencies or automatic subsystem creation for the moment - I expect the user / defaults to have defined a set of subsystems that work without crashing  You're correct of course that Nasal has many assumptions about subsystems, but I think that can be improved incrementally on the Nasal side.
  
For the test mode, I really want to start Nasal-the-langauge very early, without loading all the modules in Nasal/ immediately. (Or maybe load a 'safe' subset). That's going to take some thought and I didn't get that far yet!<ref>{{cite web |url=http://www.flightgear.org/forums |title=incremental initialization of SGSubsystems|author=James Turner |date=Oct 01, 2012}}</ref>|James Turner}}
+
For the test mode, I really want to start Nasal-the-langauge very early, without loading all the modules in Nasal/ immediately. (Or maybe load a 'safe' subset). That's going to take some thought and I didn't get that far yet!<ref>{{cite web |url=http://forum.flightgear.org |title=incremental initialization of SGSubsystems|author=James Turner |date=Oct 01, 2012}}</ref>|James Turner}}
  
 +
 +
{{cquote|I'm working on a prototype of this - with no X11 / VLC required at all. It will load scenery tiles (for ground-intersections) but no views or rendering. But don't be in a rush for something to appear in Git, it's just an experiment.<ref>{{cite web |url=https://code.google.com/p/flightgear-bugs/issues/detail?id=882 |title=Issues 882 (Improved CI / regression testing support by providing options to disable interactive features (sound/window))|author=James Turner |date=Sep 26, 2012}}</ref>|James Turner}}
 +
 +
{{FGCquote
 +
  |A "headless" mode is a long-standing feature request, especially requested by developers and contributors who are hoping to do better/automated regression testing (valgrind, gdb, gprof). At the moment, it isn't directly possible - not without modifying the C++ source code and and rebuilding FG from source. You would basically have to change our OSG init code such that the window is not visible, note that you would still require OSG and X11 obviously, the window just wouldn't be shown - so that FG could be run from a console/shell
 +
  |{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=179978#p179978
 +
    |title=<nowiki>Re: run without osg</nowiki>
 +
    |author=<nowiki>Hooray</nowiki>
 +
    |date=<nowiki>Sun Mar 24</nowiki>
 +
  }}
 +
}}
 +
 +
== Proof of concept (patch) ==
 +
{{Note|This will not actually disable rendering, but merely hide the window by rendering to a pbuffer (FBO)}}
 +
The main camera is created in CameraGroup::buildCamera and a pbuffer is allocated when you set osg::GraphicContext::Traits.pbuffer to true. These traits are available in buildCamera because we set width and height there. For debugging purpose, you can add a line like
 +
<syntaxhighlight lang="cpp">window->gc->getTraits()->pbuffer=true;
 +
</syntaxhighlight>at the end of this function.<ref>{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=179978#p179978
 +
    |title=<nowiki>Re: run without osg</nowiki>
 +
    |author=<nowiki>FredB</nowiki>
 +
    |date=<nowiki>Sun Mar 24</nowiki>
 +
  }}
 +
</ref>
 +
 +
<syntaxhighlight lang="diff">
 +
diff --git a/src/Main/options.cxx b/src/Main/options.cxx
 +
index db5d84d..fb63dd1 100644
 +
--- a/src/Main/options.cxx
 +
+++ b/src/Main/options.cxx
 +
@@ -1797,6 +1797,7 @@ struct OptionDesc {
 +
    {"prop",                        true,  OPTION_FUNC | OPTION_MULTI,  "", false, "", fgOptSetProperty},
 +
    {"load-tape",                    true,  OPTION_FUNC,  "", false, "", fgOptLoadTape },
 +
    {"developer",                    true,  OPTION_IGNORE | OPTION_BOOL, "", false, "", nullptr },
 +
+    {"enable-headless",              false, OPTION_BOOL, "/sim/startup/headless-mode", true, "",0 },
 +
    {0}
 +
};
 +
 +
diff --git a/src/Viewer/WindowBuilder.cxx b/src/Viewer/WindowBuilder.cxx
 +
index f4d3c33..a1ffb84 100644
 +
--- a/src/Viewer/WindowBuilder.cxx
 +
+++ b/src/Viewer/WindowBuilder.cxx
 +
@@ -149,6 +149,12 @@ void WindowBuilder::makeDefaultTraits(bool stencil)
 +
            SG_LOG(SG_VIEW,SG_DEBUG,"Using initial window size: " << w << " x " << h);
 +
        }
 +
    }
 +
+bool headless = fgGetBool("/sim/startup/headless-mode", false);
 +
+if (headless) {
 +
+ SG_LOG(SG_VIEW, SG_ALERT, "Headless view required: rendering window to pbuffer");
 +
+ traits->pbuffer = true;
 +
+} // enable headless
 +
+
 +
}
 +
   
 +
} // of namespace flightgear
 +
</syntaxhighlight>
 +
 +
 +
For reference, to disable the window. It's the last thing before the return statement in WindowBuilder::makeDefaultTraits(bool stencil):
 +
<syntaxhighlight lang="cpp">    char* fg_gui = getenv("FG_GUI");
 +
    if (fg_gui != NULL && strcmp(fg_gui, "disabled") == 0) {
 +
      printf ("danna_fg_gui: %s\n", fg_gui);
 +
      traits->pbuffer = true;
 +
    }
 +
</syntaxhighlight>
 +
 +
just set an environment variable in my shell called FG_GUI and then this little hack works.<ref>{{cite web
 +
  |url    =  https://forum.flightgear.org/viewtopic.php?p=291699#p291699
 +
  |title  =  <nowiki> Re: Headless mode with access to property tree </nowiki>
 +
  |author =  <nowiki> alexklibisz </nowiki>
 +
  |date  =  Aug 1st, 2016
 +
  |added  =  Aug 1st, 2016
 +
  |script_version = 0.40
 +
  }}</ref>
 +
 +
<syntaxhighlight lang="bash">
 +
export FG_GUI=disabled
 +
</syntaxhighlight>
 +
 +
== Status (07/2014) ==
 +
 +
{{Note|As of mid 2014, this is being worked on via [[Initializing Nasal early]], in part, this effort is also overlapping with the [[Reset & re-init]] and [[FlightGear Run Levels]] efforts.}}
 +
{{FGCquote
 +
|1= One could write automatic tests for FlightGear using the scripting and scenario support.
 +
|2= {{cite web
 +
  | url    = http://sourceforge.net/p/flightgear/mailman/message/15444457/
 +
  | title  = <nowiki>Re: [Flightgear-devel] development process</nowiki>
 +
  | author = <nowiki>Tim Moore</nowiki>
 +
  | date  = Jul 17th, 2007
 +
  | added  = Jul 17th, 2007
 +
  | script_version = 0.23
 +
  }}
 +
}}
 +
 +
{{FGCquote
 +
|1= I organised Simgear into 'core' and 'scene' libraries; the 'core' part is also what we call 'headless' simgear, i.e can be compiled without requiring OpenSceneGraph or any GUI systems. (Which is how SG is used by terragear and potentially some other users).
 +
|2= {{cite web
 +
  | url    = http://sourceforge.net/p/flightgear/mailman/message/29733462/
 +
  | title  = <nowiki>[Flightgear-devel] SimGear libraries</nowiki>
 +
  | author = <nowiki>James Turner</nowiki>
 +
  | date  = Aug 28th, 2012
 +
  }}
 +
}}
 +
 +
{{cquote|I'm writing some automated testing code for pieces of FG, so that I can experiment with changes to various internal bits of code with more confidence that I haven't broken anything. These aren't quite unit-tests (they test multiple areas of the code at once) and they're pretty crude, but any testing is better than none (and quicker than manual testing).<ref>{{cite web |url=http://www.mail-archive.com/flightgear-devel%40lists.sourceforge.net/msg17543.html |title=Startup position offsets (fg_init)|author=James Turner |date=Sun, 24 Aug 2008 13:53:58 -0700}}</ref>|James Turner}}
 +
 +
{{cquote|<nowiki>I only need the graphical output for the real aircraft and want to disable the unnecessary  graphical outputs (to reduce work load).</nowiki><ref>{{cite web |url=http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg40521.html|title=<nowiki>[Flightgear-devel] Disable grahpical output in flightgear / interface flightgear server instead of flightgear</nowiki>|author=<nowiki>Rahlf, Jonas (ext.)</nowiki>|date=<nowiki>Wed, 31 Jul 2013 04:42:48 -0700</nowiki>}}</ref>|<nowiki>Rahlf, Jonas (ext.)</nowiki>}}
 +
 +
{{cquote|I'm interested in the capability of doing multiple builds with different versions, branches and options and in doing some kind of automated testing on the resulting builds<ref>{{cite web |url=http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg40416.html|title=<nowiki>Re: [Flightgear-devel] FG 2.12 RC Broken ?</nowiki>|author=Pat|date=Wed, 03 Jul 2013 17:24:19 -0700}}</ref>|Pat}}
 +
 +
{{cquote|Make subsystems create-able and removable from commands. Only some subsystems are supported so far, since many have non-default constructors or other complexities.
 +
 +
With this, change, it's possible to dynamically add and remove the traffic-manager at runtime, for example:
 +
<syntaxhighlight lang="nasal">fgcommand("add-subsystem", props.Node.new({ "subsystem": "traffic-manager", "name":"traffic-manager", "do-bind-init":1}));</syntaxhighlight><ref>{{cite web |url={{flightgear url|commit=8608a480736651999c5ea31a489343ee097ee915}} |title=Initial work on dynamic subsystem creation.|author=James Turner |date=Oct 01, 2012}}</ref>|James Turner}}
 +
 +
{{cquote|could some thought be given to producing a benchmark suite for Flightgear. It would need to take in all of the, by now well known, variables - making it by no means a simple beast to manage.
 +
If this could be automated in some way it would be much easier to capture, and then submit, consistent data. <ref>{{cite web |url=http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg40294.html|title=<nowiki>[Flightgear-devel] Benchmark matrix</nowiki>|author=Alan Teeder|date=Thu, 20 Jun 2013 10:15:42 -0700}}</ref>|Alan Teeder}}
 +
 +
{{cquote|A scripted run would be an EXCELLENT tool.<ref>{{cite web |url=http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg40295.html|title=<nowiki>Re: [Flightgear-devel] Benchmark matrix</nowiki>|author=geneb|date=Thu, 20 Jun 2013 10:20:41 -0700}}</ref>|geneb}}
 +
 +
{{cquote|Is there a way to do that "headless", without any graphic output? I wasn't being clear - I wanted to run *FlightGear* in a "headless" mode,
 +
not the browser.<ref>{{cite web |url=http://sourceforge.net/p/flightgear/mailman/message/32259769/|title=<nowiki>Property documentation...</nowiki>|author=geneb|date=2014-04-23 14:22:57}}</ref>|geneb}}
 +
 +
{{cquote|I don't think you can run FlightGear without a window. There used to be some property "draw-otw" that - if set - stops fg from rendering anything to
 +
the screen, but it still needs an opengl screen.<ref>{{cite web |url=http://sourceforge.net/p/flightgear/mailman/message/32260799/|title=<nowiki>Property documentation...</nowiki>|author=TorstenD|date=2014-04-23 14:22:57}}</ref>|TorstenD}}
 +
 +
{{FGCquote
 +
|1= I regularly test a rather small subset of FG features, i.e. never use MP or the replay system, no multi-monitor setup, no flights beyond ~90 minutes, no Rembrandt, no Basic Weather, rarely life weather,... so what we need to worry about most are use cases not covered by us
 +
|2= {{cite web
 +
  | url    = http://sourceforge.net/p/flightgear/mailman/message/34632894/
 +
  | title  = <nowiki>Re: [Flightgear-devel] Some thoughts about the release process</nowiki>
 +
  | author = <nowiki>Thorsten Renk</nowiki>
 +
  | date  = Nov 19th, 2015
 +
  }}
 +
}}
 +
 +
Also see, [[FlightGear Benchmark]]
  
 
<references/>
 
<references/>
  
 
== Regression Testing in FlightGear ==
 
== Regression Testing in FlightGear ==
 +
{{FGCquote
 +
|1= the intel driver is more strict on these and throws errors when others are happily moving along, so that got me into thinking whether we could somehow wire the shader compiler into Jenkins to have these checked right away? Just like it's useful in catching compilation errors on windows and mac, that are less common architectures among the developers.
 +
|2= {{cite web
 +
  | url    = http://sourceforge.net/p/flightgear/mailman/message/34857657/
 +
  | title  = <nowiki>Re: [Flightgear-devel] To Release or not?</nowiki>
 +
  | author = <nowiki>Tuomas Kuosmanen</nowiki>
 +
  | date  = Feb 17th, 2016
 +
  | added  = Feb 17th, 2016
 +
  | script_version = 0.25
 +
  }}
 +
}}
 +
 
While introducing unit tests and regression tests into the FlightGear project has been repeatedly brought up by several long-term contributors [http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg12321.html] and core developers [http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg11825.html], it isn't yet in wide or regular use in FlightGear, even though it is generally understood to be a worthwhile addition to FlightGear in order to do automated testing of individual features, for example when preparing releases [http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg11832.html].
 
While introducing unit tests and regression tests into the FlightGear project has been repeatedly brought up by several long-term contributors [http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg12321.html] and core developers [http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg11825.html], it isn't yet in wide or regular use in FlightGear, even though it is generally understood to be a worthwhile addition to FlightGear in order to do automated testing of individual features, for example when preparing releases [http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg11832.html].
  
Line 41: Line 308:
 
These obstacles in debugging such highly specific issues are also highlighted by core developers to severely limit the troubleshooting process
 
These obstacles in debugging such highly specific issues are also highlighted by core developers to severely limit the troubleshooting process
 
[http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg21664.html].
 
[http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg21664.html].
 +
 +
For a more recent discussion of regression testing and benchmarking, see [[FlightGear Benchmark]].
  
 
== Background ==
 
== Background ==
Line 54: Line 323:
  
 
== Introducing Regression Tests to FlightGear ==
 
== Introducing Regression Tests to FlightGear ==
 +
{{See also|Testing}}
 +
 +
{{FGCquote
 +
|1= I am working on a comprehensive test suite for the flightgear codebase (simgear already has its own). I will present this later on, but only once I have fully functional Windows support. The design of this test suite will allow individual subsystems to be initialised in full isolation. It will allow for the standard subsystem methods (init, reinit, bind, unbind, update, etc.) to be called in any order in a highly repetitive manner. It is a little like Hooray's segfault paradise - the reinit.nas reset/re-init control panel [1] - in that it is great for fishing out a tonne of segfaults. I have used this for stabilising the FGPythonSys experiment [2]. This would really help with implementing the above design and making the OSG heavy subsystems fully independent.
 +
|2= {{cite web
 +
  | url    = http://sourceforge.net/p/flightgear/mailman/message/35048478/
 +
  | title  = <nowiki>[Flightgear-devel] Integration of the new subsystem design and the
 +
OSG scene graph management.</nowiki>
 +
  | author = <nowiki>Edward d'Auvergne</nowiki>
 +
  | date  = Apr 28th, 2016
 +
  | added  = Apr 28th, 2016
 +
  | script_version = 0.26
 +
  }}
 +
}}
 +
 +
{{FGCquote
 +
|1= Actually my thinking is along these lines - these are exactly the same types of changes I was thinking of for fully decoupling all the various parts of FlightGear for use in a CppUnit-based test suite ;)  Something for the future though.
 +
|2= {{cite web
 +
  | url    = http://forum.flightgear.org/viewtopic.php?p=277299#p277299
 +
  | title  = <nowiki>Re: FGPython an propose for Python as an nasal alternative</nowiki>
 +
  | author = <nowiki>bugman</nowiki>
 +
  | date  = Feb 23rd, 2016
 +
  | added  = Feb 23rd, 2016
 +
  | script_version = 0.25
 +
  }}
 +
}}
 +
 +
{{FGCquote
 +
|1= Personally I would be reluctant to add a framework dependency (CppUnit / Google Test) unless the benefits are large and thus far I wasn’t convinced. The nasty macro-based tests are my hack and could / should be improved. Boost is a bit of a debating point in the project, I’d suggest to avoid it for tests again unless there is some massive benefit. Reflecting on this have written the above, if there was a common framework which could replace my hacks, but also avoid Boost, I’d personally be okay with that providing it’s an optional compile time requirement. Whatever the solution, it should be runnable via CTest, and if it generates output that Jenkins can parse, that would be fantastic. Non-Ctest compatible solutions would be possible but mean some build infrastructure changes.
 +
|2= {{cite web
 +
  | url    = http://sourceforge.net/p/flightgear/mailman/message/33628094/
 +
  | title  = <nowiki>Re: [Flightgear-devel] SimGear/FlightGear Portability Patches</nowiki>
 +
  | author = <nowiki>James Turner</nowiki>
 +
  | date  = Mar 23rd, 2015
 +
  | added  = Mar 23rd, 2015
 +
  | script_version = 0.25
 +
  }}
 +
}}
 +
 +
{{FGCquote
 +
  |Unit-testing for the FlightGear classes would be fantastic, but in general the ‘FGGlobals’ structure makes it hard. To unit-test most areas will mean a standard test harness which starts up a skeletal subsystem manager, the NavCache (for some classes, e.g. radio testing), and somehow makes an FGGlobals which works (maybe a mock one) but doesn’t require linking in 80% of the codebase. Any maintainable, not-too-intrusive idea on doing that is welcome!
 +
  |{{cite web |url=http://sourceforge.net/p/flightgear/mailman/message/33628094/
 +
    |title=<nowiki>Re: [Flightgear-devel] SimGear/FlightGear Portability Patches</nowiki>
 +
    |author=<nowiki>James Turner</nowiki>
 +
    |date=<nowiki>2015-03-23</nowiki>
 +
  }}
 +
}}
 +
 +
{{FGCquote
 +
  |I've done some initial measurement to identify potential memory leaks in SimGear.
 +
Detection was limited to execution of code covered by unit tests.
 +
  |{{cite web |url=http://sourceforge.net/p/flightgear/mailman/message/33630731/
 +
    |title=<nowiki>[Flightgear-devel] SimGear Memory Leaks</nowiki>
 +
    |author=<nowiki>xDraconian</nowiki>
 +
    |date=<nowiki>2015-03-24</nowiki>
 +
  }}
 +
}}
  
 
The task of introducing regression tests isn't that easily achieved in FlightGear's case:  
 
The task of introducing regression tests isn't that easily achieved in FlightGear's case:  
Line 75: Line 401:
 
* debugging (running FlightGear non-interactively, without requiring user input)
 
* debugging (running FlightGear non-interactively, without requiring user input)
 
* unit tests (e.g. to facilitate refactoring efforts)
 
* unit tests (e.g. to facilitate refactoring efforts)
* automated release preparations (e.g. to test individual subsystems but also complete aircraft)
+
* automated release preparations (e.g. to test individual subsystems but also complete aircraft, or even flights/flight plans)
* benchmarking the whole system or individual subsystems
+
* benchmarking the whole system or individual subsystems {{Progressbar|60}}
 +
* feature-scaling by making some subsystems optional or better configurable
  
 
== Approach ==
 
== Approach ==
Line 91: Line 418:
  
 
The recent additions to the [[Autopilot]] and [[Route Manager]] systems in FlightGear also make it increasingly feasible to create completely scripted test flights for automatically doing certain portions of a flight without relying on user input.
 
The recent additions to the [[Autopilot]] and [[Route Manager]] systems in FlightGear also make it increasingly feasible to create completely scripted test flights for automatically doing certain portions of a flight without relying on user input.
 +
 +
{{FGCquote
 +
  | the most flexible approach would be using scripting - FligthGear has a built-in scripting language (called "Nasal"), that can be used to control aircraft, and even to instantiate multiple AI aircraft, one of the most straightforward examples is the "tanker.nas" script, which creates a fully scripted AI tanker - that could be easily extended to create dozens of tankers, and obviously you could also change the 3D model if you wanted to. In fact, we have a separate addon, named "bombable" that adds "AI bots" to the simulator, for dogfighting purposes - none of that required C++ changes, it's all done in scripting space. Another users implemented a fully scripted missile (fox2.nas) that tracks aircraft
 +
  |{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=189488#p189488
 +
    |title=<nowiki>Re: Suitability of this software to run a swarm simulation</nowiki>
 +
    |author=<nowiki>Hooray</nowiki>
 +
    |date=<nowiki>Wed Aug 28</nowiki>
 +
  }}
 +
}}
 +
{{FGCquote
 +
  |Curt did that a while back for the f14b, which did a fully automated carrier approach using just Nasal scripting:
 +
* http://diydrones.com/profiles/blogs/uas-search-and-rescue-demo-simulation
 +
* http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg33987.html
 +
* http://www.flightgear.org/forums/viewtopic.php?f{{=}}4&amp;t{{=}}13615
 +
{{#ev:youtube|cvbtSG9cy20}}
 +
  |{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=176169&sid=bc24534a101c0e4349ee42eb1e9476a3#p176169
 +
    |title=<nowiki>Re: Possibility to run a fully automatized mission ?</nowiki>
 +
    |author=<nowiki>Hooray</nowiki>
 +
    |date=<nowiki>Mon Feb 04</nowiki>
 +
  }}
 +
}}
  
 
== Milestones ==
 
== Milestones ==
* allow FlightGear to be run without creating a visible GUI window, i.e. in "headless" mode, using a --headless parameter {{Pending}}
+
* allow FlightGear to be run without creating a visible GUI window, i.e. in "headless" mode, using a --headless parameter {{Done}}
 
* allow FlightGear to be optionally compiled and run without any sound support/dependencies (OpenAL) {{Done}}
 
* allow FlightGear to be optionally compiled and run without any sound support/dependencies (OpenAL) {{Done}}
 
* allow replay buffers to be saved to a file in order to be replayed for automated test/demo flights, so that users can share saved replay buffers when reporting a bug {{Done}} (supported via the flight recorder subsystem)
 
* allow replay buffers to be saved to a file in order to be replayed for automated test/demo flights, so that users can share saved replay buffers when reporting a bug {{Done}} (supported via the flight recorder subsystem)
Line 102: Line 450:
 
== Related ==
 
== Related ==
 
* http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg02181.html
 
* http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg02181.html
* http://flightgear.org/forums/viewtopic.php?f=18&t=19502&p=179978&hilit=#p179968
+
* http://forum.flightgear.org/viewtopic.php?f=18&t=19502&p=179978&hilit=#p179968
 
* https://code.google.com/p/flightgear-bugs/issues/detail?id=882
 
* https://code.google.com/p/flightgear-bugs/issues/detail?id=882
 +
* {{flightgear commit|af6611d7f61a6d993ce94fa6efade0b4485eb7a5}}
 +
 +
== Devel List ==
 +
* http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg40521.html
 +
* http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg29510.html
 +
* http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg29907.html
 +
* http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg02028.html
 +
 +
== References ==
 +
{{Appendix}}
  
 
[[Category:Core development projects]]
 
[[Category:Core development projects]]

Latest revision as of 15:09, 10 August 2017

This article describes content/features that may not yet be available in the latest stable version of FlightGear (2018.3).
You may need to install some extra components, use the latest development (Git) version or even rebuild FlightGear from source, possibly from a custom topic branch using special build settings: .

This feature is scheduled for FlightGear 2018.x. 30}% completed

If you'd like to learn more about getting your own ideas into FlightGear, check out Implementing new features for FlightGear.

FlightGear Headless

1rightarrow.png See Supporting multiple renderers for the main article about this subject.

there is no document listing the minimum set of stable features. If someone wants to write one, they can do it, but I guess they won’t volunteer to test release candidates against it; indeed getting people to even run them has been a struggle.[1]

with the new test framework it should be possible to add tests for carrier start, as I already did to ensure runway selection with METAR. Indeed the position init code is a very good candidate for testing this way since we broken different aspects (parking position startup, WXR runway selection, the carrier, probably some more things[2]


The original FG scene graph design is based on rigid assumptions of the ordering of the sequence of events for start up and shut down. Branch management is highly inconsistent, and is scattered all over the flightgear and simgear codebases. But it worked perfectly due to the fixed event sequences. The new subsystem design however brings in a much more flexible design, whereby systems can be dynamically turned on, turned off, restarted, initialised, reinitialised, etc. This is essential for enabling rebooting, and allows for custom FlightGear usage of only a small subset of systems and for a headless mode. The OSG+subsystem breakage occurred once the scene manager subsystem was fully converted to the new design. The OSG scene graph root is created at the start of FGScenery::init(). With the subsystem design, the scene graph root may not be present when other flightgear/simgear code paths are called to create, manage, or destroy the various scene graph branches.[3]

Another option is to make FGRenderer the optional piece - James Turner started doing some work in this direction in December 2015. I.e without FGRenderer there is no rendering at all, and hence no scene-graph.[4]

we should really do with with subclassing, making an SGDisplaySubsystem with:

  • virtual void initVisuals();
  • virtual void updateVisuals();
  • virtual void shutdownVisuals();

With the expectation that the manager layer (FGScenery or FGRenderer) calls initVisuals / updateVisuals / shutdownVisuals if it exists. This also forces the implementation of each subsystem to distinguish simulation pieces (modifies properties) from display pieces (reads but hopefully does not modify properties, modifies OSG/rendering state, i.e. visuals) [5]

  1. James Turner  (Aug 10th, 2017).  Re: [Flightgear-devel] carrier features: is there any modification , your policy related ? .
  2. James Turner  (Aug 10th, 2017).  Re: [Flightgear-devel] carrier features: is there any modification , your policy related ? .
  3. Edward d'Auvergne (Apr 28th, 2016). [Flightgear-devel] Integration of the new subsystem design and the OSG scene graph management..
  4. James Turner (May 4th, 2016). Re: [Flightgear-devel] Integration of the new subsystem design and the OSG scene graph management..
  5. James Turner (May 4th, 2016). Re: [Flightgear-devel] Integration of the new subsystem design and the OSG scene graph management..

Motivation

Cquote1.png these are exactly the same types of changes I was thinking of for fully decoupling all the various parts of FlightGear for use in a CppUnit-based test suite
Cquote2.png
Cquote1.png what remains true is the that overwhelming majority of the FlightGear main loop time is spent rendering the graphics.
Cquote2.png
Cquote1.png That's not to say there aren't some bottlenecks in our rendering pipeline that need to eventually be dealt with, but these are more tied to our use of properties and nasal in graphics system call backs (much of this we inherited from the original design which was build around plib's SG (scene graph) library.)
Cquote2.png
Screen shot showing a the performance monitor in a patched version of FlightGear 3.2 where subsystem initialization is made better configurable and increasingly optional by allowing subsystems to be explicitly disabled/enabled during startup. Decoupling internal subsystem dependencies means that we can more easily provide support for benchmarking, but also headless regression testing - and eventually, also a standalone FGCanvas startup mode.
Note  Also see Testing
Cquote1.png If we want to remove an subsystem from the flight gear source code, without leaving any trace of it.. What are the essential steps ?(for example, if the sound update has to be removed, what are steps to be followed).
Cquote2.png
Cquote1.png the route manager has a fairly nasty bug that results in memory consumption increasing throughout a flight. The rate at which it increases appears to increase with the number of waypoints, with a flight of 20 waypoints experiencing an increase of over 2GB per hour in my tests. The longer the flight, the more waypoints it tends to have and the more unsustainable the growth becomes.
— Richard Senior (2015-02-25 22:00:53). Memory leak most likely from route manager in 3.4.0 and next.
(powered by Instant-Cquotes)
Cquote2.png
Cquote1.png I am wishing I could easily add tests for the route-path / GPS code, but anything that needs ‘globals’ is hard to test in isolation.
Cquote2.png
Cquote1.png I appreciate that would require us to have a unit-testing infrastructure, and way to bring up a subset of globals and the nav-cache, in a test environment. I have made a few abortive attempts at exactly that but the sprawling nature of FGGlobals keeps defeating me.


(The ability to run many tests efficiently was actually one of the smaller reasons for introducing the nav-cache, since without it any automated testing system would be horribly slow)


Cquote2.png


Cquote1.png The primary goals are:
  • make startup more predictable and less hard-coded.
  • allow running flightgear in a server/test mode with only some subsystems, and no rendering

Obviously supporting a standalone 'fgcanvas' would be quite a small extension from those. I'm not worrying about dynamic dependencies or automatic subsystem creation for the moment - I expect the user / defaults to have defined a set of subsystems that work without crashing You're correct of course that Nasal has many assumptions about subsystems, but I think that can be improved incrementally on the Nasal side.

For the test mode, I really want to start Nasal-the-langauge very early, without loading all the modules in Nasal/ immediately. (Or maybe load a 'safe' subset). That's going to take some thought and I didn't get that far yet![1]
— James Turner
Cquote2.png


Cquote1.png I'm working on a prototype of this - with no X11 / VLC required at all. It will load scenery tiles (for ground-intersections) but no views or rendering. But don't be in a rush for something to appear in Git, it's just an experiment.[2]
— James Turner
Cquote2.png
Cquote1.png A "headless" mode is a long-standing feature request, especially requested by developers and contributors who are hoping to do better/automated regression testing (valgrind, gdb, gprof). At the moment, it isn't directly possible - not without modifying the C++ source code and and rebuilding FG from source. You would basically have to change our OSG init code such that the window is not visible, note that you would still require OSG and X11 obviously, the window just wouldn't be shown - so that FG could be run from a console/shell
— Hooray (Sun Mar 24). Re: run without osg.
(powered by Instant-Cquotes)
Cquote2.png

Proof of concept (patch)

Note  This will not actually disable rendering, but merely hide the window by rendering to a pbuffer (FBO)

The main camera is created in CameraGroup::buildCamera and a pbuffer is allocated when you set osg::GraphicContext::Traits.pbuffer to true. These traits are available in buildCamera because we set width and height there. For debugging purpose, you can add a line like

window->gc->getTraits()->pbuffer=true;
at the end of this function.[3]
diff --git a/src/Main/options.cxx b/src/Main/options.cxx
index db5d84d..fb63dd1 100644
--- a/src/Main/options.cxx
+++ b/src/Main/options.cxx
@@ -1797,6 +1797,7 @@ struct OptionDesc {
     {"prop",                         true,  OPTION_FUNC | OPTION_MULTI,   "", false, "", fgOptSetProperty},
     {"load-tape",                    true,  OPTION_FUNC,   "", false, "", fgOptLoadTape },
     {"developer",                    true,  OPTION_IGNORE | OPTION_BOOL, "", false, "", nullptr },
+    {"enable-headless",              false, OPTION_BOOL, "/sim/startup/headless-mode", true, "",0 },
     {0}
 };
 
diff --git a/src/Viewer/WindowBuilder.cxx b/src/Viewer/WindowBuilder.cxx
index f4d3c33..a1ffb84 100644
--- a/src/Viewer/WindowBuilder.cxx
+++ b/src/Viewer/WindowBuilder.cxx
@@ -149,6 +149,12 @@ void WindowBuilder::makeDefaultTraits(bool stencil)
             SG_LOG(SG_VIEW,SG_DEBUG,"Using initial window size: " << w << " x " << h);
         }
     }
+bool headless = fgGetBool("/sim/startup/headless-mode", false);
+if (headless) {
+ SG_LOG(SG_VIEW, SG_ALERT, "Headless view required: rendering window to pbuffer");
+ traits->pbuffer = true;
+} // enable headless
+
 }
     
 } // of namespace flightgear


For reference, to disable the window. It's the last thing before the return statement in WindowBuilder::makeDefaultTraits(bool stencil):

    char* fg_gui = getenv("FG_GUI");
    if (fg_gui != NULL && strcmp(fg_gui, "disabled") == 0) {
      printf ("danna_fg_gui: %s\n", fg_gui);
      traits->pbuffer = true;
    }

just set an environment variable in my shell called FG_GUI and then this little hack works.[4]

export FG_GUI=disabled

Status (07/2014)

Note  As of mid 2014, this is being worked on via Initializing Nasal early, in part, this effort is also overlapping with the Reset & re-init and FlightGear Run Levels efforts.
Cquote1.png One could write automatic tests for FlightGear using the scripting and scenario support.
— Tim Moore (Jul 17th, 2007). Re: [Flightgear-devel] development process.
(powered by Instant-Cquotes)
Cquote2.png
Cquote1.png I organised Simgear into 'core' and 'scene' libraries; the 'core' part is also what we call 'headless' simgear, i.e can be compiled without requiring OpenSceneGraph or any GUI systems. (Which is how SG is used by terragear and potentially some other users).
— James Turner (Aug 28th, 2012). [Flightgear-devel] SimGear libraries.
(powered by Instant-Cquotes)
Cquote2.png
Cquote1.png I'm writing some automated testing code for pieces of FG, so that I can experiment with changes to various internal bits of code with more confidence that I haven't broken anything. These aren't quite unit-tests (they test multiple areas of the code at once) and they're pretty crude, but any testing is better than none (and quicker than manual testing).[5]
— James Turner
Cquote2.png
Cquote1.png I only need the graphical output for the real aircraft and want to disable the unnecessary graphical outputs (to reduce work load).[6]
— Rahlf, Jonas (ext.)
Cquote2.png
Cquote1.png I'm interested in the capability of doing multiple builds with different versions, branches and options and in doing some kind of automated testing on the resulting builds[7]
— Pat
Cquote2.png
Cquote1.png Make subsystems create-able and removable from commands. Only some subsystems are supported so far, since many have non-default constructors or other complexities.

With this, change, it's possible to dynamically add and remove the traffic-manager at runtime, for example:

fgcommand("add-subsystem", props.Node.new({ "subsystem": "traffic-manager", "name":"traffic-manager", "do-bind-init":1}));
[8]
— James Turner
Cquote2.png
Cquote1.png could some thought be given to producing a benchmark suite for Flightgear. It would need to take in all of the, by now well known, variables - making it by no means a simple beast to manage. If this could be automated in some way it would be much easier to capture, and then submit, consistent data. [9]
— Alan Teeder
Cquote2.png
Cquote1.png A scripted run would be an EXCELLENT tool.[10]
— geneb
Cquote2.png
Cquote1.png Is there a way to do that "headless", without any graphic output? I wasn't being clear - I wanted to run *FlightGear* in a "headless" mode, not the browser.[11]
— geneb
Cquote2.png
Cquote1.png I don't think you can run FlightGear without a window. There used to be some property "draw-otw" that - if set - stops fg from rendering anything to the screen, but it still needs an opengl screen.[12]
— TorstenD
Cquote2.png
Cquote1.png I regularly test a rather small subset of FG features, i.e. never use MP or the replay system, no multi-monitor setup, no flights beyond ~90 minutes, no Rembrandt, no Basic Weather, rarely life weather,... so what we need to worry about most are use cases not covered by us
— Thorsten Renk (Nov 19th, 2015). Re: [Flightgear-devel] Some thoughts about the release process.
(powered by Instant-Cquotes)
Cquote2.png

Also see, FlightGear Benchmark

  1. James Turner (Oct 01, 2012). incremental initialization of SGSubsystems.
  2. James Turner (Sep 26, 2012). Issues 882 (Improved CI / regression testing support by providing options to disable interactive features (sound/window)).
  3. FredB (Sun Mar 24). Re: run without osg.
  4. alexklibisz  (Aug 1st, 2016).  Re: Headless mode with access to property tree .
  5. James Turner (Sun, 24 Aug 2008 13:53:58 -0700). Startup position offsets (fg_init).
  6. Rahlf, Jonas (ext.) (Wed, 31 Jul 2013 04:42:48 -0700). [Flightgear-devel] Disable grahpical output in flightgear / interface flightgear server instead of flightgear.
  7. Pat (Wed, 03 Jul 2013 17:24:19 -0700). Re: [Flightgear-devel] FG 2.12 RC Broken ?.
  8. James Turner (Oct 01, 2012). Initial work on dynamic subsystem creation..
  9. Alan Teeder (Thu, 20 Jun 2013 10:15:42 -0700). [Flightgear-devel] Benchmark matrix.
  10. geneb (Thu, 20 Jun 2013 10:20:41 -0700). Re: [Flightgear-devel] Benchmark matrix.
  11. geneb (2014-04-23 14:22:57). Property documentation....
  12. TorstenD (2014-04-23 14:22:57). Property documentation....

Regression Testing in FlightGear

Cquote1.png the intel driver is more strict on these and throws errors when others are happily moving along, so that got me into thinking whether we could somehow wire the shader compiler into Jenkins to have these checked right away? Just like it's useful in catching compilation errors on windows and mac, that are less common architectures among the developers.
— Tuomas Kuosmanen (Feb 17th, 2016). Re: [Flightgear-devel] To Release or not?.
(powered by Instant-Cquotes)
Cquote2.png

While introducing unit tests and regression tests into the FlightGear project has been repeatedly brought up by several long-term contributors [1] and core developers [2], it isn't yet in wide or regular use in FlightGear, even though it is generally understood to be a worthwhile addition to FlightGear in order to do automated testing of individual features, for example when preparing releases [3].

And while there are indeed some minor build tests provided by both, the SimGear and FlightGear projects, such test cases aren't really commonly provided our updated by developers when introducing modified or new code. Also, these are just low level tests for specific APIs - and do not lend themselves to be used for testing high level features.

Increasingly, FlightGear users are facing issues that are highly specific to their usage of FlightGear so that it isn't directly or easily possible to reproduce certain issues without exactly reproducing possibly an entire flight including identical startup and runtime settings, a fact that is also frequently acknowledged by FlightGear core developers [4].

This is however not only a tedious and long-winded process, but also a process that may require certain usage patterns and background information or a specific set of skills (such as for example landing a specific aircraft on an aircraft carrier).

In fact, the corresponding bug reports are often fairly long winded and complicated in that they try to provide all information necessary in order to allow developers to redo a certain flight segment that resulted in an error (see for example [5] or [6], [7]).

These obstacles in debugging such highly specific issues are also highlighted by core developers to severely limit the troubleshooting process [8].

For a more recent discussion of regression testing and benchmarking, see FlightGear Benchmark.

Background

FlightGear in its current form is an application that was primarily designed as an interactive graphical simulator, in other words, it is meant to be used by a user sitting in front of one or multiple screens, controlled by means such as a keyboard, mouse and other optional hardware such as joysticks/yokes and possibly also rudder/yaw pedals.

While confining FlightGear's design and use cases to this standard use scenario was of course very valid and feasible (as this is definitely the primary use) this restriction isn't necessarily ideal or even appropriate for the project to eventually be able to leverage itself for increasingly important purposes such as automated unit testing or automated benchmarking of individual FlightGear components in order to do regression testing.

This is an approach that is already used by the jsbsim project to some extent [9].

This RFC is meant to discuss the possible merits and approaches of allowing FlightGear to be used non-interactively, i.e. in an automated fashion such as for example by invoking it via shell scripts, so that FlightGear doesn't necessarily have to rely on user input or even a graphical output window in order to do a certain, well-defined and limited job, such as for example running certain subsystems for benchmarking purposes or by running scripted flights to fly standard patterns in order to generally help test aircraft that are considered for inclusion in upcoming FlightGear releases.

While there are certainly various thinkable scenarios for employing such facilities in other interesting contexts, this RFC will merely focus on the benefits for FlightGear itself.

Introducing Regression Tests to FlightGear

Cquote1.png I am working on a comprehensive test suite for the flightgear codebase (simgear already has its own). I will present this later on, but only once I have fully functional Windows support. The design of this test suite will allow individual subsystems to be initialised in full isolation. It will allow for the standard subsystem methods (init, reinit, bind, unbind, update, etc.) to be called in any order in a highly repetitive manner. It is a little like Hooray's segfault paradise - the reinit.nas reset/re-init control panel [1] - in that it is great for fishing out a tonne of segfaults. I have used this for stabilising the FGPythonSys experiment [2]. This would really help with implementing the above design and making the OSG heavy subsystems fully independent.
Cquote2.png
Cquote1.png Actually my thinking is along these lines - these are exactly the same types of changes I was thinking of for fully decoupling all the various parts of FlightGear for use in a CppUnit-based test suite ;) Something for the future though.
Cquote2.png
Cquote1.png Personally I would be reluctant to add a framework dependency (CppUnit / Google Test) unless the benefits are large and thus far I wasn’t convinced. The nasty macro-based tests are my hack and could / should be improved. Boost is a bit of a debating point in the project, I’d suggest to avoid it for tests again unless there is some massive benefit. Reflecting on this have written the above, if there was a common framework which could replace my hacks, but also avoid Boost, I’d personally be okay with that providing it’s an optional compile time requirement. Whatever the solution, it should be runnable via CTest, and if it generates output that Jenkins can parse, that would be fantastic. Non-Ctest compatible solutions would be possible but mean some build infrastructure changes.
— James Turner (Mar 23rd, 2015). Re: [Flightgear-devel] SimGear/FlightGear Portability Patches.
(powered by Instant-Cquotes)
Cquote2.png
Cquote1.png Unit-testing for the FlightGear classes would be fantastic, but in general the ‘FGGlobals’ structure makes it hard. To unit-test most areas will mean a standard test harness which starts up a skeletal subsystem manager, the NavCache (for some classes, e.g. radio testing), and somehow makes an FGGlobals which works (maybe a mock one) but doesn’t require linking in 80% of the codebase. Any maintainable, not-too-intrusive idea on doing that is welcome!
Cquote2.png
Cquote1.png I've done some initial measurement to identify potential memory leaks in SimGear.

Detection was limited to execution of code covered by unit tests.


— xDraconian (2015-03-24). [Flightgear-devel] SimGear Memory Leaks.
(powered by Instant-Cquotes)
Cquote2.png

The task of introducing regression tests isn't that easily achieved in FlightGear's case:

FlightGear has largely become an independent system and platform, so while it would be fairly straightforward (but still very tedious) to introduce individual unit tests in order to validate the correct behavior of low level C++ components, such as the SimGear APIs, it wouldn't really be that easy to properly test the various abstract, high level features that are provided by FlightGear as a functionality provider and simulation framework/platform with all its various subsystems providing support for abstract features.

In fact, conventional regression tests would inevitably fail when it comes to supporting base package resources, simply because FlightGear is the sole target platform for these resources.

While base package resources do generally make use of well-understood and established technologies or standards (i.e. textures, XML, scripts, 3D models, text files etc), it is only the specific combination of these resources inside FlightGear, that define a real purpose and use.

So, doing proper regression testing for such high level features would be very difficult without writing lots of redundant test code, which would probably end up being a maintenance burden in the long term - probably resulting in a situation similar to the current one, where tests simply end up being neglected and ignored at some point.

Thus, this discussion of bringing regression testing to FlightGear favors an approach where FlightGear itself is used as the regression testing framework.

So this isn't about doing low-level unit testing for individual FlightGear C++ code, but much more abstractly do regression testing by making use of the FlightGear platform to test abstract FlightGear features by making use of FlightGear's native support for technologies such as XML, scripting and networking.

Goals

Leverage FlightGear as its own regression testing framework, for purposes such as for example:

  • debugging (running FlightGear non-interactively, without requiring user input)
  • unit tests (e.g. to facilitate refactoring efforts)
  • automated release preparations (e.g. to test individual subsystems but also complete aircraft, or even flights/flight plans)
  • benchmarking the whole system or individual subsystems 60}% completed
  • feature-scaling by making some subsystems optional or better configurable

Approach

Due to FlightGear's extensive support for flexible software interfaces (such as e.g. networking, scripting and XML), FlightGear can in many scenarios theoretically already be used for serving as its own test platform.

In fact, the major obstacle really limiting FlightGear to be used by automated/scripted tests is its reliance on having a graphical output window available and opened.

If FlightGear provided an option to be run in non-interactive/headless mode, so that it wouldn't necessarily create a visible output window but could just run silently in a shell environment, it could already be easily used by shell scripts to do simple things such as for example profiling the fgfs process while running a specific Nasal script non-interactively and automatically terminating afterwards.

It's worth pointing out that this is indeed already possible: Nasal scripts can terminate the simulator by invoking an fgcommand, so this really isn't that much off the table and would facilitate scenarios where Nasal scripts may run certain test suites and automatically report status back to the caller (shell script). So, this would be just one scenario for running fgfs non-interactively in order to profile the Nasal interpreter. Another possible use might be scripted flights to have aircraft fly standard patterns or instrument approaches, while using a network interface such as the telnet facility to monitor the state of the flight during all phases of the flight.

Also, FlightGear's reliance on user input via means such as the mouse/keyboard and other hardware peripherals doesn't really pose a real problem, because all of these inputs are already internally handled by a combination of XML and scripting, so that emulating arbitrary user input by making use of scripts or by automatically writing to the property tree via network sockets is fairly straightforward and could also be accomplished by running shell scripts, that may for example invoke "netcat" specifically for this purpose.

The recent additions to the Autopilot and Route Manager systems in FlightGear also make it increasingly feasible to create completely scripted test flights for automatically doing certain portions of a flight without relying on user input.

Cquote1.png the most flexible approach would be using scripting - FligthGear has a built-in scripting language (called "Nasal"), that can be used to control aircraft, and even to instantiate multiple AI aircraft, one of the most straightforward examples is the "tanker.nas" script, which creates a fully scripted AI tanker - that could be easily extended to create dozens of tankers, and obviously you could also change the 3D model if you wanted to. In fact, we have a separate addon, named "bombable" that adds "AI bots" to the simulator, for dogfighting purposes - none of that required C++ changes, it's all done in scripting space. Another users implemented a fully scripted missile (fox2.nas) that tracks aircraft
Cquote2.png
Cquote1.png Curt did that a while back for the f14b, which did a fully automated carrier approach using just Nasal scripting:

Cquote2.png

Milestones

  • allow FlightGear to be run without creating a visible GUI window, i.e. in "headless" mode, using a --headless parameter Done Done
  • allow FlightGear to be optionally compiled and run without any sound support/dependencies (OpenAL) Done Done
  • allow replay buffers to be saved to a file in order to be replayed for automated test/demo flights, so that users can share saved replay buffers when reporting a bug Done Done (supported via the flight recorder subsystem)
  • allow arbitrary user inputs to be simulated via property tree modifications (pretty much possible already) Done Done
  • allow individual subsystems to be enabled/disabled dynamically (via properties), so that profiling and debugging can be restricted to specific usage scenarios and components 30}% completed
  • extend the Nasal API in order to facilitate "remote controlling" the simulator largely using scripts

Related

Devel List

References

References