FlightGear run levels

Request for Comments:

FlightGear Run Levels SGModule - standalone SGSubsystems An Integrated AI Traffic System Autopilot Enhancements Backwards Compatibility Initiative CDI instrument Decoupling the AI Traffic System Multiplayer Enhancements Modularizing, parallelizing and distributing FlightGear FDM engine feature standardization FlightGear Headless FlightGear Sessions Keyboard function priority list Next Generation Scenery Property Tree Reorganization Recommended Property Tree Enhancements Recommended Project Policies Redesigning the Replay System Remote Properties Simplifying Aircraft Deployment

Update 10/2012

Screen shot showing a the performance monitor in a patched version of FlightGear 3.2 where subsystem initialization is made better configurable and increasingly optional by allowing subsystems to be explicitly disabled/enabled during startup. Decoupling internal subsystem dependencies means that we can more easily provide support for benchmarking, but also headless regression testing - and eventually, also a standalone FGCanvas startup mode.

As of 10/2012, this is being worked on with a focus on better regression testing support (also see FlightGear Headless):

f we want to remove an subsystem from the flight gear source code, without leaving any trace of it.. What are the essential steps ?(for example, if the sound update has to be removed, what are steps to be followed). — Divi (Aug 31st, 2015). Steps for Removing subsystems in flightgear source code. (powered by Instant-Cquotes)

I'm working on a prototype of this - with no X11 / VLC required at all. It will load scenery tiles (for ground-intersections) but no views or rendering. But don't be in a rush for something to appear in Git, it's just an experiment. — James Turner

Make subsystems create-able and removable from commands. Only some subsystems are supported so far, since many have non-default constructors or other complexities. With this, change, it's possible to dynamically add and remove the traffic-manager at runtime, for example: fgcommand("add-subsystem", props.Node.new({ "subsystem": "traffic-manager", "name":"traffic-manager", "do-bind-init":1})); — James Turner

The primary goals are: make startup more predictable and less hard-coded. allow running flightgear in a server/test mode with only some subsystems, and no rendering Obviously supporting a standalone 'fgcanvas' would be quite a small extension from those. I'm not worrying about dynamic dependencies or automatic subsystem creation for the moment - I expect the user / defaults to have defined a set of subsystems that work without crashing You're correct of course that Nasal has many assumptions about subsystems, but I think that can be improved incrementally on the Nasal side. For the test mode, I really want to start Nasal-the-langauge very early, without loading all the modules in Nasal/ immediately. (Or maybe load a 'safe' subset). That's going to take some thought and I didn't get that far yet! — James Turner

Also see Reset & re-init (01/2013).

In addition, this effort will help to further modularize the FlightGear system such that we can increasingly make use of OpenRTI/HLA:

here's a larger issue here, that 'soon' (likely during the summer) I want to start restructuring the codebase into multiple processes, so we can support different rendering architectures (and use multiple CPUs) better. That's Mathias Fröhlich's HLA/RTI work, and indeed he has done the recent work on extending fgviewer to test changes to the current terrain system. — James Turner

Problem

checking how difficult it would be to put all aircraft related subsystems (fdm, replay, history, controls etc) into a single SGSubsystemGroup named "main-aircraft" to easily make the whole shebang optional using a single --prop for "FGCanvas" use, but also to check if it's feasible to prepare things for later reuse by the AI traffic system (for AI traffic that uses actual FDMs, APs and RMs - but also so that things are affected by the environment) , and it's actually working - even though reset/re-init is obviously hard-coded currently, which I am breaking by shuffling around subsystems, but as long as each SGSubsystemGroup implements the full SGSubsystem interface (postinit, reinit, shutdown etc), this could help clean up fg_init.cxx quite considerably [1]

FlightGear has a number of subsystems which cannot currently be disabled, so that they are always running because their initialization is entirely hard-coded. However, many of these systems are not always required and their initialization should be optional,

On the one hand, running subsystems which are not required may eat up computing resources that could be better used for different purposes/subsystems (i.e. to improve the frame rate), on the other hand - it is becoming increasingly difficult to reset and re-initialize the simulator properly, due to the number of subsystems which cannot be easily restarted: Reset & re-init.

Also, it is currently next to impossible to use FlightGear for purposes other than the primary purpose, i.e. "simulation", because the FlightGear initialization code works such that it always assumes that there's a full simulator running, including ALL the subsystems like FDM, autopilot, sound, scenery etc:

FlightGear is an old code base, and lots of the old assumptions (like a single aircraft) need to be teased out of the code before progress can be made on new features. This kind of work isn't glamorous, and often requires more effort than the new development does. But it's not brain surgery either. The problem with some great new features is that they show up with code that is "ready" to integrate, but without the integration work done. So they languish in the CVS tree until everyone forgets about them. I can recall at least one occasion where a unused module got replaced by a simpler (and arguably less functional) one precicely because the original never got integrated very well and the replacement actually worked. The extreme programming cult manages to get this idea right (every religion has a kernal of truth, right?) with their insistence on constant refactoring and integration. Features are useless in isolation.

In other words, to reuse the FlightGear binary for other purposes, such as a standalone "viewer" (see FGViewer, or a FGPanel/FGCanvas mode or a standalone ATC client, requires some refactoring in order to make subsystem initialization optional and fully runtime-configurable.

Also, this work should align well with other currently ongoing plans, such as for example decoupling the viewer from the rest of the simulation loop:

I have now checked in a small tool in flightgear that can be used just like osgviewer. The huge difference is that it also works for btg and with some knowledge about the internals of the model loading process also for stg files. The tool is aimed at people working on scenery and willing to see how their scenery modifications will look like. The can do so now without starting flightgear. This is a basic version of that tool that might grow it there is a need. I hope to include more flightgear internal scenery loading stuff so that you can at some time also load aircraft models an see if the static parts animations/transforms/postprocessing steps are as expected without the need to start the whole simulation.

I understand the idea behind the 'fgviewer' tool as creating a distinct viewer component (yet still in the early stage of development) which, while still remaining compilant with the FlightGear environment, is trying to adopt as little dependencies from FlightGear as possible and therefore does not necessarily has to follow every rule of "how things are done in fgfs" in order to achieve its fine goal. Actually I'm convinced that carefully cutting some of the old ties (some call them "cruft"), for example by keeping the viewer part as independent from the FlightGear core as possible, might serve as a good platform for future development. It's obvious that FlightGear, as every visual simulation, has to depend on the viewer. But the opposite way of depending the viewer part heavily on core FlightGear components is certainly not going into the outlined direction.

I also think it's a good idea to factor out the dependencies that the visual part of flightgear has on the whole flightgear implementation. It's not right at the head of my queue, but I support the idea and will look for ways to move it along.

I can see several components in this area: The viewer. That was the original reason I started fgviewer. I want to have a pure viewer application that does not do any simulation.

Currently I for myself am more working on pulling out different components from the simulation. As a side effect I am doing that work on fgviewer which should do a standalone viewer component. I cannot recommend to use fgviewer for anything today because plenty of stuff is missing. But that was the place that I thought would be a good start to implement this kind of doing fog/atmospheric scattering.

Background

As part of the Canvas project, we also talked about providing a standalone FGCanvas startup mode, analogous to how FGPanel works, however as part of the main FlightGear code base, rather than duplicating the existing source code (for maintenance reasons). However, this turned out to be a little tricky due a number of hard coded assumptions:

Unfortunately, there are many hard coded assumptions in the code currently. The code OFTEN assumes that there's a an aircraft, FDM scenery, sound etc. Basically, all the stuff and subsystems that won't be needed by a standalone FGCanvas/FGPanel. And the next funny thing is that all the Nasal code in $FG_ROOT/Nasal is also making lots of these implicit design assumptions (such as geodinfo() calls (depending on scenery/tile mgr) etc). Basically, this means that there will be lots of refactoring involved for a standalone canvas client, not just C++ code, but also Nasal code.

According to a mailing list discussion in 2006, this is a long-standing issue:

What would simplify this greatly is if subsystem registration was totally separated out from (re-)initialisation, and if sub-systems had run-level or priority associated with them. All the subsystems could be registered via add_subsystem, and then during fgInitSubsystems, the runlevel would gradually advance to the final value. This also means the dependencies between subsystems can be expressed (higher numbers depend on lower numbers), and might make things like reset more simple (lower the run-level back to some determined value, and the bring it back up again). As an additional enhancement, subsystems could be notified of all run-level changes above their threshold. This would solve the Nasal issue; starting up the interpreter (the first part of FGNasalSys::init) can be done very early (and quickly), and the subsytem would then wait for a relatively high-valued 'init' call before running scripts (the part that needs all other properties to be defined). In the even longer run, we'd actually want to associate the Nasal scripts with run-levels (/etc/rc.d, anyone?), since the frontend GUI might want a few scripts loaded, while I assume most are only relevant when actually flying. Such a change also makes postinit() unnecessary, I think - since the effect can always be achieved by having init() watch for a higher run-level.

Objective

Come up with a design to make subsystem initialization more explicit, and provide dependency resolution (run levels or system groups), so that subsystem initialization can be better controlled and explicitly enabled/disabled on an individual basis during startup.

Also, we want to work out the details to start up Nasal as early as possible, so that it can be used for most of the currently hard-coded initialization code, which isn't directly performance-critical - i.e. because it just sets up property defaults and parses/processes command line arguments. [2]

Nasal dependency resolution can be implemented analogous to how the import() command is done in the Nasal repository [3], possibly with some listener glue code.

We probably want "run levels" (groups) to be defined in XML space, so that they can be easily modified and extended as required:

 <runlevels>

  <runlevel n="0">
    <subsystem>
     <name>sound</name>
     <property-signal>/runlevels/signals/sound</property-signal>
    </subsystem>

    <subsystem>
     <name>autopilot</name>
     <property-signal>/runlevels/signals/autopilot</property-signal>
    </subsystem>

    <subsystem>
     <name>route-manager</name>
     <property-signal>/runlevels/signals/route-manager</property-signal>
    </subsystem>

  </runlevel>
 </runlevels>

In addition, we'll want to declare subsystem dependencies in XML space, too - using "groups" which are resursively started - so that the "aircraft" group starts the ROUTE MANAGER, AUTOPILOT, FDM, TERRAIN subsystems recursively.

If we could come up with a working design for this, it should also be possible to provide a fully re-initializable simulator experience, where aircraft can be selected and changed at runtime, without having to exit and restart the flight simulator (see FlightGear Sessions).

Ideas

My 'run-level' idea is almost the same, but rather than a flag, I'd have a /sim/runlevel property, and have the listeners watch for it changing. Eg, have the listener fire when run level was > 4 or 8, and stop when the run-level dropped lower again. Having separate signals for each subsystem seems like overkill, but what I am considering is naming the groups, and then having a /sim/signals/ or /sim/subsystems/ entry for each; of course we would need to introduce finer-grained groups but that's easy enough to do - the code already works with everything in just one flat group! As I say, I'm still trying to decide what approach is cleaner; the run-level concept is just one property, and nicely encapsulates the hierarchy of subsystems (if you're switching to level 8, all lower levels must already be initialized). But having names and separate flags makes the actual dependency clearer: 'this script depends on the aircraft subsystems' or 'this script depends on the environmental subsystems'. I guess it partly depends how complex the dependencies for any given script are in practice - hopefully they're actually quite simple.

Properties and Listeners to the Rescue

It is worth noting that all startup arguments implemented via properties (--prop:) can be also trivially supported at runtime using listeners or conventional polling, i.e. supporting runtime changes of these properties is comparatively simple and a well-understood task.

So looking at the code in fg_init.cxx, the major problem is indeed all the static, hard coded, initialization code which assumes a statically defined initialization order.

In other words, our problem is that we have routines which are sequentially called and which call each other. Routines that were never designed to be called individually, in a different order.

These hard coded init assumptions are exactly what makes runtime re-initialization so difficult and non-deterministic currently (see FlightGear Sessions for more info). Once we start using properties and listeners to implement all existing startup arguments, we would automatically have a fully runtime-configurable system, because callbacks would never be invoked directly at the C++ level, but instead be indirectly invoked by setting properties.

For example, at the moment we have code in $FG_SRC/src/Main/options.cxx#l1282 which maps the startup parameters to callbacks, which in turn call custom parsers to further process these arguments. What is really needed is a single SGPropertyChangeListener interface class which wraps the concept of a startup/runtime argument, so that all the argument-related code is wrapped inside a single helper class, instead of being spread over many different places in legacy procedural code.

Such a class would then register property listeners and it would transparently map its properties to startup arguments, so that the valueChanged() callback will be invoked by FlightGear, regardless of it being invoked during startup or at run time.

Furthermore, it is worth noting that we can significantly reduce the complexity of the options/init handling code by moving code to Nasal space and initializing the Nasal interpreter earlier. This applies especially to all the legacy C++ code setting up property defaults, i.e. see [4]. Most of this would be better dealt with in Nasal space.

Once core features and subsystems are initialized via properties, it would be possible to even implement complex run level setups in scripting space.

Also see: http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg03418.html

A Listener based Allocator/Deallocator for SGSubsystems

It would probably make sense to introduce a custom class template for "optional" subsystems, so that these can be easily made optional by registering listeners and invoking callbacks to new/delete SGSubsystems dynamically after recursively loading/releasing dependant subsystems by also firing signals. For example:

template <class T>
class SGOptional : public SGPropertyChangeListener {
public:
 SGOptional(const char* p) : _toggle_switch(p) {
  fgAddChangeListener(p, this);
  valueChanged(); // pick up initial property value (i.e. --prop args)
 }

void valueChanged() {
 // enable/disable system, call notification callbacks, set properties and allocate/free memory
 
}

void startup_dependencies() {
 // set properties to startup dependencies (if not initialized already)
}

// TODO: add one method to manage the refcount for dependencies (allocation/deallocation)

protected:
private:
 SGPropertyNode_ptr _toggle_switch;
 T _type;
};

Just by overloading a bunch of operators, most of the existing source code would not need to be changed. So that globals.hxx could be directly changed accordingly.

For example, instead of having this::

   // Global autopilot "route"
    FGRouteMgr *route_mgr;
    // ATC manager
    FGATISMgr *ATIS_mgr;

    // Navigational Aids
    FGNavList *navlist;
    FGNavList *loclist;
    FGNavList *gslist;
    FGNavList *dmelist;
    FGNavList *tacanlist;
    FGNavList *carrierlist;
    FGTACANList *channellist;

One could use the new SGOptional<T> template:

   // Global autopilot "route"
    SGOptional<FGRouteMgr*> route_mgr;
    // ATC manager
    SGOptional<FGATISMgr*> ATIS_mgr;

    // Navigational Aids
    SGOptional<FGNavList*> navlist;
    SGOptional<FGNavList*> loclist;
    SGOptional<FGNavList*> gslist;
    SGOptional<FGNavList*> dmelist;
    SGOptional<FGNavList*> tacanlist;
    SGOptional<FGNavList*> carrierlist;
    SGOptional<FGTACANList*> channellist;

This would make it possible to neatly encapsulate optional subsystems, which could also be linked to a property listener - so that features could be switched on/off at runtime.

The initialization code in fg_init.cxx would then need to be adapted to delegate subsystem initialization to corresponding SGOptional<T> specialization, which would make this dependant on a property switch:

    ////////////////////////////////////////////////////////////////////
    // Initialize the sound subsystem.
    ////////////////////////////////////////////////////////////////////
    // Sound manager uses an own subsystem group "SOUND" which is the last
    // to be updated in every loop.
    // Sound manager is updated last so it can use the CPU while the GPU
    // is processing the scenery (doubled the frame-rate for me) -EMH-
    globals->add_subsystem("sound", new SGOptional<SGSoundMgr>("/enable/sound"), SGSubsystemMgr::SOUND);

Also, many extension functions are specific to certain subsystems, i.e. property tree, sound, FDM, tile manager etc - these would also need to be registered and removed based on the "parent" subsystem's property signals.

Preventing raw pointer access to subsystems

Now, the next problem are obviously all "raw" pointer accesses, i.e. code in the form of:

  SGPropertyNode *sim = globals->get_props()->getNode("sim/gui", true);

or

 _route = static_cast<FGRouteMgr*>(globals->get_subsystem("route-manager"));

These are problematic because the requested subsystem may not yet be initialized.

So we need to make sure that "optional" subsystems are "clean" in that they are not directly used by any other subsystems. For starters, it would probably suffice to overload the -> operator for the Globals class and simply allocate the requested subsystems lazily by setting a property signal to call the SGPropertyChangeListener class that loads all the dependencies and then allocates the object.

fgcommands

it would be better to extend the fgcommand-basd re-initialiation mechanism that Zakalawe came up with ($FG_SRC/Main/subsystemFactor.?xx) to specifically support such initialization groups, analogous to "run-levels". That would then allow us to script initialization, without having any hard-coded stuff in fg_init.cxx that should be optional, and it would allow people to use the same mechanism to support benchmarking, feature-scaling or even regression testing. — Hooray (Jul 8th, 2014). Re: FGCanvas Experiments & Updates. (powered by Instant-Cquotes)

Nasal Scripts

Dealing with subsystem-specific Nasal APIs -like geodinfo()- is a nice challenge actually - for now, we can probably ignore it. But our way of keeping our scripting APIs separate from the actual subsystems that provide said functionality is making things pretty interesting - otherwise, each subsystem could simply use its own postinit() method to expose/release Nasal APIs, but the way FGNasalSys::init() is structured, there's the assumption that Nasal will always see a "full" session with all systems up and running, and expose the APIs accordingly. — Hooray (Jul 8th, 2014). Re: FGCanvas Experiments & Updates. (powered by Instant-Cquotes)

It occurred to me that the same "dependency" problem we have already WRT properties that depend on other subsystems. For that, core developers extended SGSubsystem to provide bind() and unbind() methods, where property-tying would normally take place: http://docs.freeflightsim.org/simgear/classSGSubsystem.html#a99b0666964ba74394784f79145a9565f — Hooray (Jul 9th, 2014). Re: FGCanvas Experiments & Updates. (powered by Instant-Cquotes)

on the FG/C++ side, the main systems that should/could probably become optional can be categoried into groups like these: networking (httpd, telnet, generic, multiplay etc) aircraft (fdm, control, autopilot, route manager, replay, history) world (scenery, tile manager, environment, lighting, ai) audio (sound, fgcom, voice) visuals (gui, canvas, canvasgui) — Hooray (Jul 8th, 2014). Re: FGCanvas Experiments & Updates. (powered by Instant-Cquotes)

It's also worth mentioning that base package files (XML, Nasal etc) may be trying to use disabled subsystems via some extension functions, so this needs further attention by only loading such Nasal modules via listeners (using the existing "Sub module" approach),so that they can be cleanly shut down when a subsystem is terminated, and replaced with a "dummy" routine which prints an error to the console, indicating that the requested subsystem isn't available.

Nasal scripts would need to register listeners so that they are aware of running/disabled subsystems that they depend on, and deal with any status change properly. Also, we would probably want to differentiate between Nasal scripts loaded as part of an aircraft, and "system-wide" scripts - to ensure that resetting the sim, and possibly changing the aircraft can be supported by suspending and removing all aircraft-specific Nasal scripts automatically.