Modularizing, parallelizing and distributing FlightGear

From wiki.flightgear.org

Jump to: navigation, search

Contents

Synopsis

This page is dedicated to starting discussion of the potential merits of, and approaches to, favoring a process-based strategy to help modularize, parallelize and possibly also distribute certain very specific FlightGear components (that need to lend themselves to this sort of modularization: 

"One of the big problems I had with FG was its pseudo asynchronous operation, which still meant 
that the rates at which you could run things like the FDM, autopilot and Nasal were effectively  
limited by the frame rate and which could lead to an aircraft being stable on one system but 
unstable on another.  I would really like to see these subsystems running on their own cores, or 
even more preferably, on their own networked hardware, so that I could run them at much higher and
guaranteed rates."[1]
  • "One issue you are probably seeing is that even though the FlightGear flight dynamics engines are setup to run at a guaranteed fixed rate of 120hz already, the autopilot update rate floats with the graphics update rate. Ideally the autopilot would update at the same rate as the flight dynamics.This was the case at one point in the project, but somehow that got lost during some portion of some restructure project."[2]
  • "Some simulation require the stability of their update frequency. With these, you can't have a process that interrupts and ocassionaly lengthens the length of one iteration."[3]
  • "One thing to remember when saying that the graphics accounts for 90-95% of the [cpu] utilisation is that it doesn't necessarily follow that all the other subsystems get the resources that they require. For example, I'd like to be able to run the autopilot pid controllers at much higher and more consistent rates than is possible now and even though the FDM rates are supposed to be guaranteed, in practice, it doesn't always seem to be the case." [4]
  • "IMHO the one important threading benefit is if we could get all of the rendering off the main simulation loop, meaning that the model runs independent of the presentation. (Ok, expensive environment eye candy like the traffic manager or wild fire CA would also be beneficial to move into threads - but they don't have tight synchronization needs wrt the main aircraft.)" [5]
  • "I think that being able to run the FDM, autopilot and Nasal at several hundred to a thousand+ Hz, instead of just around 120 Hz would be quite a big improvement. The fact that the autopilot subsystem got slightly borked and hasn't really been fixed since shows to me that FG has outstanding quality issues."[6]
  • "I don't think we need to achieve strict real time processing here, but we could achieve both higher rates and proportionally smaller variations in those rates using the existing timing techniques in FG."[7] 
"With much of our increasing processing power coming from multiprocessing, it seems to 
be a good idea to make FlightGear fully multithreading-capable.[...]
There are many ways we could design FlightGear with multiprocessing support." [8]
  • "If the people who actually make CPUs think that parallelism is the way to go, I wouldn't hold out too much hope on significant speed increases in the future. Sure, things will get faster, but not at the same rate we've seen so far." [9]
  • "As the hardware continues to develop in this direction high-performance software has to follow it if it's not to be left behind. Although FG doesn't need to be mpp capable right now, in a few years it will and now is probably a good time to start thinking about it if a _good_ solution is to be found by the time that it's needed."[10]
  • "In the end though, I just don't really think there's any other alternative. The h/w world is betting on parallelism for the future and software has got to fit the h/w."[11]
  • "Note that we're going to have to start thinking about threadin designs soon anyway, if we want to take advantage of all the fancy multi-core/multi-thread CPUs coming down the pipe. Rendering obviously has to stay within a single thread, but how much non-rendering work can we push out to worker threads? Ideally, everything would be "handed" to the renderer thread each frame, with all the matrices pre-cooked and ready for OpenGL calls."[12]
  • "The best candidates for separate threads [or parallelization in general] are processes that have relatively little input or output but require a lot of computation."[13]
  • "Threading is relatively safe if each thread is forced to play in a sandbox. A subsystem running in a separate thread must *not* touch any other subsystem (including the property manager), and must send and receive all information through a single, tightly-controlled interface."[14]
  • "That implies that a subsystem in a separate thread should not even include other FlightGear header files, much less access FGGlobals, FGInterface, the property manager, or anything else directly."[15]
  • "Making the property manager threadsafe isn't the issue; sure it can be done. The problem is that *using* the property manager (or anything) from multiple threads leaves you open to race conditions that occur due to unsynchronized changes to the property values."[16]
  • "Now, you can fix this via explicit locking ("/controls/trigger-safety-lock-lock", heh), or implicit design (only one thread sets the /controls tree)."[17]
  • "I think it would be better for the main thread to read the information from the property manager and then pass it to an object that the other thread can access."[18]
  • "Really! We should go out of our way to find a workable non-threaded solution before we add new threads to the code."[19]
  • "Threading (is) best avoided in my opinion, pretty much for the reasons you give. Instead of thinking of FG as a single threaded application, it needs to be a collection of standalone programs that run collaboratively - go back to thinking in terms of the external FDM option."[20]
  • "I'm not really thinking in terms of 'threading' at all, which I think is a very limited and half-house sort of technique. But neither though do I think it needs to be thought of as a pure real time system. Rather, I'm thinking in terms of the external FDM mechanism already present in FG. Running the FDM on it's own hardware system doesn't need to be any more real time than the FDM running within FG on the same system but because it's not going to be limited by the frame rate it could safely be run much faster and with proportionately more consistency than with FG. If you're running it at say 100Hz within FG I would expect to be able to run it several times faster, if not tens of times faster if the system it was running on wasn't spending most of its time rendering. You'll still get a variation in the rate that the FDM runs at but I suspect that the variation would be about the same in absolute terms." [21]
  • "If necessary it would be possible for parts of the program to hold it's own property tree, which is inaccessible from the global property tree, by keeping track of it's own root-node." [22]
  • "Most stuff can be decoupled so that the computation can even be split between different processes, which may even run on different machines [distribution]. Take again visualization (OTW) for example: It's (dynamic) input is position and orientation of the aircraft(s), output is altitude above ground. These are rather minor amounts of data which can easily go across a network. Most commercial FTDs and FNPTs (including these of the company I used to work for) do it that way. It pays to identify and separate the involved concepts using well-defined interfaces. Doing so leads to clean, extensible and efficient design." [23]
  • "I think that splitting FG into several parts would actually help reduce bugs, not increase them. Bugs would be limited to their particular subsystems and couldn't manifest themselves in other parts of the system, as they can do with a single monolithic system. Each discrete subsystem can only pass data back and forth, not bugs."[24]
  • "Personally, I think that the idea of threading in the context of FlightGear is a *very* scary idea, especially from the standpoint of long term maintanence and keeping our code robust. I'd perhaps favor splitting our code out into separate applications that use networking or shared memory or pipes to communicate. You lose some of the context switching efficiency of threads, but I think the code becomes more robust and maintanable because changes have less of a chance to propagate elsewhere to unintended areas ... "[25]
  • "Personally, I think the most sane approach with the highest chance of succeeding will be to pick some particular submodule that really makes sense to run on another machine or in a separate process and figure out how to split that off."[26]
  • "One could always break FlightGear into sub-applications and let them communicate via UDP ports."[27]
  • "To my mind flightgear can be broken down into distinct plugin "modules". There is the FDM, the "external world" visualisation,the cockpit input and output (ie joystick,pedals,switches and displays) and possibly a motion system. These can be interconnected with some inter process communication scheme. All of the modules could be run on a SMP hardware (e.g dual dual-core cpus) or on separate computers. [distribution]"[28]
  • "[...]without an operating system that can migrate threads to other networked processors then I think a more flexible approach may be "self-contained" modules communicating by passing "properties" over TCP. The "remote" FDM is already a possibility and there is an example of a remote joystick but how easy would it be to break up the rest of flightgear?"[29]
  • "The architecture document you refer to has been around for a while, and while it offers an improved architecture for a flight simulator, it isn't at all clear how we can separate the existing monolithic core of FG into the different components." [30]
  • "it's finding the parallelism that's the hardest part but things like the FDM, autopilot and Nasal do seem like obvious candidates. Even if we can't precisely balance the load, we could still improve the performance of parts of FG."[31]
  • "I agree here, putting the FDM in it's own process would be a good idea." [32]
  • "I would love to see all the FG core stuff going that route - one network capable API that everything can work through including FG itself.

As far as I can see FG looks like it's already half way there." [33]

  • "I have a long, long, long term plan to improve multi-threading support, by enforcing subsystems to *only* communicate via the property tree [...] it would then become possible to run any 'clean' subsystem on a pool of worker threads (maybe just one, maybe more)."[34]
  • "I have been working on extending the property code to add an SGRemoteProperty class to access properties on a remote host instead of locally." [35]
  • "the bottom line is that FG is simply going to have to face up to this issue at some point. A few people here have been bringing this topic up for a few years and now that multicore processors are the norm it's clear that the issue isn't going to go away. Like it or not, and I mean no offence or criticism by this, the current FG architecture is now obsolete. While single core and single processor systems were the norm it was fine - the software design was well fitted to the systems on which it ran - but parallel processing has always been the way things were going to go. It has been inevitable ever since super-computer designers realised that the only way that ever increasing performance could be achieved was by parallelism and now it's well and truly on the desktop."[36]
  • "I don't think anyone really _likes_ the idea of the extra work involved - it's going to be difficult and hard work, but living in the past isn't going to work either."[37]
  • "Of course, it's not going to be easy, but denying it won't make the issue go away." [38]


So, this discussion focuses solely on implementing a process-based component-system where modules (subsystems) only need to modify simulator state by getting/setting PropertyTree variables, with the implicit design assumption that only one single thread accesses the core property tree. Such components (run in a separate process) may however internally use their own property tree instance in order to provide and access state to/from the fgfs core process. All required sandboxing (as mentioned above) will be implicitly handled by running components in separate processes.

Background & Motivation

Related wiki entries

Related discussions

This proposal is largely based on the following discussions:

Why not threading?

In this proposal, a process-based approach to parallelization is favored over threading, as threading would probably introduce a whole number of possible problems, namely:

  • threading can become incredibly tricky in any sort of non-trivial application
  • platform support, processes don't need additional dependencies
  • synchronization
  • locking of shared resources/data

While process-based parallelization is comparatively 'expensive', it is antiticpated that this should turn out to be an acceptable tradeoff, as process-based parallelization of applications encourages a truly modular, loosely coupled design where data and resources may only be "shared" by providing explicit means and mechanisms for doing so (rather than having access to all global variables within a process supporting multiple threads).

Furthermore, establishing building blocks to provide support for process-based parallelization in FlightGear would eventually not only help FlightGear becoming more modularized and "parallel", but would also provide a possible foundation for future efforts to help FlightGear components support networked distribution, as well.

Objective

The priorities in descending order are to develop and discuss a potential design draft that would enable specific FlightGear components to be:

  • easily modularized
  • parallelized and
  • eventually also distributed

By using a design where individual components are implemented in a sufficiently generic fashion to facilitate being run as standalone programs/processes(communicating with the FlightGear core using its already available network support), rather than being necessarily directly run within the fgfs core executable process space (be it within the main loop or as a separate worker thread, none of which would help FG modularization).

Requirements

Modules must be totally self-contained and their only means of IPC with the core process is the FlightGear PropertyTree itself, being accessed via network interfaces already provided by FlightGear. Such modules may only communicate data provided via the PropertyTree.

Assumptions & Simpflications

In order to simplify a possible design, there are several assumptions that are made about said standalone modules in the initial drafting phase (some of which might be lifted later on):

  • all "modules" (standalone processes) are single-threaded using a conventional main loop.
  • modules may only be connected once to a running instance of the FlightGear core
  • there is no support for traditional shared memory.
  • the only way for standalone FlightGear components/modules (processes) to communicate with the FlightGear PropertyTree is using FlightGear's networking capabilities (i.e. setting/getting properties via telnet).
  • all data exchanges between modules and the FlightGear core have to take place by using the FlightGear property tree, that is data types are determined by the primitives supported by the property tree.
  • standalone components are reduced to being merely consumers and/or providers of data that is (made) accessible in the FlightGear PropertyTree
  • that is, there is no direct means provided for standalone components to do RPC across process boundaries, if RPC functionality is desired it needs to be explicitly implemented using existing means to trigger actions in the (core or module) process by writing to a SGPropertyListener-bound property node, which in turn executes a corresponding callback function.
  • all modules are supposed to route their communications (get/set requests) through the FlightGear core, there is no concept of supporting direct (p2p)inter-module communications that would bypass the FlightGear PropertyTree.
  • performance considerations are not important for the moment, the point is just to modularize and parallelize certain FlightGear components by splitting them up, so that they can be run as separate processes


While it is clear that these are serious restrictions and that several types of subsystems could not be implemented with these limitations, it is also much more feasible to actually start designing a corresponding infrastructure.

Expected Challenges

Allowing a possibly large/increasing number of different processes to query/modify state variables/properties within the FlightGear Property Tree asks for clean organization and structuring to avoid potential problems that may, among others for example result from processes 'tie'ing properties that would cause conflicts by resulting in circular callback invocation in different processes, or processes writing to properties that they should normally not have access to, or even multiple processes trying to write to the same properties, basically invalidating previous state thereby.

Workarounds

However, it is expected that enriching the current concept of properties with a couple of attributive meta information to enable enforcement of legitimate/valid accesses, would help declare a safer framework, namely by providing support for the following features:

  • introduce the concept of property owners (i.e. on subsystem/component basis)
  • introduce the concept of exclusive property ownership
  • introduce the concept of property permissions (i.e. read/write/delete access)
  • introduce the concept of property units (to optionally ensure proper data is being written to a node)
  • introduce the concept of property ranges (to specify valid value ranges for a property)
  • introduce the concept of property requirements/conditions (i.e. MUST_EXIST, MUST_NOT_EXIST, MUST_NOT_BE_TIED, etc)


Having such attributes in place that reflect this sort of meta information, would help formalizing and enforcing access specifications for certain properties, to ensure encapsulated (i.e. private/protected) use of properties where subsystem/component implementations can make the assumption that such internal state is immutable from the outside, even though it may be publicly available in the core's PropertyTree. Also see Recommended Property Tree Enhancements.

Further Assumptions

  • for each module-specific property there may be only one single function tied to it with (exclusive) writing priviledges, all other listeners are reduced to being read-only, this is to avoid modules without ownership executing uncontrolled writes.

Execution schemes

A process-based module/component system, while being comparatively straight-forward to design (at least compared to threading), has its own challenges due to the fact that the component or module would not be initialized in a hardcoded fashion, but rather it would support being run using different schemes or scenarios, each of which has its own advantages and challenges:

  • the core executable may fork -after startup-a separate, standalone executable (on the same machine) to create the module's process: this is straightforward to implement, does however not easily work for possible scenarios where distribution among different machines/architectures may play a role (at least not without having to resort to remotely forking via SSH)-also, when the main process terminates, all forked processes are also terminated-however, key init state can be easily provided to the forked child process by using command line parameters)
  • the core executable may -after startup- fork itself (again, on the same machine) using special parameters, to fork the module's process (pretty straightforward to implement, shortcomings are image duplication of the executable in memory, as well as potential modules having the same library dependencies as the main executable-also, when the main process terminates, all forked processes are also terminated, changing/adding (recompiling) module code at runtime cannot be easily accomplished as this would involve overwriting the core executable as well-however, key init state can be easily provided to the forked child process by using command line parameters)
  • the module's process may be started manually after core startup, so that it has to register with the core executable's process (this requires a certain startup order and manual interaction, as the module would then need to contact the core (via network) in order to register itself, in this scenario the core would be a server, which would facilitate modules connecting/disconnecting at runtime, so that for example a recompiled module could simply be tested without having to shutdown/restart the core process, however key init state needs to be passed either manually or retrieved from a standard location within the network)
  • the core executable may want to register itself with an already running module (in this scenario, where the core would act as a client, the module itself would need to act as a server-while this may sound overly complex and involved, it does have some bearing given that such a possibility would enable designs where simulator components may keep running, while the core could be shutdown/restarted, or even recompiled while several simulator components could remain running and simply wait for a reconnect-however key init state needs to be passed either manually or retrieved from a standard location within the network)


While the latter two usage scenarios are more appealing from a distribution-centric point of view, the other two scenarios are easier to implement, as they would not require any sort of network-based central registration service, where components would have to register themselves before being visible to other processes-basically similar to CORBA's POA/Name Service (which would also add the requirement to start the registration process first, OTOH this could also be automatically handled, too).

Providing RPC support

Emulated RPC support would be based on the current concept of registered SGPropertyListeners invoking callbacks that would in turn make sure to deal with any (optional) call arguments and return whatever state is required.

However, to be truly useful and genereic in a wider context, the current use of PropertyTree/RPC would need to be slightly extended and somewhat more formalized by means of providing standard ways to create and execute RPCs, as well as deal with any relevant return state.

Limitations using current FG RPC mechanisms

While the basic building blocks to support all sorts of RPC using the PropertyTree/network combination are already available in FlightGear, doing so is currently not yet very convenient, this is mainly because:

  • RPC callers have to manually "marshall" RPC parameters into a hierarchical SGPropertyNode format (as is i.e. also the case for arguments to fgcommands), so that the arguments are representable within the structure of the PropertyTree, using temporary PropertyTree nodes.
  • RPCs may have parameters of certain type (optional-ones, too) or not, there is currently no way to formally specify such requirements and ensure that this is validated at runtime
  • RPC implementors have to manually process/parse the arguments (SGPropertyNode*) passed to the RPC to look for any required parameters and deal with potential inconsistencies.

For any type of function call that uses the property tree as IPC mechanism, function attributes such as the function's signature or return state need to be manually prepared/processed for each call, possibly including redundant validation code as well.

This involves basically every time the following steps:

  • manually marshall arguments into propery tree format
  • temporarily save the created sub tree somewhere in the tree
  • call RPC function using a pointer to the node as parameter
  • manually decompose structure provided by pointer
  • validate types, obtain actual values
  • execute RPC code if types/values are valid
  • marshall return results into SGPropertyNode/PropertyTree format
  • post-process/evaluate results in callee

Module types

For starters, standalone modules can be classified to fall into the following groups:

  • permanently running
  • temporarily running/on demand

Furthermore, future modules can also be categorized based on whether they :

  • are allowed to be forked/connected multiple times to the core process (i.e. different FDM processes)
  • may only be forked/connected once to the core process

Module control

Based on the subsystem functionality provided by SGSubSystem, control over individual standalone components could be exercised using the following triggers (exposed within the FlightGear core PropertyTree):

  • start-module
  • stop-module
  • suspend-module
  • resume-module
  • update-module

These triggers could be implemented by exposing corresponding properties within the core's PropertyTree, ensuring that the component itself has exclusive ownership.

To enable the core, as well as other modules to query the actual state of a given module, providing additional state information is also likely to be useful:

  • current-state (starting,running,stopping,suspending,resuming,updating)
  • current-update-interval-hz

Module Registration

In order to enable modules to interact with the FlightGear core, they need to publish their services (functionality) within the core (again, using the PropertyTree), including information such as:

  • module name (i.e. used to set up a specific node path in the PropertyTree)
  • module description
  • module version


Module Workflow

Module Initialization

  • parse command line options
  • connect to FlightGear core via network (i.e. telnet address:port)
  • set up exclusive node directory, i.e. under /modules/$name, where $name would be the unique name of the corresponding module
  • publish meta information about module

Module Registration

  • set up aliased properties for reading/writing state variables of simulator

Module Updating

  • in the module's main loop poll all relevant simulator state
  • modify relevant state variables and publish in simulator

Module Suspension

  • stop polling all simulator state, except for module state

Module Termination

  • untie all tied properties
  • remove module from module namespace in property tree
  • terminate process
Retrieved from "http://wiki.flightgear.org/index.php/Modularizing,_parallelizing_and_distributing_FlightGear"