Modularizing, parallelizing and distributing FlightGear

From FlightGear wiki
Jump to navigation Jump to search
Caution  Developer information:

Some, and possibly most, of the details below are likely to be affected, or even deprecated, by the ongoing work on implementing and improving HLA support in FlightGear. For recent updates, please refer to HLA Timeline.
Please see: High-Level Architecture for further information
 To avoid conflicting efforts, you are advised not to start working on anything directly related to this without first coordinating your ideas with FlightGear core developers using the FlightGear developers mailing list. talk page..

Multicore

Configuration

Ongoing Efforts

Proposals & RFCs

Background

For developers

Synopsis

This page is dedicated to starting discussion of the potential merits of, and approaches to, favoring a process-based strategy to help modularize, parallelize and possibly also distribute certain very specific FlightGear components (that need to lend themselves to this sort of modularization):

Issues

Cquote1.png One of the big problems I had with FG was its pseudo asynchronous operation, which still meant that the rates at which you could run things like the FDM, autopilot and Nasal were effectively limited by the frame rate and which could lead to an aircraft being stable on one system but unstable on another. I would really like to see these subsystems running on their own cores, or even more preferably, on their own networked hardware, so that I could run them at much higher and guaranteed rates.
— leee (2009-08-04). Re: [Flightgear-devel] Multithreading support.
(powered by Instant-Cquotes)
 Cquote2.png
  • "One issue you are probably seeing is that even though the FlightGear flight dynamics engines are setup to run at a guaranteed fixed rate of 120hz already, the autopilot update rate floats with the graphics update rate. Ideally the autopilot would update at the same rate as the flight dynamics.This was the case at one point in the project, but somehow that got lost during some portion of some restructure project."[1]
  • "For example, while I was doing a couple of tests for what appears to be an A/P problem I noticed that the rate of the controllers were varying even though I'd specified a <Ts> period. This is certainly going to result in some unpredictable behaviour across different systems and I wouldn't be surprised if some of the other subsystems are equally limited in their rates." [2]
  • "The AP timing is a problem ... but that's more a side effect of too many cooks in the stew I think. At one point I had setup the AP to run along with the FDM loop so the two were tied together and ran at the same update rate with the same dt. That was changed (and I think was changed a long time ago) so the AP runs at graphics update rates ... as you point out, these are not necessarily consistant and will definitely change across different installations."[3]
  • "Some simulation require the stability of their update frequency. With these, you can't have a process that interrupts and ocassionaly lengthens the length of one iteration."[4]
  • "One thing to remember when saying that the graphics accounts for 90-95% of the [cpu] utilisation is that it doesn't necessarily follow that all the other subsystems get the resources that they require. For example, I'd like to be able to run the autopilot pid controllers at much higher and more consistent rates than is possible now and even though the FDM rates are supposed to be guaranteed, in practice, it doesn't always seem to be the case." [5]
  • "I have a Core2 Duo 2.66 ( E6600 ) and a 7600GT. I always saw the greatest fps increase after upgrading CPU and was disappointed by several GPU-only upgrade." [6]
  • "Although the FlightGear design fairly modular it's provided as a single binary.Everyone who wants to create a new I/O module must patch the FlightGear sources and compile the FlightGear binary from scratch. This may discourage those who want to use FlightGear as a tool and extend it in some way. Moreover, it's not always possible to include all functions in a single binary. Some functions may be mutually exclusive." [7]

Goals

  • "IMHO the one important threading benefit is if we could get all of the rendering off the main simulation loop, meaning that the model runs independent of the presentation. (Ok, expensive environment eye candy like the traffic manager or wild fire CA would also be beneficial to move into threads - but they don't have tight synchronization needs wrt the main aircraft.)" [8]
  • "I think that being able to run the FDM, autopilot and Nasal at several hundred to a thousand+ Hz, instead of just around 120 Hz would be quite a big improvement. The fact that the autopilot subsystem got slightly borked and hasn't really been fixed since shows to me that FG has outstanding quality issues."[9]
  • "I don't think we need to achieve strict real time processing here, but we could achieve both higher rates and proportionally smaller variations in those rates using the existing timing techniques in FG."[10]
  • "With a dependency graph between subsystems (which I want to add for other reasons any way) it would then become possible to run any 'clean' subsystem on a pool of worker threads (maybe just one, maybe more).[11]

Utilizing SMP Hardware

  • "With much of our increasing processing power coming from multiprocessing, it seems to be a good idea to make FlightGear fully multithreading-capable.[...] There are many ways we could design FlightGear with multiprocessing support." [12]
  • "If the people who actually make CPUs think that parallelism is the way to go, I wouldn't hold out too much hope on significant speed increases in the future. Sure, things will get faster, but not at the same rate we've seen so far." [13]
  • "As the hardware continues to develop in this direction high-performance software has to follow it if it's not to be left behind. Although FG doesn't need to be mpp capable right now, in a few years it will and now is probably a good time to start thinking about it if a _good_ solution is to be found by the time that it's needed."[14]
  • "In the end though, I just don't really think there's any other alternative. The h/w world is betting on parallelism for the future and software has got to fit the h/w."[15]
  • "Ultimately, mpp is inevitable because it is the only realistic way of achieving potentially un-limited processing power and we are seeing the general acceptance of this principle with the current generation of multi-core x86 processors." [16]
  • "Note that we're going to have to start thinking about threadin designs soon anyway, if we want to take advantage of all the fancy multi-core/multi-thread CPUs coming down the pipe. Rendering obviously has to stay within a single thread, but how much non-rendering work can we push out to worker threads? Ideally, everything would be "handed" to the renderer thread each frame, with all the matrices pre-cooked and ready for OpenGL calls."[17]
  • "I guess I'm thinking about how FG is pretty monolithic and wondering how much of an over-head there might be in making it more modular. Might also be worth thinking about parallelism aspects." [18]

Requirements

  • "The best candidates for separate threads [or parallelization in general] are processes that have relatively little input or output but require a lot of computation."[19]
  • "Threading is relatively safe if each thread is forced to play in a sandbox. A subsystem running in a separate thread must *not* touch any other subsystem (including the property manager), and must send and receive all information through a single, tightly-controlled interface."[20]
  • "That implies that a subsystem in a separate thread should not even include other FlightGear header files, much less access FGGlobals, FGInterface, the property manager, or anything else directly."[21]
  • "I think it would be better for the main thread to read the information from the property manager and then pass it to an object that the other thread can access."[22]
  • "This is the Great Myth of multithreading. Threadsafe components are not sufficient to protect against threadsafety violations. The only way to avoid deadlocks and race conditions is for the whole architecture to support them from top to bottom." [23]
  • "The point, again: *all* multithreaded code is susceptible to race conditions and deadlocks. There is *no* way around this. The only way to avoid them is to be very, very careful with your design. You cannot rely on libraries to save you. You cannot rely on simple techniques to save you. "[24]
  • "Really! We should go out of our way to find a workable non-threaded solution before we add new threads to the code."[25]
  • "Personally, I think that the idea of threading in the context of FlightGear is a *very* scary idea, especially from the standpoint of long term maintanence and keeping our code robust."[26]
  • "Threading (is) best avoided in my opinion, pretty much for the reasons you give. Instead of thinking of FG as a single threaded application, it needs to be a collection of standalone programs that run collaboratively - go back to thinking in terms of the external FDM option."[27]


Decoupling the graphics system

  • "I think the single most important step would be to run the graphics subsystem in it's own process, splitting it from everything else. On multi-core systems this would mean that the graphics subsystem gets the resources freed by the 'everything else' and the 'everything else' gets the resources freed by the graphics subsystem."[28]
  • "This would be a relatively small gain for the graphics subsystem and a much bigger gain for everything else, where it's arguable that it's needed, but it would allow much higher and more consistent rates in the FDMs, autopilot controllers & filters and Nasal."[29]
  • "The thing is though, that the graphics subsystem needs a lot of data and it's questionable that it could be transferred quickly enough. Therefore it's likely that the scenery & model loaders would need to be included in the graphics subsystem so once it's told what data it needs it can fetch it itself.[30]
  • "With a core to itself, the 'everything else' part of FG would benefit less by further splitting but if it was well designed it should make plug-ins much easier to implement and maintain.[31]
  • "In the longer term, thought needs to be given to 'box-rendering' the graphics - splitting the scene into several regions and processing them in parallel - but this is much easier said than done, especially as rendering is h/w based. Still, this is the sort of thing that newer versions of OGL/OSG will _have_ to address in the future, if they haven't already got some features in this area."[32]
  • "Ensuring that as much rendering work happens on other threads (OSG can be very aggressive about this) is a likely area of investigation" [33]
  • "In theory OSG can be doing DB paging, model and texture loading, updates and culls on worker threads, and the actual drawing on another. I have no notion how much of that is happening with the current code. My guess would be that pieces ported from PLIB/ SSG may not be playing 'nice' with the aggressive OSG pipe-lining, but equally, updating those pieces of code to be more OSG friendly isn't a quick hack either."[34]
  • "In general, FG has quite a few data dependencies internally which make multi-threading challenging right now - there's groundwork to make the data-dependencies more explicit (i.e, via the property tree) that has to happen before pieces can easily move to other threads."[35]


Multi-Process instead of Multi-Thread

  • "In projects like these I like to think multi-process could simplify things quite a bit." [36]
  • "I'm not really thinking in terms of 'threading' at all, which I think is a very limited and half-house sort of technique. But neither though do I think it needs to be thought of as a pure real time system. Rather, I'm thinking in terms of the external FDM mechanism already present in FG. Running the FDM on it's own hardware system doesn't need to be any more real time than the FDM running within FG on the same system but because it's not going to be limited by the frame rate it could safely be run much faster and with proportionately more consistency than with FG. If you're running it at say 100Hz within FG I would expect to be able to run it several times faster, if not tens of times faster if the system it was running on wasn't spending most of its time rendering. You'll still get a variation in the rate that the FDM runs at but I suspect that the variation would be about the same in absolute terms." [37]
  • "Most stuff can be decoupled so that the computation can even be split between different processes, which may even run on different machines [distribution]. Take again visualization (OTW) for example: It's (dynamic) input is position and orientation of the aircraft(s), output is altitude above ground. These are rather minor amounts of data which can easily go across a network. Most commercial FTDs and FNPTs (including these of the company I used to work for) do it that way. It pays to identify and separate the involved concepts using well-defined interfaces. Doing so leads to clean, extensible and efficient design." [38]
  • I'd perhaps favor splitting our code out into separate applications that use networking or shared memory or pipes to communicate. You lose some of the context switching efficiency of threads, but I think the code becomes more robust and maintanable because changes have less of a chance to propagate elsewhere to unintended areas ... "[39]
  • "Personally, I think the most sane approach with the highest chance of succeeding will be to pick some particular submodule that really makes sense to run on another machine or in a separate process and figure out how to split that off."[40]
  • "FWIW, I think that separating out as many FG subsystems as possible would be a very good way to go, especially in view of how CPU development is now solidly focussed on multiple core CPUs rather than ramping processing speeds, as was inevitable in the longer term." [41]
  • "Splitting up FlightGear into multiple functional units is something I'm really voting for. Especially, when you use a well-defined interface for every module. That way you can create a plugin-driven system that is easily extended upon, and could easily be split up physically using multiple machines across a network. I'm more into developing glass cockpits myself, but I'm also interested in creating a complete full-mission flight simulation suite. Especially since that would attract commercial funders (aviation industry companies like CAE, FlightSafety and Boeing) and possibly achieve FAA certification." [42]
  • "I like the consept of multiple programs (communicating through sockets or pipes) over threading anyhow, and that *forces* you to think about it :-)" [43]
  • "To follow the "do things right" rule I think it would be great to implement a generic interface or standalone I/O modules. Both Micro$oft FSX and X-Plane have such interface. The M&S HLA users would just need to build a shared module (.dll or .so) for a particular HLA RTI and load it via the standard FlightGear plug-in interface." [44]
  • "The architecture document you refer to has been around for a while, and while it offers an improved architecture for a flight simulator, it isn't at all clear how we can separate the existing monolithic core of FG into the different components." [45]

Benefits

  • "running the different sub-systems on different inter-linked computers and indeed on different hardware" [46]
  • "tuning the "frame" rate of individual sub-systems (the FDM itself may benefit from different and variable frame rate for the "lateral" and "rotational" elements of the aerodynamical derivative integrations)"[47]
  • "An advantage of this modularisation and inter-process communication is that as flightgear is multi-platform we can select the optimum platform for each module even utilising RTOS where needed or maybe specialist graphics engines." [48]
  • "I think that splitting FG into several parts would actually help reduce bugs, not increase them. Bugs would be limited to their particular subsystems and couldn't manifest themselves in other parts of the system, as they can do with a single monolithic system. Each discrete subsystem can only pass data back and forth, not bugs."[49]
  • "It pays to identify and separate the involved concepts using well-defined interfaces. Doing so leads to clean, extensible and efficient design." [50]

A proven Strategy

  • "A flight simulator as supplied by a commercial producer of training devices, has been split into a set of large functional blocks, each of them consisting of different modules" [51]
  • "Real-world flight simulation systems, as used in flight training devices, have been using this tactic for decades."[52]

Partitioning

  • "One problem is to identify parts that we will gain anything from moving to a separate thread. I have seen the FDM suggested in the past, but even on my (ancient) system JSBSim corresponds to about 1-5% of the CPU usage (estimated by looking at the rate sim time progresses in the standalone version of JSBSim). Andy has told me YASim is more expensive (it does more at runtime) but it is probably at most 20-30% of the CPU usage (guesstimate :).So, the prospective gains there do not look that large. Doing some profiling might make the picture clearer."[53]
  • "It'd be worth identifying which subsystems are the big time sinks (FDM? AItraffic?) to prioritise this." [54]
  • "I think the main targets for parallelization are the rendering pipeline and various "add-on" systems, like the traffic manager.Personally, I'd like to have threads (possibly with very limited interaction abilities) available in Nasal for isolated and computation intensive tasks (e.g. fast forwarding my fire cellular automaton :)."[55]
  • "Another thing that would work well is to proxy all nasal script invocations to a Nasal helper thread - again this assumes scripts basically interact with the sim via properties (which they already do) and that any system functions they call are thread safe - not very hard to do. As more and more functions get moved to nasal, this might become a very easy way to balance the CPU usage." [56]
  • "it's finding the parallelism that's the hardest part but things like the FDM, autopilot and Nasal do seem like obvious candidates. Even if we can't precisely balance the load, we could still improve the performance of parts of FG."[57]
  • "My first thoughts are that fgfs divides logically into a number of sub-systems: a) Flight Dynamics Modelling (FDM), b) Exterior View, c) Cockpit input and output (i.e. joystick controls,switches and displays), d) Motion system"[58]
  • "Some discussions have already taken place on JSBsim devel mailing list regards communication between "modules" of flightgear. My thoughts are that flightgear divides "naturally" into four major sub-system modules" [59]
  • "To my mind flightgear can be broken down into distinct plugin "modules". There is the FDM, the "external world" visualisation,the cockpit input and output (ie joystick,pedals,switches and displays) and possibly a motion system. These can be interconnected with some inter process communication scheme. All of the modules could be run on a SMP hardware (e.g dual dual-core cpus) or on separate computers. [distribution]"[60]
  • "I agree here, putting the FDM in it's own process would be a good idea." [61]
  • "We've toyed with the idea of an FDM server for a while, and there was even some work done on JSBSim towards making that happen. Recently, HDWysong has added the capability to use FlightGear as a visual front end for JSBSim as a separate, scripted application. It certainly would be a huge paradigm shift." [62]

Approach

  • "I agree that the approach you suggest, of starting with a single subsystem/module is probably the best way to start tackling this issue."[63]
  • "The main issue is inter-processor communication between modules ie between modules multi-threaded on multi-processor computers,between modules in processes on same computer,between modules in processes on networked computers."[64]
  • "I have been looking at the idea of using some database containing the property lists which would be read and updated asynchronously by the various modules at whatever frame rate is suitable for the module (not necessarily its "internal framerate")" [65]
  • "I am familiar with OpenLDAP and it looks like an LDAP schema can be derived fairly easily from the property list or better still from the XML schema (maybe programs exist to do this already?)."[66]
  • "One should not forget that FG has allready some networking capacity. This alone has allready allowed ppl to split fdm and rendering on several machines. Perhaps there is something to reuse here." [67]
  • "Keep in mind that the Generic protocol is one of the most inefficient for real-time data communication, as it communicates via plain text. If you want something that goes over 100Hz, you need to think of a binary system." [68]
  • "One could always break FlightGear into sub-applications and let them communicate via UDP ports."[69]
  • "If necessary it would be possible for parts of the program to hold it's own property tree, which is inaccessible from the global property tree, by keeping track of it's own root-node." [70]
  • "[...]I think a more flexible approach may be "self-contained" modules communicating by passing "properties" over TCP. The "remote" FDM is already a possibility and there is an example of a remote joystick but how easy would it be to break up the rest of flightgear?"[71]

Using the PropertyTree for IPC across processes (or threads)

Note  Also see Property threading
  • "Making the property manager threadsafe isn't the issue; sure it can be done. The problem is that *using* the property manager (or anything) from multiple threads leaves you open to race conditions that occur due to unsynchronized changes to the property values."[72]
  • "Now, you can fix this via explicit locking ("/controls/trigger-safety-lock-lock", heh), or implicit design (only one thread sets the /controls tree)."[73]
  • "I would love to see all the FG core stuff going that route - one network capable API that everything can work through including FG itself. As far as I can see FG looks like it's already half way there." [74]
  • "The main issue is inter-processor communication between modules ie between modules multi-threaded on multi-processor computers,between modules in processes on same computer,between modules in processes on networked computers." [75]
  • "The property manager seems like an ideal candidate for IPC messaging, so if we want, it can be done." [76]
  • "I have a long, long, long term plan to improve multi-threading support, by enforcing subsystems to *only* communicate via the property tree [...] it would then become possible to run any 'clean' subsystem on a pool of worker threads (maybe just one, maybe more)."[77]
  • "Maybe the property lists could be promulgated using a simple "directory" style database (eg LDAP if it has the performance needed) as most modules will probably write properties to be read by other modules so simultaneous writing of a particular property will not be a problem."[78]
  • "I have been working on extending the property code to add an SGRemoteProperty class to access properties on a remote host instead of locally." [79]
  • "I have been working on extending the property code to add an SGRemoteProperty class to access properties on a remote host instead of locally. Most of the tree is cached locally, but queering the value of the end-node is done through a socket connection." [80]
  • "I have this sort of working (but need more debugging).Next to the SGRemotePropertyNode I've also been considering queering multiple properties in one shot, instead of one by one."[81]
  • "The property system is already a great mechanism for communication among the sub-systems of FG. A great project would be to look at making the property system thread-safe, and follow that up by turning it into a "blackboard" that can be used by distributed copies of FlightGear."[82]
  • "The key thing is, we have a good modularity thanks the to SGSubsystem and properties. It needs some enhancements in some limited ways, and we have state that violates the property-tree abstraction for various (valid!) reasons, but I don't believe there is any major obstacle to running batches of subsystems on worker threads in the medium term. It's not a high priority compared to other things, since the biggest multi-thread enhancements come 'for free' thanks to OSG. But it does not need some major re-think or re-design, as far as I can see."[83]
  • "Simply put, the mechanics for doing this with FlightGear are already in place, you only need to take a slight detour over a communications link. This has its advantages too, such as added security (no possible code injection) and inherent networkability. Downside is that it takes a little more brain-food to make it work." [84]

Conclusions

  • "the _need_ for a redesign of FG to run on MPP systems as it gets ever clearer that significant increases in computing power have more or less stalled in terms of height (cpu speed) and in future, will come instead from width (parallel processing) - FG's current design is effectively obsolete]" [85]
  • "Yes, it might be time for that. However, the recent work on [threaded] model loading is certainly a step in the right direction." [86]
  • "What has happened is that FG was designed, in computing terms, a long time ago, when computing at this level was still in it's very early days and just making a workable solution with the available hardware and software was an achievement in itself.Since then however, not only has an awful lot has been learned about system design and the practicalities of maintenance but a paradigm shift in computer hardware has swept over everything. FG's architecture was designed around h/w and s/w designs that were leading edge at the time but everything has moved on since then."[87]
  • "as new ideas and features have been thought of and implemented, in the light of new technological advancements, they've had to be massaged to fit an architecture that has become obsolete, both in design, maintenance and in terms of the h/w it runs on."[88]
  • "In summary then, for FG to survive in any greater form than a quaint alternative to better alternatives, which I admit may not exist yet but will certainly do so at some point in the future, FG _needs_ to be re-designed with an architecture that not only makes use of current hardware, specially with regard to multi-core/distributed computing systems, but also enforces scope of sub-systems and allows multiple concurrent versions of sub-system."[89]
  • "the bottom line is that FG is simply going to have to face up to this issue at some point. A few people here have been bringing this topic up for a few years and now that multicore processors are the norm it's clear that the issue isn't going to go away. Like it or not, and I mean no offence or criticism by this, the current FG architecture is now obsolete. While single core and single processor systems were the norm it was fine - the software design was well fitted to the systems on which it ran - but parallel processing has always been the way things were going to go. It has been inevitable ever since super-computer designers realised that the only way that ever increasing performance could be achieved was by parallelism and now it's well and truly on the desktop."[90]
  • "I don't think anyone really _likes_ the idea of the extra work involved - it's going to be difficult and hard work, but living in the past isn't going to work either."[91]
  • "Doing this is a daunting but necessary task. It also cannot be the only task that people work on, but it has to be done nonetheless, and sooner rather than later. It will also not be a quick and easy task - it will probably take a couple of years of thinking, experimenting and testing to just come up with a viable architecture, let alone implementing a 100% solution, but it needs to be started now."[92]
  • "Of course, it's not going to be easy, but denying it won't make the issue go away." [93]


So, this discussion focuses solely on implementing a process-based component-system where modules (subsystems) only need to modify simulator state by getting/setting PropertyTree variables, with the implicit design assumption that only one single thread accesses the core property tree. Such components (run in a separate process) may however internally use their own property tree instance in order to provide and access state to/from the fgfs core process. All required sandboxing (as mentioned above) will be implicitly handled by running components in separate processes.

Background & Motivation

Related wiki entries

Related discussions

This proposal is largely based on the following discussions:

Why not threading?

In this proposal, a process-based approach to parallelization is favored over threading, as threading would probably introduce a whole number of possible problems, namely:

  • threading can become incredibly tricky in any sort of non-trivial application
  • platform support, processes don't need additional dependencies
  • synchronization
  • locking of shared resources/data

While process-based parallelization is comparatively 'expensive', it is antiticpated that this should turn out to be an acceptable tradeoff, as process-based parallelization of applications encourages a truly modular, loosely coupled design where data and resources may only be "shared" by providing explicit means and mechanisms for doing so (rather than having access to all global variables within a process supporting multiple threads).

Furthermore, establishing building blocks to provide support for process-based parallelization in FlightGear would eventually not only help FlightGear becoming more modularized and "parallel", but would also provide a possible foundation for future efforts to help FlightGear components support networked distribution, as well.

Objective

The priorities in descending order are to develop and discuss a potential design draft that would enable specific FlightGear components to be:

  • easily modularized
  • parallelized and
  • eventually also distributed

By using a design where individual components are implemented in a sufficiently generic fashion to facilitate being run as standalone programs/processes(communicating with the FlightGear core using its already available network support), rather than being necessarily directly run within the fgfs core executable process space (be it within the main loop or as a separate worker thread, none of which would help FG modularization).

Requirements

Modules must be totally self-contained and their only means of IPC with the core process is the FlightGear PropertyTree itself, being accessed via network interfaces already provided by FlightGear. Such modules may only communicate data provided via the PropertyTree.

Assumptions & Simpflications

In order to simplify a possible design, there are several assumptions that are made about said standalone modules in the initial drafting phase (some of which might be lifted later on):

  • all "modules" (standalone processes) are single-threaded using a conventional main loop.
  • modules may only be connected once to a running instance of the FlightGear core
  • there is no support for traditional shared memory.
  • the only way for standalone FlightGear components/modules (processes) to communicate with the FlightGear PropertyTree is using FlightGear's networking capabilities (i.e. setting/getting properties via telnet).
  • all data exchanges between modules and the FlightGear core have to take place by using the FlightGear property tree, that is data types are determined by the primitives supported by the property tree.
  • standalone components are reduced to being merely consumers and/or providers of data that is (made) accessible in the FlightGear PropertyTree
  • that is, there is no direct means provided for standalone components to do RPC across process boundaries, if RPC functionality is desired it needs to be explicitly implemented using existing means to trigger actions in the (core or module) process by writing to a SGPropertyListener-bound property node, which in turn executes a corresponding callback function.
  • all modules are supposed to route their communications (get/set requests) through the FlightGear core, there is no concept of supporting direct (p2p)inter-module communications that would bypass the FlightGear PropertyTree.
  • performance considerations are not important for the moment, the point is just to modularize and parallelize certain FlightGear components by splitting them up, so that they can be run as separate processes


While it is clear that these are serious restrictions and that several types of subsystems could not be implemented with these limitations, it is also much more feasible to actually start designing a corresponding infrastructure.

Expected Challenges

Allowing a possibly large/increasing number of different processes to query/modify state variables/properties within the FlightGear Property Tree asks for clean organization and structuring to avoid potential problems that may, among others for example result from processes 'tie'ing properties that would cause conflicts by resulting in circular callback invocation in different processes, or processes writing to properties that they should normally not have access to, or even multiple processes trying to write to the same properties, basically invalidating previous state thereby.

Workarounds

However, it is expected that enriching the current concept of properties with a couple of attributive meta information to enable enforcement of legitimate/valid accesses, would help declare a safer framework, namely by providing support for the following features:

  • introduce the concept of property owners (i.e. on subsystem/component basis)
  • introduce the concept of exclusive property ownership
  • introduce the concept of property permissions (i.e. read/write/delete access)
  • introduce the concept of property units (to optionally ensure proper data is being written to a node)
  • introduce the concept of property ranges (to specify valid value ranges for a property)
  • introduce the concept of property requirements/conditions (i.e. MUST_EXIST, MUST_NOT_EXIST, MUST_NOT_BE_TIED, etc)


Having such attributes in place that reflect this sort of meta information, would help formalizing and enforcing access specifications for certain properties, to ensure encapsulated (i.e. private/protected) use of properties where subsystem/component implementations can make the assumption that such internal state is immutable from the outside, even though it may be publicly available in the core's PropertyTree. Also see Recommended Property Tree Enhancements.

Further Assumptions

  • for each module-specific property there may be only one single function tied to it with (exclusive) writing priviledges, all other listeners are reduced to being read-only, this is to avoid modules without ownership executing uncontrolled writes.

Execution schemes

A process-based module/component system, while being comparatively straight-forward to design (at least compared to threading), has its own challenges due to the fact that the component or module would not be initialized in a hardcoded fashion, but rather it would support being run using different schemes or scenarios, each of which has its own advantages and challenges:

  • the core executable may fork -after startup-a separate, standalone executable (on the same machine) to create the module's process: this is straightforward to implement, does however not easily work for possible scenarios where distribution among different machines/architectures may play a role (at least not without having to resort to remotely forking via SSH)-also, when the main process terminates, all forked processes are also terminated-however, key init state can be easily provided to the forked child process by using command line parameters)
  • the core executable may -after startup- fork itself (again, on the same machine) using special parameters, to fork the module's process (pretty straightforward to implement, shortcomings are image duplication of the executable in memory, as well as potential modules having the same library dependencies as the main executable-also, when the main process terminates, all forked processes are also terminated, changing/adding (recompiling) module code at runtime cannot be easily accomplished as this would involve overwriting the core executable as well-however, key init state can be easily provided to the forked child process by using command line parameters)
  • the module's process may be started manually after core startup, so that it has to register with the core executable's process (this requires a certain startup order and manual interaction, as the module would then need to contact the core (via network) in order to register itself, in this scenario the core would be a server, which would facilitate modules connecting/disconnecting at runtime, so that for example a recompiled module could simply be tested without having to shutdown/restart the core process, however key init state needs to be passed either manually or retrieved from a standard location within the network)
  • the core executable may want to register itself with an already running module (in this scenario, where the core would act as a client, the module itself would need to act as a server-while this may sound overly complex and involved, it does have some bearing given that such a possibility would enable designs where simulator components may keep running, while the core could be shutdown/restarted, or even recompiled while several simulator components could remain running and simply wait for a reconnect-however key init state needs to be passed either manually or retrieved from a standard location within the network)


While the latter two usage scenarios are more appealing from a distribution-centric point of view, the other two scenarios are easier to implement, as they would not require any sort of network-based central registration service, where components would have to register themselves before being visible to other processes-basically similar to CORBA's POA/Name Service (which would also add the requirement to start the registration process first, OTOH this could also be automatically handled, too).

Providing RPC support

Emulated RPC support would be based on the current concept of registered SGPropertyListeners invoking callbacks that would in turn make sure to deal with any (optional) call arguments and return whatever state is required.

However, to be truly useful and genereic in a wider context, the current use of PropertyTree/RPC would need to be slightly extended and somewhat more formalized by means of providing standard ways to create and execute RPCs, as well as deal with any relevant return state.

Limitations using current FG RPC mechanisms

While the basic building blocks to support all sorts of RPC using the PropertyTree/network combination are already available in FlightGear, doing so is currently not yet very convenient, this is mainly because:

  • RPC callers have to manually "marshall" RPC parameters into a hierarchical SGPropertyNode format (as is i.e. also the case for arguments to fgcommands), so that the arguments are representable within the structure of the PropertyTree, using temporary PropertyTree nodes.
  • RPCs may have parameters of certain type (optional-ones, too) or not, there is currently no way to formally specify such requirements and ensure that this is validated at runtime
  • RPC implementors have to manually process/parse the arguments (SGPropertyNode*) passed to the RPC to look for any required parameters and deal with potential inconsistencies.

For any type of function call that uses the property tree as IPC mechanism, function attributes such as the function's signature or return state need to be manually prepared/processed for each call, possibly including redundant validation code as well.

This involves basically every time the following steps:

  • manually marshall arguments into propery tree format
  • temporarily save the created sub tree somewhere in the tree
  • call RPC function using a pointer to the node as parameter
  • manually decompose structure provided by pointer
  • validate types, obtain actual values
  • execute RPC code if types/values are valid
  • marshall return results into SGPropertyNode/PropertyTree format
  • post-process/evaluate results in callee

Module types

For starters, standalone modules can be classified to fall into the following groups:

  • permanently running
  • temporarily running/on demand

Furthermore, future modules can also be categorized based on whether they :

  • are allowed to be forked/connected multiple times to the core process (i.e. different FDM processes)
  • may only be forked/connected once to the core process

Module control

Based on the subsystem functionality provided by SGSubSystem, control over individual standalone components could be exercised using the following triggers (exposed within the FlightGear core PropertyTree):

  • start-module
  • stop-module
  • suspend-module
  • resume-module
  • update-module

These triggers could be implemented by exposing corresponding properties within the core's PropertyTree, ensuring that the component itself has exclusive ownership.

To enable the core, as well as other modules to query the actual state of a given module, providing additional state information is also likely to be useful:

  • current-state (starting,running,stopping,suspending,resuming,updating)
  • current-update-interval-hz

Module Registration

In order to enable modules to interact with the FlightGear core, they need to publish their services (functionality) within the core (again, using the PropertyTree), including information such as:

  • module name (i.e. used to set up a specific node path in the PropertyTree)
  • module description
  • module version


Module Workflow

Module Initialization

  • parse command line options
  • connect to FlightGear core via network (i.e. telnet address:port)
  • set up exclusive node directory, i.e. under /modules/$name, where $name would be the unique name of the corresponding module
  • publish meta information about module

Module Registration

  • set up aliased properties for reading/writing state variables of simulator

Module Updating

  • in the module's main loop poll all relevant simulator state
  • modify relevant state variables and publish in simulator

Module Suspension

  • stop polling all simulator state, except for module state

Module Termination

  • untie all tied properties
  • remove module from module namespace in property tree
  • terminate process
Retrieved from "https://wiki.flightgear.org/w/index.php?title=Modularizing,_parallelizing_and_distributing_FlightGear&oldid=83130"