High-Level Architecture: Difference between revisions

Jump to navigation Jump to search
m
No edit summary
(67 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Official Core Development Effort}}
{{infobox subsystem
|name = HLA Support
|image=Hla1-12-2015-screenshot-by-Stuart-Buchanan.png|
|alt=Screenshot showing HLA prototype at LOWI
|started= 05/2009
|description = Implementing support for the High Level Architecture to modularize FlightGear
|status = Under active development (2009-2016)
|developers = Mathias Fröhlich[http://sourceforge.net/p/flightgear/mailman/message/23723508/][http://sourceforge.net/p/flightgear/mailman/message/22914687/], James Turner[http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg39693.html], Stuart Buchanan[http://sourceforge.net/p/flightgear/mailman/message/34600516/], Richard Harrison [http://sourceforge.net/p/flightgear/mailman/message/34632142/]
| changelog =
* {{Git File History|project=flightgear|path=/src/Network/HLA}}
* {{Git File History|project=simgear|path=/simgear/hla}}
}}
{{Multicore}}
{{Multicore}}


'''High-Level Architecture''' ('''HLA''') is a general purpose architecture for distributed computer simulation systems.
'''High-Level Architecture''' ('''HLA''') is a general purpose architecture for distributed computer simulation systems.


Using a HLA, computer simulations can interact with other computer simulations regardless of the computing platforms. The interactions between the simulations are managed by a ''Run-Time Infrastructure'' (''RTI'').
Rather than have the entire simulation within a single executable, the simulation is split into different [[Federate|Federates]], which interact with each other by a ''Run-Time Infrastructure'' (''RTI'', a message bus that handles serialization of messages, events and objects), with federates typically running in their own threads/processes and each federate process having access to the full virtual process address space provided by the OS instead of having to share it with other subsystems (i.e. 32 bit platforms may make better use of virtual RAM that way).
 
There are three big advantages to this over a monolithic simulation (e.g. FlightGear V3.6):
# It provides a robust environment to make the simulator multi-threaded, taking advantage of computers with multiple cores, or indeed running different parts of the simulation on different computers (including even different platforms and operating systems).
# It allows us to split out parts of the simulator such as [[AI]] (by [[Decoupling the AI Traffic System]]), the [[FDM]], [[Nasal]] scripting <ref>http://forum.flightgear.org/viewtopic.php?p=265721#p265721</ref> and [[Supporting multiple renderers|Renderer]] from each other and less time-critical sub-systems such as [[Advanced weather|weather]] so that we can get consistent (and perhaps higher) frame-rates (i.e. reduced [[How the Nasal GC works|Nasal GC]] impact on frame rate).
# It provides a very good framework to allow anyone to create components that interact with FlightGear using programming languages other than C/C++ (think Ada, Java, Python etc), which may be running in their own threads, and reside in separate binaries<ref>http://sourceforge.net/p/flightgear/mailman/message/34196458/</ref>, which will be also easier to debug/troubleshoot (think regression testing, i.e. running a self-contained subsystem in a dedicated gdb/valgrind session), without having to know how to modify/patch and rebuild FlightGear.
 
A good overview of how the HLA architecture works can be found here http://www.pitchtechnologies.com/wp-content/uploads/2014/04/TheHLAtutorial.pdf
 
==Design==


{{FGCquote
The planned overall design is as follows:
  |Ever heard about how the firefox/google chrome browsers are hoping to better leverage multicore systems by using a separate process for each website (tab) ?<br/>
* We intend to use OpenRTI as the underlying RTI.  This is an open source IEEE 1516 standard RTI, written by Mathias Fröhlich and available from https://sourceforge.net/projects/openrti/.
That's basically in line with how FlightGear is going to work once HLA materializes a little more.<br/>
* Mathias has also written an open source toolkit to act as an interface library, with a working name of SimKit.  This sits on top of the RTI and simplifies interfacing with the RTI massively.  It also has excellent Python integration, making it very easy to write scripted Federates. It's available from: https://sourceforge.net/projects/sim-kit/
<br/>
 
The main problem here is that most "legacy" software (like FlightGear) was originally written for a computer with a single processor (CPU). <br/>
Using [[SimKit]], the integration point is within the FlightGear code (rather than SimGear), in particular the code under $FG_SRC/Network/HLA.
<br/>
 
These days however, it is common for a computer to have multiple processors, with each having multiple cores. Nowadays, it is hard to come across new computers that have &lt;{{=}} 4 cores. Which basically means that there are 4 CPUs that can handle operations independently. Imagine it like having a car (or airplane!) with 4 engines, while the "driver" (you) only knows how to operate a single engine.<br/>
==Federate Object Model==
However, old software was written for a single processor, typically using a single main loop that assumes there's a single FAST processor. Which is why people would see better frame rates whenever they upgraded their CPU. This basically means that FlightGear would typically be CPU-bound (limited by the speed of the CPU, that is supposed to hand over  jobs to the GPU).
 
  |{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=218069#p218069
A key part of the design is writing the Federation Object Model (FOM), which defines the objects and updates that are published by the RTI.  While it might at first glance seem a good idea to use the FOM to share the internal property tree across multiple federates, this is probably the wrong way to use HLA as the granularity is too low <ref>https://sourceforge.net/p/flightgear/mailman/message/34795859/</ref><ref>https://sourceforge.net/p/flightgear/mailman/message/34803436/</ref><ref>https://sourceforge.net/p/flightgear/mailman/message/34794675/</ref> and it's likely to lead to synchonization issues. Instead, we'll need to make explicit decisions about the data models to communicate.
    |title=<nowiki>Re: How dangerous is Rembrandt in general?</nowiki>
 
    |author=<nowiki>Hooray</nowiki>
The FOM is a set of XML files in [https://sourceforge.net/p/flightgear/fgdata/ci/next/tree/HLA/ fgdata/HLA/]
    |date=<nowiki>Tue Sep 09</nowiki>
 
  }}
The orientation data in the FOM (SGOrientationWGS84) are Quaternions ([https://en.wikipedia.org/wiki/Quaternion https://en.wikipedia.org/wiki/Quaternion]), and those are the i,j,k values.  From memory, the orientation is relative to a earth-centered-frame, and there are functions in SimGear to convert to heading/pitch/roll in combination with the object's position.
}}
<ref> {{cite web
  | url    = http://forum.flightgear.org/viewtopic.php?p=280098#p280098
  | title  = <nowiki>Re: HLA </nowiki>
  | author = <nowiki>stuart</nowiki>
  | date  = Mar 21st, 2016
  | added  = Mar 21st, 2016
  | script_version = 0.25
  }}</ref> <ref>{{cite web
  | url    = http://sourceforge.net/p/flightgear/mailman/message/34963798/
  | title  = <nowiki>Re: [Flightgear-devel] HLA</nowiki>
  | author = <nowiki>Stuart Buchanan</nowiki>
  | date  = Mar 24th, 2016
  | added  = Mar 24th, 2016
  | script_version = 0.25
  }}
</ref>
 
Richard's [[Emesary]] system would also probably work quite well with the HLA FOM to give us a way for models to communicate with other models.<ref> {{cite web
  | url    = http://sourceforge.net/p/flightgear/mailman/message/34985731/
  | title  = <nowiki>[Flightgear-devel] Message passing interface for Nasal</nowiki>
  | author = <nowiki>Richard Harrison</nowiki>
  | date  = Apr 1st, 2016
  | added  = Apr 1st, 2016
  | script_version = 0.25
  }}
</ref>
{{Appendix}}
 
==Federates==
 
Federates can be written in most languages, but SimKit provides very good hooks for writing them in C++ and Python.
 
For some Subsystems split off from the existing FlightGear source, it's fairly easy to create an executable with its own property tree<ref>https://sourceforge.net/p/flightgear/mailman/message/34632142/</ref> and have shared C++ code to map FOM objects to property values. However, this is an implementation detail - the whole point of HLA and the FOM is that it makes no assumptions about what Federates do with the data.
 
Regarding weather, in a HLA/RTI context the way to do this would be to have a weather engine as an RTI Federate. This would run any required weather simulation, and pass (very) local weather conditions to each of the aircraft/tower/windsock in the simulation, plus publish position information on clouds for use by visualiation engines.<ref>{{cite web
  |url   = https://sourceforge.net/p/flightgear/mailman/message/35595300/
  |title = <nowiki> Re: [Flightgear-devel] Traffic 2020: Towards a new Development
Model for FlightGear AI Traffic </nowiki>  
  |author = <nowiki> Stuart Buchanan </nowiki>  
  |date   =  Jan 10th, 2017
  |added  = Jan 10th, 2017
  |script_version = 0.40
  }}</ref>
 
==Current Status ==
Last updated: (13/01/2017)
 
Stuart is cautiously optimistic that he may be able to provide the start of a HLA implementation quite soon, as one of the blocking factors may be resolved soon.<ref>{{cite web
  |url    =  https://sourceforge.net/p/flightgear/mailman/message/35595300/
  |title  =  <nowiki> Re: [Flightgear-devel] Traffic 2020: Towards a new Development
Model for FlightGear AI Traffic </nowiki>
  |author =  <nowiki> Stuart Buchanan </nowiki>  
  |date  =  Jan 10th, 2017
  |added  =  Jan 10th, 2017
  |script_version = 0.40
  }}</ref>
 
 
Currently, there is some very old HLA support in SimGear. This is very out of date and should be ignored.
 
Stuart has HLA support using the latest OpenRTI and SimKit working on a local build, but is waiting for Mathias to officially publish his SimKit before he pushes his changes.  So all of the following is local-only, but included here for information.
 
The Flightgear core currently supports HLA as follows.
* SimKit integration, reading the SimKit FOM and connecting to an OpenRTI RTI.
* Instantiating MP AI objects so users can view objects published over the RTI by other Federates.  This is currently somewhat unsatisfactory as it overloads the MP code, where really these objects are more basic.
* Pick up Environment Objects for this instance and use them to set the local METAR string for weather generation.


{{FGCquote
{{See also|FGTraffic}}
  |Given the multicore revolution, what people have been doing typically is to use separate threads for certain, well-defined, tasks - such as running a dedicated thread for sound or for loading AI models. <br/>
We currently have the following other Federates:
In addition, OSG itself does have fairly extensive multithreading support as long as its coding patterns and protocols are used, so that OSG can internally decide which part of the job it can render asynchronously, on another core.<br/>
* '''fgogel''' - An AI model written in python the publishes over to the RTI. Part of SimKit, but handy nevertheless!
<br/>
* '''fgtraffic''' -  to run an AI Scenario externally to the FG Core
But otherwise, FlightGear's main loop still is built around the "single-CPU" concept, which is why there is a ton of stuff done in the main loop that could be done separately at some point, which would free up resources for better frame rates/latencies. The other issue with conventional threading is that threading is fairly low-level, it is very easy to write code that works ~80% of the time and crashes ~20% of the time (or vice versa). Processes are easier to understand than threads, and it is far more difficult to crash the main process - simply because a process doesn't typically have access to internal data structures. That is however also a problem because each FlightGear process needs a way to talk to the main process and invoke commands, get data etc - that is why some kind of inter-process communication is needed (IPC).
* '''fgmetar''' - written in python that retrieves the closest METAR station for other published Federates and publishes the METAR for them to pick up. This could be expanded to provide [[FGPythonSys#Weather_Simulation|a general Weather Engine]].
  |{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=218069#p218069
    |title=<nowiki>Re: How dangerous is Rembrandt in general?</nowiki>
    |author=<nowiki>Hooray</nowiki>
    |date=<nowiki>Tue Sep 09</nowiki>
  }}
}}


{{FGCquote
Stuart now has the [[AI]] Manager working as an external binary, and publishing updates over the RTI which are displayed as MP Aircraft. Stuart has created a little python script that looks for objects with locations being published over the RTI and publishes the closest METAR to them, then modified FG to pick up the appropriate METAR string for this instance and use it as the real-time input for weather generation.  
  |HLA is a fancy message-bus framework where each component ({{=}}called a "federate") can talk with other components across a bus called a REAL-TIME INFRASTRUCTURE (RTI) using an IPC framework that works across sockets and/or shared memory.
  |{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=218069#p218069
    |title=<nowiki>Re: How dangerous is Rembrandt in general?</nowiki>
    |author=<nowiki>Hooray</nowiki>
    |date=<nowiki>Tue Sep 09</nowiki>
  }}
}}


{{FGCquote
The python script could be fairly easily enhanced into a more substantial weather engine allowing a single coherent weather environment to be used across the simulation. Stuart also started work modifying [[FGViewer]] so that it works with SimKit. This is a fairly big deal and quite a bit of fairly dull plumbing work but would allow us at least have a view of the simulation entirely separate from the running simulation.
  |In non-coding terms this means that FlightGear currently works such that FlightGear is a "single person" doing a well-defined job (which is HUGE, it's basically a list of 10000 things each frame) - for some tasks, there are "helpers" ({{=}}other people) that can help with certain tasks, such as doing sound processing for example - but usually, there's a single entity responsible for processing the main loop. <br/>
<ref> {{cite web
<br/>
  | url   = http://sourceforge.net/p/flightgear/mailman/message/34877355/
The "problem" is that modern computers provide a "team" of people and Flightgear in its current form doesn't really know how to leverage this "team" (of processors/cores) efficiently, so depending on your hardware,  60-70% of your CPU resources may be sitting "idle" - waiting for work that FlightGear doesn't know how to assign.<br/>
  | title = <nowiki>[Flightgear-devel] HLA Status Update 23/2/2016</nowiki>
<br/>
  | author = <nowiki>Stuart Buchanan</nowiki>
While there are some ways to add more team members to the existing FG "group", the next issue is that work needs to be coordinated, or the whole thing will "explode" sooner or later due to sheer chaos, because someone (on your team) is doing something that the other one is unaware of, which is a violation - and which triggers a segfault/crash, i.e. your operating system is watching your "team" of FG people/CPUs and once it sees that the "flightgear team" is doing something crazy, it simply kills the whole team because it didn't behave properly  :D <br/>
  | date   = Feb 23rd, 2016
<br/>
  | added  = Feb 23rd, 2016
Equally, the OS may decide to kill FG because it doesn't manage memory properly
  | script_version = 0.25
  |{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=218069#p218069
  }}
    |title=<nowiki>Re: How dangerous is Rembrandt in general?</nowiki>
</ref>
    |author=<nowiki>Hooray</nowiki>
    |date=<nowiki>Tue Sep 09</nowiki>
  }}
}}


{{FGCquote
Separately, Erik has been doing some preparatory work suitable for supporting HLA in JSBSim by adopting so called PropertyObjects to hopefully get rid of tied properties (http://sourceforge.net/p/jsbsim/mailman/message/34720784/)
  |So what core developers are working towards is looking at the existing FlightGear main loop, to come up with a list of things/tasks that can be done asynchronously, i.e. in parallel with other tasks - once that list is complete, people are looking at what each "team member" needs in terms of "data" and "functionality" - e.g. some team members may only need to access the sound systems, others need to access the property tree, the AI system, the FDM and so on.<br/>
<br/>
The next thing is that core developers need to come up with a bunch of interfaces (so called APIs) that can be provided/used by each "team member" (task/federate). At that point, it's mainly a matter of wiring up each team member and letting them form a so called "federation" of workers. All of a sudden the main loop would become fairly lightweight and all the non-rendering tasks would be handled by "coworkers".
  |{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=218069#p218069
    |title=<nowiki>Re: How dangerous is Rembrandt in general?</nowiki>
    |author=<nowiki>Hooray</nowiki>
    |date=<nowiki>Tue Sep 09</nowiki>
  }}
}}


{{FGCquote
<references/>
  |The added advanatge here is that there's a per-process memory limit, and there will be left more RAM for people on 32 bit OS because each process will have its own address space - equally, debugging/profiling and troubleshooting FlightGear will become easier because individual components can be tested, without having to fire up the whole thing.<br/>
Also, FlightGear is currently written primarily in C++ - once HLA/RTI is in place, future components could be developed using other languages, such as Python, Java or Ada for example.<br/>
<br/>
(HLA is partially supported (as a multiplayer protocol substitute) already. But the way core developers are hoping to use HLA is to split up the simulator into a bunch of processes (think programs) that communicate with each other, to allow the main loop to be rendering only. That way, all the non-rendering tasks will be running on separate cores (threads/processes), while the main loop will only be doing rendering stuff, blazingly fast.)
  |{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=218069#p218069
    |title=<nowiki>Re: How dangerous is Rembrandt in general?</nowiki>
    |author=<nowiki>Hooray</nowiki>
    |date=<nowiki>Tue Sep 09</nowiki>
  }}
}}


For additional information, please see [[FlightGear HLA support (High Level Architecture)]]. Also see [[FlightGear CIGI Support (Common Image Generator Interface)]].
== Related ==
For additional information, please see:
* [[Technical_Reports#A_New_Architecture_for_FlightGear_Flight_Simulator]]
* [[Decoupling the AI Traffic System]] (currently work-in progress, highlights the limitations of the current AI system related to distributed/multi-instance and multiplayer usage)
* [[FlightGear HLA support (High Level Architecture)]] (outdated, documenting the original HLA infrastructure code)
* [[FlightGear CIGI Support (Common Image Generator Interface)]].
* [[Modularizing, parallelizing and distributing FlightGear]] (background info/discussions pre-dating HLA adoption in FlightGear)
* [[Distributed Interactive Simulation]] (DIS specific review of the FlightGear multiplayer system/protocol and its limitations, which can be addressed via HLA)


== External links ==
== External links ==
160

edits

Navigation menu