Decoupling the AI traffic system

Caution  Developer information: Some, and possibly most, of the details below are likely to be affected, or even deprecated, by the ongoing work on FGTraffic. This is because Durk is inclined to break away from the approach he took for the current AI traffic system. In his opinion, the code has become too clunky too be maintainable, he will try to spend a limited portion of his development time for bug fixing the current code, but please don't expect any major changes any more. The new approach he has in mind should surpass everything we currently have, but it's going to take some time until it's all implemented. To avoid conflicting efforts, you are advised not to start working on anything directly related to this without first coordinating your ideas with FlightGear core developers using the FlightGear developers mailing list. talk page..

checking how difficult it would be to put all aircraft related subsystems (fdm, replay, history, controls etc) into a single SGSubsystemGroup named "main-aircraft" to easily make the whole shebang optional using a single --prop for "FGCanvas" use, but also to check if it's feasible to prepare things for later reuse by the AI traffic system (for AI traffic that uses actual FDMs, APs and RMs - but also so that things are affected by the environment) , and it's actually working - even though reset/re-init is obviously hard-coded currently, which I am breaking by shuffling around subsystems, but as long as each SGSubsystemGroup implements the full SGSubsystem interface (postinit, reinit, shutdown etc), this could help clean up fg_init.cxx quite considerably [1]

Status / News

This article should be considered outdated, while the underlying issues still persist, this is currently being worked on via the HLA effort. In particular, there's a prototype named "fgai" that demonstrates how to use HLA in order to decouple an AI component from the FG main loop, please see: [2] [3].

The long term plan is to move the current AI system into its own HLA component, so that it can be run separately from the fgfs main thread and to ensure that AI traffic can be properly synchronized among multiple clients (multiplayer) or multiple fgfs instances that are configured in a master/slave fashion (think FSWeekend/LinuxTag).

I've started looking at the HLA integration for FG, after some very useful discussion with Mathias Froehlich. This is just a note to say what I'm looking at and how I'm thinking of developing it. Any comments/advice is welcome - this is a new area for me. As background, we currently have a partial HLA implementation in SimGear which has a dependency on OpenRTI. As the name suggests, OpenRTI is an open-source RTI, which is developed by Mathias himself. — Stuart Buchanan (Nov 6th, 2015). [Flightgear-devel] HLA developments. (powered by Instant-Cquotes)

As a first use-case I expect to start looking at a very basic AI integration so other aircraft in the federation are displayed within FG. That doesn't really leverage the power of HLA, but should at least allow me to get to grips with the problems and architecture before moving onto more ambitious plans, including the Holy Grail of separating the viewer from the FDM. — Stuart Buchanan (Nov 6th, 2015). [Flightgear-devel] HLA developments. (powered by Instant-Cquotes)

I've successfully replaced the HLA implemented in SimGear with SimKit (the name of Mathias' abstracting layer). This has allowed me to replace ~ 17000 lines of code in simgear/flightgear with 500 in flightgear alone. I'll need some help with the cmake plumbing before it can be checked in. I've also successfully joined an OpenRTI HLA federation, and received updates from an external source, which are displayed as an MP aircraft in the property tree. — Stuart Buchanan (Nov 16th, 2015). Re: [Flightgear-devel] HLA developments. (powered by Instant-Cquotes)

I've now got an external HLA Federate being displayed in FlightGear. The implementation right now is very basic - I have a KC-135 circling above LOWI at 10kts and 500ft AGL - but it's enough to build on. The next steps I'm planning are as follows: - Flesh out the Viewer Federate implementation, possibly to include mapping of HLA data to properties (not sure exactly how to do this yet). - Publish the FlightGear aircraft instance position to the RTI so it can be picked up by other viewers. - Create a Multiplayer Proxy as a Federate that proxies between the MP network and the RTI. This will run in a separate thread or as a separate executable. - Split out the traffic Manager in a similar way. This is obviously focussing on the viewer aspect, and doesn't address splitting out the FDM or control elements. — Stuart Buchanan (Nov 17th, 2015). Re: [Flightgear-devel] HLA developments. (powered by Instant-Cquotes)

I thought I should provide an update on where I've got to in HLA integration. I've managed to split the AI Manager out as a separate Federate and executable. So AI scenarios can now be run outside of FlightGear itself, and viewed within the FG instance. — Stuart Buchanan (Dec 29th, 2015). [Flightgear-devel] HLA Update. (powered by Instant-Cquotes)

Goals

Decouple the AI traffic system from FlightGear so that it may eventually become a standalone component to be run in a separate thread or process where all communications with the fgfs core process would be handled and dispatched through an IPC-enabled version of the property tree (see Remote Properties), which would ultimately not only help improve runtime performance (of FlightGear, as well as the AI system) but also provide a possibility to handle AI state synchronization across multiple FlightGear multiplayer clients by feeding in all AI traffic via the FlightGear multiplayer server, so that all connected clients would get to see identical AI traffic, that is properly synchronized.

Wouldn't it be cool if all the carriers in the flightgear world were simultaneously MP and simultaneously shared? Wouldn't it be cool if we could inject additional AI traffic that was then seen simultaneously by everyone? I don't know how easy or hard this would be, but I do think it would be cool to push some development in this area and it could really add some interesting challenges and features to our shared environments. — Curtis Olson (Jan 10th, 2011). Re: [Flightgear-devel] Towable Object Idea. (powered by Instant-Cquotes)

I've always thought that it would be really interesting to be able to inject "AI" type objects into the MP system. — Curtis Olson (Jan 10th, 2011). Re: [Flightgear-devel] Towable Object Idea. (powered by Instant-Cquotes)

I've been thinking about this idea for some time now; separate out the "intelligent part of the AI code" into a separate program, and run this as a dedicated standalone program that feeds aircraft positions into flightgear. This would solve many initialization problems, because the server could run completely independently of FlightGear itself. If somebody would care to assist in working out the details of the networking code, I'd be happy to investigate the possibilities of setting up a dedicated AI traffic server. — Durk Talsma (2007-11-06). Re: [Flightgear-devel] Programmatic control of AI traffic. (powered by Instant-Cquotes)

Partitioning rationale

• "[...]I think a more flexible approach may be "self-contained" modules communicating by passing "properties" over TCP. The "remote" FDM is already a possibility and there is an example of a remote joystick but how easy would it be to break up the rest of flightgear?"[4]

• "Instead of thinking of FG as a single threaded application, it needs to be a collection of standalone programs that run collaboratively - go back to thinking in terms of the external FDM option."[5]

• "I agree that the approach you suggest, of starting with a single subsystem/module is probably the best way to start tackling this issue."[6]

• "It'd be worth identifying which subsystems are the big time sinks (FDM? AItraffic?) to prioritise this." [7]

• "I think the main targets for parallelization are the rendering pipeline and various "add-on" systems, like the traffic manager."[8]

• "From a multi-player standpoint. Any additional "AI" aircraft to be seeded into the world, should be handled by the server. The server will create and delete them, the server will fly them, script them, etc. etc."[9]

• "This way, an instance of FlightGear needs to send the position of the aircraft it is responsible for to the server, and then it will receive the positions of any other aircraft (AI or human controlled) in the vicinity."[10]

• "Once that is fleshed out, we can worry about seeding additional AI aircraft into the world to make things more interesting. At that point we can decide how to fly them, script them, animate them, etc. But I think most of that should be handled on the server side.A single instance of flightgear would get the same information on other aircraft regardless of whether they are human or AI controlled."[11]

• "And I envision a "client" that handles multiple AI aircraft on behalf of a server thats plenty busy enuf handling message passing and other management functionality (this "client" really it could be considered part of the server, but so much of the code is the same compared to a client, there really isnt a reason not to leverage the existing client code and distribute the processing to other machines, and the same code will be in the server so if the requirements are light enough, the server could be instancing the planes)" [12]

• "my preference would be to control all the AI traffic in the server and not burden the core FlightGear code with it. Note that unless you are < 100' away from an airplane, there is no way you are going to be able to tell if it's running real flight dynamics, or some ultra-simple non-physics based engine that just moves the airplane along some predetermined route. 99.9999% of the time in civilian aviation, you should never get that close to an airplane."[13]

For additional details regarding partioning FlightGear into distinct modules that may be run as standalone processes doing IPC via the property tree, please also see Modularizing, parallelizing and distributing FlightGear.

Background information

2002

• "If one of the computers taking part in the multiplayer network has generated a bunch of AI aircraft, will they all be propagated to the rest of the multiplayer members ? If so, you might be able to dodge the processor load of full aircraft simulations, by having two computers with one having the human and a graphics display and the other having all the AI and no graphics display. Just a thought. "[14]

• "What I would like to see implimented is a 'standard' DCS system (where DCS stands for dyanamic coordinate system which is industry lingo for objects that move in the scene ... their local coordinate system is 'dyanamic'." [15]

• "I'm envisioning a DCS manager where you can register 'entities' with an associated 3d model and an associated 'behavior'. The behavior could be a preset path to follow, or something more AI-ish, or even something like JSBSim or YAsim (or perhaps entity positions could be driven externally from a server to produce a 'shared' world for collaborative flying.)" [16]

• "The DCS system would take care of loading and attaching the 3d models to the correct place in the scene graph and removing them. It would call the update() routine for each of their "engines". And it would probably provide some sort of property interface to the positions, orientations, and velocities of these dynamics entities." [17]

• "Indeed - it'll be nice to have a quick and easy way of getting other aircraft in the sky, however, I think from a long term point of view automated traffic would be best managed by simply being a task which appears as another "remote" user (yes, I know the multi user stuff isn't ready quite yet, but it strikes me as being the most "obvious" way to implement it." [18]

• "Still, however the ai aircraft eventually end up getting stored in the property tree and rendered, the actual logic of when to appear, where to fly and how to communicate and interact with other traffic will still be needed and won't be wasted." [19]

• "Yes - you still need the "pilot" logic however it's done. It certainly won't be wasted."[20]

2003

• "the protocol supports the idea of multiple aircraft sharing a single server connection for FG instances that are primarily handling a number of AI planes on behalf of a multiplayer scenario" [21]

• "just one instance of FlightGear feeding the server with the AI traffic (just like it was a client, but in this case a very special one). This seems like a good approach because the AI generator could run on one, or several different machine(s)." [22]

• "It seems to me like we're swimming in half-finished multiplayer/multiaircraft/network-data/network-interface implementations. What we need really isn't another one to plug into the mess, it's someone to *finish* the whole project" [23]

• "I think it would make sense to have the server [or a separate client instance] handle any non-human controlled vehicles. It would keep the load off the client which already has its hands full doing a full systems simulation as well as doing graphics work." [24]

• "FlightGear is an old code base, and lots of the old assumptions (like a single aircraft) need to be teased out of the code before progress can be made on new features. This kind of work isn't glamorous, and often requires more effort than the new development does. But it's not brain surgery either. The problem with some great new features is that they show up with code that is "ready" to integrate, but without the integration work done. So they languish in the Git tree until everyone forgets about them. " [25]

2004

• "Is there a way to create new instances of AIAircraft or another kind on the fly, just by adding some nodes in the property tree, or running a command from the telnet interface, that is, without modifying the source code ? Is there something planned in this direction ?"[26]

• "I have plans (but very little time unfortunately) to add a property to the scenario file that will trigger the creating of an AIAircraft after a specified elapsed time, or periodically at a specified rate. This will be an inprovement" [27]

• "The traffic scheduler creates the AIAircraft according to a schedule, so new instances of AIAircraft are created during the sim run, however I don't think this scheduler is set up to allow a user to trigger the creation of an AIAircraft using a key-stroke." [28]

• "To create AIAircraft interactively will require a new module, and it will require that the newly created instance be given enough information to have a model and a fightplan." [29]

• "Creating a traffic file is probably the closest you can get to creating AIAircraft instances dynamically during the simulation, but it is not possible at the moment to start traffic by interacting with the property tree. As David already explained, creating traffic is rather involved and probably requires new code. While I like the idea and can see some uses for it, I didn't really design the traffic manager with that idea in mind, and I don't really see how the code could be adapted to do this." [30]

• "The idea would be to control the AI traffics ( creation, animation and deletion ) from an external program [or Nasal scripting] with the telnet interface. Someone ask me if it is possible lately, and I think he is prepared to do the required changes. I already suggested him to design new commands for creation/deletion and use the properties for animation. They would be 'run' from the telnet prompt." [31]

• "Hmm, that's an interesting thought, because that was actually one of the possible mechanisms that I had in mind for the traffic manager. That is, have the schedules managed by an independent program that would communicate with FlightGear through a network layer."[32]

• "In the end I decided not to go into this direction after doing some tests and finding that the the easiest way to do it was by adding the traffic manager as a subsystem to the main FlightGear program, specifically because the overhead of managing the schedules turned out to be a quite low. Doing this through a network remains an interesting option though..." [33]

• "If you're going with the "independant program" route, then wouldn't it make sense to have the AI handled in that program - this way it could feed as many flightgear systems as you like, and they'll all have a consistent view of available traffic." [34]

• "If handled in this way there'd basically be no difference between an AI aircraft, and another instance of flightgear running over the network. It also presents some interesting opportunities for ATC when you have a consistent view of the world like this." [35]

• "Curt and I have agreed that we need some sort of DCS (Distributed Content System) that synchronizes instances of dynamic models between multiple running versions of FlightGear. That way a multi display setup of FlightGear will work with ATC/AIModels/AI Traffic (and MultiPlayer) code enabled." [36]

2007

• "Well I guess, a basic solution would be to have a master/slave mode for the AIModels subsystem, where the master computer does the "intelligent" AIModels work, and sends the resulting data across the network. The remaining computers, wouldn't have the AIModels system do anything, except process the incoming data." [37]

• "I don't think that this scenario would be impossible to implement, and has actually some attractive advantages (for example, the master AIModels computer could be configured to emphasize AImodels processing, at the cost of graphics display functions, it it were to be run on a system that was hooked up to the network, but not part of the display system)." [38]

• "Yeah! How about setting up an mp-server feeding in real traffic? Or even just AI? So that every client would see the same traffic."[39]

• "In order to feed in multiple aircraft that's probably the best approach - generate a multiplayer data stream for each aircraft you want to display - it'll benefit from the internal interpolation in the mp code too (I'm guessing you won't have 20 updates a second as would be fed from an instance of flightgear, so the interpolation will help immensely). " [40]

• "I've been thinking about this idea for some time now; separate out the "intelligent part of the AI code" into a separate program, and run this as a dedicated standalone program that feeds aircraft positions into flightgear. This would solve many initialization problems, because the server could run completely independently of FlightGear itself. " [41]

• "If somebody would care to assist in working out the details of the networking code, I'd be happy to investigate the possibilities of setting up a dedicated AI traffic server." [42]

• "I really think, setting up a mp-client which feeds traffic is the right way to go. That way it does not matter, if the traffic is real or artificial. And best of all, flightgear itself does not need any changes at all." [43]

• "While the feeded vehicle is in the air, there are no problems at all. But when approaching ground, or the vehicle is bound to the ground, there is one obstacle: The used terrain. In order to have the vehicle exactly on ground, the client must have the same terrain data which flightgear has. Therefor I thought of flightgear itself as the "feeder". If you somehow manage to get the terrain ripped of in some kind of library, it would be possible to build just any kind of feeder via the mp-protocoll." [44]

• "I fully agree. To FlightGear it would be visible as multiplayer aircraft, coming in over the network. There would be some special cases to consider, but the general infrastructure of the multiplayer server could serve as a template for an AI server as well." [45]

• "The traffic manager subsystem itself not so much as the AIModels system that eventually drives it. In this respect, there is still a lot of room for optimization. " [46]

• "A particular problem with the integrated solution is that both the traffic manager and the AI models code occasionally need peaks of CPU activity, in particular during flightplan creation. Although it would be possible to spread the load across multiple frames, doing so is a lot harder within flightgear than doing this as a separate program. " [47]

• "Assuming that AI aircraft behave in a reasonably sane fashion, you'd only need to take care of elevation points across the runways and taxiways. These data could be sampled straight from the flightgear scenery, and perhaps stored locally on the server machine."[48]

• "One thing that a traffic server would require is metar parsing, so that AI traffic would make sensible runway selections. " [49]

• "FWIW, I think that separating out as many FG subsystems as possible would be a very good way to go, especially in view of how CPU development is now solidly focussed on multiple core CPUs rather than ramping processing speeds, as was inevitable in the longer term." [50]

• "Re this particular subsystem, it might be worth also thinking about handling some of the weather stuff i.e. 3d clouds, storms and thermals, so these are also consistant over MP." [51]

• "There's no reason why it shouldn't run on the same box as FG - it could communicate via loopback or localhost. Ideally, FG would automatically start a local process to use if a net or lan server isn't specified." [52]

• "A "virtual" client (perhaps a thread) for each aircraft that connects to the multiplayer server as a computer-controlled "player". Then I would connect to the server myself and perhaps fly around manually so I can see the other aircraft?"[53]

• "Should not be necessary. All you do is to output the AI aircraft positions to all clients connected. The server itself fetches the data from the internet or whatever the source. No need to have a client for every AI plane."[54]

2008

• "This would also be a great way to help start factoring out the current AI Traffic code, so that it may run separately from flightgear, possibly even standalone directly on the machine hosting a server!" [55]
• "What's even more important: having such a capability would mean that we automatically end up with a very convenient high level tool to EASILY stress-test multiplayer servers and the underlying multiplayer code!! Of course, this capability should be optional and configurable to ensure that not arbitrary "multi-aircraft clients" can connect to a server and bring it to a breakdown by inserting thousands of traffic nodes. But if there is a way for authenticated/trusted clients to employ such functionality, it would be really awesome."[56]
• "Hopefully, once we start working on a new architecture, in which AI traffic is fed from a separate application into FlightGear, using the multiplayer system, hopefully much of that load will be offloaded from FlightGear. This could be run either on a separate computer, or using a network connection from the same machine..." [57]
• "refering to the idea of moving the core of the AI traffic code out of FlightGear itself, and communicate positions of other aircraft through the multiplayer system. We discussed this idea a few months ago, and IIRC, everybody seemed to respond fairly positive to the idea. " [58]
• "FG multiplayer can support multi aircraft to fly together,but that means I need twenty persons to control twenty FG instances.My question is that how can I modify FG to adapt multi aircraft to run in one instance and I can switch from one aircraft to another to control it.When I control one aircraft ,the others could move automatically." [59]
• "I think this quite doable using the AIModels module. What is needed is a way to reinitialize the user aircraft structure from the data of the FGAIAircraft. In essence this is online plane switching, which everyone wishes for but is not implemented yet. I can see some logical issues though switching to a MultiPlayer controlled aircraft, so this needs be sorted out." [60]
• "your requirements except plane switching can be done quite easily using existing FG code. Online plane changing is a long wanted feature" [61]
• "It seems like it should be possible to setup a number of AI aircraft andthen have pretty tight control over their behavior via the property system." [62]

More opinions

• "In the future we could also consider to have one server handling all AI objects for the clients to have a coherent environment. Imagine that you are training landing on the Nimitz with your friends but the ship is at a different position on each client. This would be very weird. "[63]
• "But what about if the MP server, in addition to mpdummy, had a dummy player for Nimitz (and perhaps Foch and Eisenhower as well), with the purpose of holding a stationery position for MP carrier users to use in common when there is no MP carrier playing?"
• "one way for ai-mp controlled vehicles would be an independent server that controls ai aircraft/ships/other for multiplayer."
• "The thing is that as far as I know we can only do stationary dummy players at this time."
• "in addition to this, AI aircraft would be able to react with the tower more easily I think, because the server would be controlling both, since the tower will be AI too... If there is an MP tower available, they replace the AI tower. (reminds me of the first idea) "
• "I dunno if this would lag us much, because it is gonna be the same as the original AI. Same models, same everything... except it's the server placing it, giving us the co-ordinates, instead of our own AI system giving it to us."
• "This makes it more realistic because all AI objects will at last be in sync.. like all objects are in sync to humans in real life."

Centralized AI traffic injection via multiplayer

• "As you want to do it as an external application anyway, maybe the best way is to use the multiplayer protocol" [64]
• "In a way you "fake" connected players on your customized server." [65]
• "I would think that the big benefit of having an MP client feeding traffic is that we'd only need one traffic client for all the people connected to MP around the world." [66]
• "Personally, I think that out-weighs the performance implications for those people not connected to MP who want AI traffic and who have to run a client locally." [67]
• "Having a separate client for traffic also has the benefit of allowing the user to prioritize my own flying over that of the AI traffic. I'd rather have the AI traffic stutter than my own aircraft." [68]