Modernizing FlightGear Scripting

From FlightGear wiki
Jump to navigation Jump to search
This article is a stub. You can help the wiki by expanding it.
WIP.png Work in progress
This article or section will be worked on in the upcoming hours or days.
See history for the latest developments.

Background

FlightGear has built-in scripting capabilities using a custom ECMAScript-based C-like language called Nasal. Over the last couple of years, we've seen some recurring debates in response to requests to provide support for additional scripting languages, possibly to replace Nasal scripting in its entirety sooner or later, i.e. "ditch" Nasal in favor of a more established, more mainstream, language like Python or Lua, with better tooling, support and documentation available for end-users, developers and other contributors.

Adding another language wouldn't be that hard. Actually, we had another one before nasal and beside nasal for a while. It was called PSL (plib scripting language), and we ripped it out because Nasal was/is just better and because offering and maintaining two languages it utterly pointless . [1]

Nasal is *very* well designed, compact, and efficient. It is used heavily throughout many areas of FlightGear. So most core developers can't imagine any scenario where we would switch to some new scripting language unless a lot key developers were convinced that it was every so much better that that benefit would substantially outweigh the cost. And that scenario is hard to imagine.[2]


This article is intended to help summarize some of the key points made by various contributors and to help provide a reference for people interested in the corresponding debates.

Motivation

Nasal is de-facto unmaintained and has been unmaintained for several years meanwhile, Nasal is a niche language that is really only used by the FlightGear project, long-standing issues such as its garbage collector, which isn't suitable for a semi-realtime application/simulation like FlightGear, haven't been fixed in years.

In general, there are other, much better maintained, options for embedded scripting these days, such as e.g.:

  • Lua
  • Python
  • Perl

Today, Javascript, Python and Lua have surpassed Nasal capabilities in pretty much everything the FlightGear community cares about: performance (garbage collection, jit etc), concurrency, parallelism, bindings, language ecosystem, and especially the developer base.[3]


In addition, documentation and end-user support for more established/mainstream languages is obviously much better than anything we have for the custom FlightGear/Nasal interpreter.

Many people argue that a more standard scripting language (like e.g. Python) would also attract more potential contributors to FlightGear, also the sizable community with 3rd party modules, means that new functionality can be much more easily implemented ("batteries included", and "no yak shaving" needed).

Thus, one of the key questions commonly posed is "How coupled are Nasal and the scripting glue in FGFS ? Is there a clean break, or if not, can it be refactored without too much pain into something that would allow end dev users to use whatever scripting language they prefer, or have many modules already writen in, etc. If the coupling is not of the hair pulling type, it might be conceivable to integrate another scripting language alongside Nasal for starters, and in time, completely replace it if one is so inclined "[4]

Considerations

  1. dependency on a huge number of external libs is considered a bad thing (TM) in FG, hence even if you convince people to implement language XY, you don't get to use all the libs in anything that goes to the repository
  2. hence you're left with finding an algorithm that does what you want it to, and implementing that algorithm is the thing that takes a long while, not the week spent learning how Python syntax differs from Nasal syntax
  3. but adding more scripting languages to FG means more complexity and more breaking points - so the benefit for whatever you add should outweigh the problems[5]

Challenges

The paradigm of binding a C/C++ core to a scripting language has its own disadvantages, and that makes it very difficult to switch.[6]

let's keep in mind that despite the current situation with Nasal, many core developers are already concerned that scripting is being used too widely - regardless of Nasal being de-facto under-documented, and hardly maintained - this situation would only get worse once/if a real mainstream language is supported. In summary, it's not like Nasal's syntax is esoteric in any way - it is far more standard than Fortran, Forth, LISP or Scheme - despite those being definitely "more mainstream" than Nasal - and it being close enough to C, C++ and GLSL, also means that people doing FlightGear scripting get a feeling for the syntax that will come in handy later, when looking at other, non-nasal, FlightGear source code. [7]


FlightGear can't just ditch Nasal. Bringing another scripting language into the fold would certainly be a big undertaking.[8]


It is really important to note that in a small application which is probably 1/10 or 1/100th the sie of FlightGear, this (supporting Python) was a truly massive undertaking to make these changes ... on the order of 2-3 months of pretty much non-stop effort. For FlightGear it would be almost insurmountable. I think it's an interesting topic to discuss in theory. We can point to examples and all the good things that python could bring to the table. However, we can't get around the fact that this could be a multi-year effort to do the conversion in a way that truly modernizes FlightGear and truly leverages all the good aspects of Python ... and during that 'under construction' period of a year or more, most of FlightGear would be broken. Edward came up with an alternative scheme where he built a python interface to the existing C++ property tree and some hooks to run python scripts, but still ... dropping nasal and converting to python requries porting AND carefully testing all the scripts though all their code paths and edge cases. Just that would be a massive undertaking, and it wouldn't be fair to just port the core nasal, and hang all the aircraft and scenery developers out to flap in the wind to figure it out for themselves.[9]

The balance is that Nasal is used a little in the core program, GUI, advanced weather, etc.. and a very lot in aircraft models. Nasal works well enough and removing it would be a massive undertaking. Although many people don't particularly like Nasal it is reasonably efficient and is very adapted to operations within FlightGear. Most contributors simply have yet to see a compelling reason to change; concurrency isn't even relevant until we have a properly threaded core, and garbage collection is not something commonly seen as a problem.

Removing Nasal will break a lot of aircraft models so it's not an option that can be seriously considered without a migration path. Some core developers would argue that there is overuse of Nasal aircraft side but it is what it is and we can't change that.

[10]

On replacing Nasal

As far as replacing nasal, that project would be very hard. There are many vital parts of flightgear currently coded in nasal. There are also random bits of nasal code scattered around in joystick configurations, instrument and aircraft models, scenery models... everywhere. It would, in effect, require starting the whole data project over again.[11]

The current situation

We have a single SGSubsystemMgr based FGNasalSys class inherited from SGSubsystem, which implements support for FlightGear scripting (add references/links below):

Implementation Strategy

Coming up with some kind of unifying platform: A modern launcher UI (maybe even as a webapp) which can launch FlightGear, configure certain stuff (during FG running) and arrange communications with third-party processes implemented in Python or other languages. These would then manage scenarios, AI, maybe even input devices and aircraft systems. Of course, this would need a better integration of remote protocols into FG. The user wouldn't notice anything of the hidden processes, because the launcher/manager takes care of everything, even the spawning and killing of the auxiliary processes.[12]


The http interface that Phi uses is where I'd see a unified API growing from; but someone's got to be interested enough to do this. The main problem with a unifying platform IME is that it sounds ground but really doesn't mean much. Concrete proposals and offers of development assistance are what's needed; but adding another scripting language into the run time isn't a great idea; if anything we need to be able to provide a fully featured API available over HTTP that can be called via any platform. Switching to use Phi for more is probably a good idea; and certainly exposing more of the API is going to permit more things to be done there; but a launcher webapp would obviously require a standalone HTTP server running all the time; anyone fancy writing a FlightGear apache module -)[13]

at the end of the day, the main bottleneck is not the language/interpreter supported by FlightGear, but the way it is integrated - and the legacy SGSubsystem/Mgr approach is simply not suitable for large-scale scripting, certainly not the kind of scripting that would definitely be boosted by supporting a more mainstream language like Lua or Python. Thus, there are things that need to be addressed first, which is why it makes sense to review Richard's postings in this thread, and explore using Torsten's mongoose/Phi approach, i.e. remote properties and async fgcommands to use those as the foundation for a scripting subsystem that runs outside the fgfs main loop.

[14]


There is work that lies ahead that would benefit FGPythonSys and FGNasalSys at the same time - such as a strong IPC mechanism, like HLA, or even just Emesary via remote properties (asynchronous remote properties and fgcommands, as per Torsten's mongoose/Phi work) - Torsten basically proved that you don't need to use HLA to come up with async modules that can interface with the rest of FG - if this, his, approach were to be formalied, standardized and extened, many other subsystems could be using this, none of which would be facing the challenges that Nasal/Python are currently facing (or in fact any other SGSubsystem not using SGthread, too).[15]

And I think that's really what Richard is hinting at when he refers to Phi, i.e. its back-end, not the front-end part - and the back-end could, and arguably should, be also shared/used by other UIs, including possibly PUI and/or a Canvas-based GUI. Unfortunately, for the time being, providing an alternative to Nasal, or even replacing it in its entirety would provide you withero benefits, due to the legacy FlightGear architecture, which would require major re-architecting to make this a worthwhile goal.[16]

FGPythonSys could benefit FGNasalSys and vice versa - but for most of this to happen, many implicit dependencies need to be detangled and cleaned up - which is why being able to entirely disable Nasal, and only init some bits of it, is a crucial part of the equation - even if you never again want to run a single line of Nasal code in your fgfs process, you need to come up with a way to still "boot" into a mostly functional fgfs environment.

[17]

For the time being, FGPythonSys remains the strongest option to explore supporting other scripting languages - but that should definitely not be using the SGSubsystem/Mgr and FGNasalSys approach, but instead be built on something along the lines of Richard's IPC comments regarding mongoose/Phi, otherwise all the people wanting python would immediately learn that all the Nasal folks were right about the bottleneck being elsewhere, and Nasal not being the culprit - just imagine for a second, FGPythonSys would be available right now, and provide what FGNasalSys provides, using the same implementation strategy: We'd see all sorts of 3rd party modules using all sorts of platform/OS specific modules, quite possibly with certain aircraft, instruments and scenery requiring different versions of certain modules to be installed.

[18]


we could definitely come up with a reimplementation of a subset of FGNasalSys and FGPythonSys to have an abstract FGScriptingSys that would be designed to be async from the get-go, and like Richard said, it sure seems like the Phi approach (remote properties and remote RPC via fgcommands) could be a straightforward to make that happen, without having to wait for the HLA/Qt5 folks to retire so that they can actually scratch off some items from their virtual todo lists.[19]

References

References
  1. Melchior FRANZ  (Feb 27th, 2009).  Re: [Flightgear-devel] Nasal alternatives : possible, of course, but trivial or hair pulling task ? .
  2. Curtis Olson  (Feb 27th, 2009).  Re: [Flightgear-devel] Nasal alternatives : possible, of course, but trivial or hair pulling task ? .
  3. Beaver  (Oct 6th, 2016).  Nasal must go .
  4. Nicolas Quijano  (Feb 26th, 2009).  [Flightgear-devel] Nasal alternatives : possible, of course, but trivial or hair pulling task ? .
  5. Thorsten  (Oct 8th, 2016).  Re: Nasal must go .
  6. Beaver  (Oct 7th, 2016).  Re: Nasal must go .
  7. Hooray  (Oct 8th, 2016).  Re: Nasal must go .
  8. Beaver  (Oct 6th, 2016).  Nasal must go .
  9. curt  (Oct 6th, 2016).  Re: Nasal must go .
  10. Richard  (Oct 6th, 2016).  Re: Nasal must go .
  11. Ron Jensen  (Feb 26th, 2009).  Re: [Flightgear-devel] Nasal alternatives : possible, of course, but trivial or hair pulling task ? .
  12. Beaver  (Oct 6th, 2016).  Nasal must go .
  13. Richard  (Oct 6th, 2016).  Re: Nasal must go .
  14. Hooray  (Oct 8th, 2016).  Re: Nasal must go .
  15. Hooray  (Oct 6th, 2016).  Re: Nasal must go .
  16. Hooray  (Oct 6th, 2016).  Re: Nasal must go .
  17. Hooray  (Oct 7th, 2016).  Re: Nasal must go .
  18. Hooray  (Oct 8th, 2016).  Re: Nasal must go .
  19. Hooray  (Oct 7th, 2016).  Re: Nasal must go .