FGPythonSys

From FlightGear wiki
Jump to navigation Jump to search
This article is a stub. You can help the wiki by expanding it. 
diff --git a/src/Main/options.cxx b/src/Main/options.cxxindex 20039d6..b48dbda 100644--- a/src/Main/options.cxx+++ b/src/Main/options.cxx@@ -231,6 +231,12 @@ void fgSetDefaults ()     v->setValueReadOnly("build-number", HUDSON_BUILD_NUMBER);     v->setValueReadOnly("build-id", HUDSON_BUILD_ID);     v->setValueReadOnly("hla-support", bool(FG_HAVE_HLA));+#ifdef HAVE_PYTHON+    v->setValueReadOnly("python-support", true);+#else+    v->setValueReadOnly("python-support", false);+#endif+ #if defined(FG_NIGHTLY)     v->setValueReadOnly("nightly-build", true); #else
Embedded Python support
Started in 01/2016
Description Python scripting support for FlightGear
Maintainer(s) bugman
Contributor(s) bugman (since 01/2016), www2 [1]
Status Under active development as of 01/2017

1rightarrow.png See Modernizing FlightGear Scripting for the main article about this subject.

The point of FGPythonSys is simply implementing something for fun - pretty much the same reason that most things in open source are done. I find that justifying its existence is a little pointless. Anyway, one interesting thing that could be designed would be a set of aircraft base classes in FGData implementing some basic functionality/instruments/systems, etc. Then the aircraft can use OO (object oriented) inheritance to expand on the base class in an aircraft specific way. Of course this could be done in Nasal, but Python's OO design is far more advanced, especially with GC (garbage collection). Not that such arguments are relevant ;) By the way, I'm thinking that aircraft/content Python would be a restricted subset of the full Python framework. Anyway, all of this is in the realm of fantasy unless people are willing to get their hands dirty and play with C++ code.[2]


What is it

1rightarrow.png See Python in FlightGear for the main article about this subject.

An experimental SGSubsystem integrating an embedded Python interpreter in FlightGear using the existing SGSubsystemMgr infrastructure (currently not merged with sg/fg next).

Cquote1.png I am interested in experimenting with the FGPythonSys concept to break a number of FlightGear's current limitations. This probably has significant overlap with Stuart's HLA/RTI developments. I would like to try to experiment with the concept of subsystem creation. Specifically to add support for creating three different categories of subsystem: Cquote2.png
Cquote1.png I am happy to use FGPythonSys as a guinea pig here, as I would really like it run in its own dedicated thread. This is a good system for quickly writing many tests for catching sub-thread and child process racing conditions (this links back to my CppUnit test suite ideas). I would like to also expose subsystem creation - using all 3 of the above mechanisms - via a new Python 'subsystem' module (written purely in C++, including the HLA/RTI parts). The forked child process + HLA/RTI concept has a particularly high value with Python, as the child process will then have it's own Python global interpreter lock (GIL), isolated and independent from the FGPythonSys GIL.
 Cquote2.png

Getting involved

Talking about building something awesome, do you have coding skills? If so, what is your opinion of code implementation via test suites? If you are interested in converting your dreams into any type of reality, then join in the thread at [1] This is a link to the FlightGear forum., git check out my FGPythonSys branches (into clones of the official repositories), and start implementing awesome new features [3]


On Linux, you will want to make sure that the python3-dev and cppunit packages are installed (e.g. sudo apt-get install python3-dev cppunit, you can then set up remote tracking branches for the sg/fg and fgdata repositories, e.g.: 

git remote add bugman URL
git fetch bugman
git checkout --track bugman/python/r...
Cquote1.png Just a quick update on the FGPythonSys experiment, I have push some new branches based on 2016.2.0: 
u-edauvergne-flightgear python/r6 branch

u-edauvergne-simgear python/r4 branch

u-edauvergne-fgdata python/r2 branch
I have shifted everything around so that these branches are based on my test suite branches (u-edauvergne-flightgear cppunit/r4 and u-edauvergne-simgear cppunit/r3). This means that if you install the dependency-free CppUnit package and run "make test_suite", you will see 100% of FGPythonSys tested. This is via 242 passing tests, and one deliberately failing yet temporary test. I recommend that any Python developments based off my branches take full advantage of this infrastructure - creating tests for this infrastructure is very easy - and that the test suite remains at a 100% pass rate.
 Cquote2.png
Cquote1.png there is a test aircraft - the py-ogeL, a copy of Torsten's ogeL with <python> tags and scripts added: https://sourceforge.net/u/edauvergne/code-py-ogel/ci/python/tree/ The py-ogel included test.py Python script is a complete test suite of the current state of the Python property tree
 Cquote2.png
Cquote1.png It would be awesome if you could fork the main flightgear repository to your personal SourceForge account, track my python/r3 branch, create a new branch for this change, and push the branch. This way your contributions will be far more valuable to the advancement of FGPythonSys, and we could work in a much better collaboration.
 Cquote2.png
Cquote1.png I look forward to seeing your private branches, and to start pulling them into mine :) If this works well, then maybe I can set up a FGPythonSys development team on SourceForge, with a bug tracker and other open source infrastructure set up. Be prepared for a lot of feedback, as my Python coding standards are very high ;) To have Python accepted, we should aim for very well commented, documented, and nicely formatted code! I suggest we aim at about 30% documentation/comments, as I did with the py-ogel test.py file. The standard for acceptance should be set much higher than if you code normally.
 Cquote2.png
Cquote1.png For implementing a minor feature, the best would be if you wrote a Python test like I did with the py-ogel. So for example with the navdb, write a single unit test that performs a single operation. If you could write a test_*() Python function for each individual feature you would like, that would be ideal! I hope to take all of these tests and integrate them into a CppUnit test suite for FlightGear. If we have unit tests for each small bit of functionality you'd like implemented (ideally a few tests for each), then FGPythonSys would very quickly become the most stable and reliable part of FG - by far.
 Cquote2.png

Vision

Cquote1.png I like the idea of being able to use Python directly in FG. As many others, I have a bad intuition concerning the security implications if this were open to general aircraft scripting (even though I would love using Python instead of Nasal for that)
— rominet (Feb 12th, 2016). Re: Announcing FFGo: a new FlightGear launcher.
(powered by Instant-Cquotes)
 Cquote2.png
Cquote1.png I believe that Python is one of the simplest languages to learn. And it regularly ranked in the top 5 programming languages. Therefore I think this is attractive to include as an option for students, researchers, and others interested in experimenting. This can give them a modular tool to leverage the lowest levels of FlightGear, an angle that Nasal currently does not, and has no intention to provide. And this all comes back to the original point the code was written for!
 Cquote2.png
Cquote1.png I would present this as a technology enabler, and one like no other. I envision implementing the following Python modules: Cquote2.png
Cquote1.png This Python API idea is to expose the lowest levels of the internal FlightGear C++ infrastructure, so that new FlightGear core code can be written in pure Python. One benefit that this experiment might lead to is to enable powerful code prototyping. A subsystem could be first written in Python, and then ported to C++. Or if the subsystem is fast enough, which in most situations will actually be the case, then it could remain in Python and be part of the FlightGear distribution (e.g. the code would be installed into /usr/lib/flightgear/python/, and not be part of FGData). By integrating Python into a subsystem as I have done, there is the benefit of very low latency and fast millisecond / kH code execution that higher level IPC mechanisms can never come close to matching (HLA/RTI, websocket, etc.). Therefore someone could write Python FG code that has the potential to run at a similar speed to core C++ code. The websocket mechanism also does not give access to the lowest levels of the infrastructure. So the greatest advantage is that extremely low-level core FlightGear code can be written in Python.
 Cquote2.png
Cquote1.png The code is very clean and isolated, and I am quite capable of expanding on this Python/C++ interface. I have interest in learning more about the internals of FlightGear, and adding a Python/C++ interface to each component in the process. I'm looking forward to some feedback and Python code from Jean-Paul to start working on a NavDataCache interface, to create a 'nav_data_cache' or 'navdb' Python C/C++ module.
 Cquote2.png

Aircraft Scripting

Cquote1.png As for using this as an aircraft scripting language, then obviously we would present just a subset of the core FG functionality. An aircraft does not need access to simgear. I'll get to the important security issues soon. And, yes, this could possibly all be done with Nasal, but that's not the point. The Python language offers a high-level object oriented design, which Nasal does not, and is one of the most widely used languages for code prototyping. It may also prove to be much better in dealing with garbage collection, as a large number of people have worked hard over many, many years to make this part of Python extremely efficient (this is required because of Python's GIL). I also have a very deep knowledge of Python (I still have a functional copy of Python 1.0.1, the first version I coded for, on my system). Therefore I can construct powerful Python/C API interfaces very quickly, something I cannot do with Nasal. Anyway, I see great benefit in experimenting and enabling such a low-level API. What I don't see is a real disadvantage with this idea, and why this is proving to be so controversial.
 Cquote2.png

Use Cases

This is just a collection of existing FlightGear related scripts (Python) code that may sooner or later benefit from being able to run within the FlightGear session:

FFGo launcher

Cquote1.png Contrary to FGo! up to 1.5.5, it requires Python 3.4 or later, not Python 2. Because of this and the new dependency on CondConfigParser, the software requirements are a bit different from those of FGo!
— rominet (Aug 4th, 2015). Announcing .
(powered by Instant-Cquotes)
 Cquote2.png


Cquote1.png The built-in launcher that James is writing would seem to me to be a good candidate for doing things in Python (assuming there are good bindings to the C++ data structures): you need high-level access to FG aircraft and airport data, it is not performance critical, the need for threading is certainly rather limited, the code does not come from a random hangar and can be audited just like the C++ code when committed, probably by more people actually. Add to this that PyQt works very well (I have had a very pleasant experience with PyQt 3&4 under Python 2&3, and if I get enough time, I'll maybe port FFGo to PyQt---Tk is useful, clearly, but frustrating; for example Qt's QAbstractItemModel and QAbstractItemView were great to manage non-trivial tablular data; with Tk, table/tree cells are just strings and your program has to manage on its own to link that to the nice high-level Python data structures).
— rominet (Feb 12th, 2016). Re: Announcing FFGo: a new FlightGear launcher.
(powered by Instant-Cquotes)
 Cquote2.png
Cquote1.png depending on how much progress bugman is going to make with his "python-integration-in-FlightGear" experiments (FGPythonSys), we may at some point even be able to ship FFGo as part of FGData (e.g. $FG_ROOT/Python/Apps/FFGo) and run it via FGPythonSys directly
— Hooray (Feb 12th, 2016). Re: Announcing FFGo: a new FlightGear launcher.
(powered by Instant-Cquotes)
 Cquote2.png
Cquote1.png One interesting thing that FGPythonSys could probably bring to FFGo would be, I think, FG-compliant parsing of aircraft data. For now, I haven't done that because I found that the interesting metadata (e.g., aircraft radius) is not necessarily in the -set file, but may be included from a different one. And maybe a correct parser for these things must also handle conditions in the XML data, etc. So, despite knowing the declared radius of parking positions, I haven't provided any hint yet in comparison to the selected aircraft's radius for these reasons (and lack of time).
— rominet (Feb 12th, 2016). Re: Announcing FFGo: a new FlightGear launcher.
(powered by Instant-Cquotes)
 Cquote2.png

TerraGear GUI

It's worth pointing out, there's plenty of people who would like to work on scenery, but find the setup and import / processing / export steps very tricky - especially if they've no experience with GRASS (or QGIS). Any scripts or workflow documentation that knowledgeable people make available is a huge benefit. (I also think there's probably some easy hacks that could be made to the terra-gear tools to simplify the commands and file structure, but that's a discussion for another time)

Cquote1.png If I were to start writing terragear today, I would definitely start down that path in python. Obviously we can't do anything too intensive per-frame in a scripting language.
— curt (Dec 28th, 2015). Re: FGPython an propose for .
(powered by Instant-Cquotes)
 Cquote2.png

OSM2City

See also Project3000.
Cquote1.png could we run a C++ version of the script in real-time? As an alternative / replacement for random-buildings. This relies on two assumptions: that the input data per tile is sensible sied (tens of kbytes, or worst case a hundred for centre of really built up city), and that the script could run in 'almost real time' on the OSG Pager thread. So we'd add the outline data as an additional file in each Objects/foo/foo/dir, and if this mode is enabled, look for the file and run the buildings creation script at that time.
— zakalawe (Oct 23rd, 2013). Re: OSM buidings EHLE.
(powered by Instant-Cquotes)
 Cquote2.png

Development

Testing

Edward d'Auvergne has been implementing a test suite infrastructure using CppUnit for flightgear itself (not simgear) as part of his FGPythonSys experiments [1], and noticed that modularisation is problematic due to the dependency of globals and everything that then brings in. I wonder if coming up with a way of breaking the dependency on globals.hxx - passing the data structure (or a mimic in the case of a functional test) into the subsystem using the new subsystem infrastructure, using property subtrees and synchronisation, etc. would allow the subsystems to be truly independent. [4]

Edward developed this as part of his FGPythonSys experiments to have full system test coverage [1]. The test suite is based on the venerable, yet old CppUnit framework [2]. Edward d'Auvergne has spun this out into its own standalone branches rebased to 2016.4.0 and made independent of embedded Python [3]. The test suite is an almost direct Python->C++ translation my own comprehensive test suite for one of my own software projects [4]. The benefits of such a framework, assuming a good test coverage, include: - Enabling quick and brutal refactorisation of the entire code base - ensuring that the end result works perfectly well. - Interest in independence and modularisation of all FlightGear components to simplify tests. - If all subsystems are tested for standalone creation, destruction, init(), reinit(), shutdown(), etc., then repetitive resetting of FlightGear should work flawlessly. - Allow valgrind and other heavy tools to be run on the absolute minimal code paths, by running individual tests. The functionality of the test suite can be very much expanded in the future [5]. For example there are CppUnit classes for XML output, rather than text, so this test suite is ideal for automated testing and presenting the results through web pages (e.g. Jenkins).[5]

Reset/re-init

Cquote1.png Once FGPythonSys is able to dynamically shutdown/restart using the corresponding fgcommands, and once it can be entirely disabled during startup, it would already be in a much better shape than 99% of the existing SGSubsystems in FlightGear
 Cquote2.png

Subsystems

Cquote1.png to make that work, you would need to register a new extension function along the lines of "registerSubsystem()" and "unregisterSubsystem()", which takes a Python class as ist argument that inherits from the equivalent of the SGSubsystem/Mgr parent class, i.e. with init(), update(), reinit() etc methods The registerSubsystem() API would then look pretty much like what you have added to fg_init.cxx, but you would need to manually/expicitly initialie such classes, because intialization is likely to have completed already by then.
 Cquote2.png

Understanding Nasal APIs

Cquote1.png I can construct powerful Python/C API interfaces very quickly, something I cannot do with Nasal.
 Cquote2.png

The Nasal C APIs are pretty straightforward actually (especially for people familiar with C and embedded scripting), and it may make sense to look at the corresponding docs - even if just to learn how to port existing Nasal APIs to Python:

Status

the focus of the efforts in my experimental FGPythonSys branches This is a link to the FlightGear forum. is more to provide a Python interface to core FlightGear programming rather than replace Nasal. For example one aim is to convert the whole of the SimGear programming interface (API) into a Python library. Another is to expose the subsystem interface to allow for Python modules to create new FlightGear subsystems. As a biarre aside, if FGPythonSys was designed to replace Nasal, it would also be able to fully replace the XML parts of aircraft as well. In any case, I have been a bit busy with RL and there is no one helping in the task of advancing FGPythonSys. So at this rate, replacing Nasal with Python will only be a pipe dream for a few content developers.[6]

Here is another update for the FGPythonSys code. I have rebased the branches on the current 2016.4.0 flightgear/simgear/fgdata branches. Not only that, but the branches now work on MS Windows. As these worked on GNU/Linux previously, and Mac OS X should just work, the branches should now be functional on all operating systems! Note though that only the CppUnit code needed updating to work on MS Windows, the Python code worked without modification. The branches are:

As before, these are based on the CppUnit test suite branches:


Cquote1.png I'm also battling with VS 2015 (msvc14) on a VirtualBox VM with Windows 7. I need to have this system up and running, because Windows support for FGPythonSys is essential.
— bugman (Apr 22nd, 2016). Re: FGPython an propose for .
(powered by Instant-Cquotes)
 Cquote2.png
Cquote1.png Just a quick update, now that I have a test suite up and running (I'll post full details on the devel mailing list), I'm now working on a 'fg_sys' Python module to supply 'sys' module equivalent functions. The initial aim of this basic infrastructure is to replace sys.stdout and sys.stderr so that this works through the SG_LOG() system, allowing all printouts and feedback to go through the normal FlightGear/SimGear IO channels.
 Cquote2.png

https://sourceforge.net/u/edauvergne/flightgear/ci/521679677be33afa61860abbc1cdb17a50468267/log/?path=

Motivation

Cquote1.png The difference in speed between Python and C/C++ for complex operations is not that great. For more than 90% of the code in software, the language speed is not an issue. When it is, you write small Python modules in C to do the work. That could be done here, or the prototyping code all converted to C++. Anyway, orders of magnitude more important than the language speed differences is the design and algorithm.
 Cquote2.png
Cquote1.png a problem I'd like to solve for the FGPythonSys experiment. And that is to spawn independent subsystems from within Python. This itself has two purposes - to provide a Python interface to enable core code to be written in Python (strictly disabled for content developers), and to allow for a new, thread-safe mechanism to perform aircraft and other updates via Python.
 Cquote2.png
Cquote1.png With the new FGPythonSys, I could help work on making core infrastructure thread-safe right now without affecting the operation of the rest of the program. Placing FGPythonSys in its own subsystem thread would be the ultimate tool for hardening the core - it would highlight all the points in FlightGear that are prone to racing and locking. For example the global event manager and timer system with function call-backs is probably not thread-safe. And the property tree will clearly race all over the place. If there is interest, I can use these experiments to make these systems thread-aware and thread-safe, while maintaining full compatibility with all the other non-threaded subsystems. This might help eventually shift parts of Nasal, such as it's garbage collector, outside of the main thread (something that HLA/RTI does not appear to be suitable for).
 Cquote2.png

Nasal Issues

Cquote1.png the problem is that Nasal, despite being a high-level scripting language, may still be too low-level for most people - i.e. because resources like timers and listeners will typically need to be managed explicitly. Likewise, support for reset/re-init needs to be manually implemented.

And even complete "subsystems" are implemented on top of concepts like timers and listeners instead of having a more rigid framework-structure that encourages and requires a more conventional design. We do have some fairly "large" systems implemented entirely in scripting space, that still only use a subset of the language's features - often in a copy&paste fashion. And this is causing problems in other places. So I am not sure if we really need a proper OOP framework or even a DSL, but some way to encourage and enforce "best practices" in order not to break other simulator features would be pretty crucial sooner or later. Things that would need to be covered would probably include:

  • logging/debugging (printlog/die)
  • providing a subsystem interface (SGSubsystem)
  • supporting reset/re-init
  • saving/loading/resuming flights
  • resource management (timers, listeners - and possibly threads)
  • split-frame loop management
  • support suspending/reloading code from disk (RAD)
  • formalie multi-instance synchronization, no matter if it's between multiple MP instances or a multi-instance multi-screen setup: we need a way to sync/replicate certain state across Nasal-based subsystems to provide a consistent experience


Currently, many of these things need to be separately implemented by people, and are typically not at all supported by scripting space subsystems. But currently we're seeing an increasing number of subsystems implemented in scripting space, and even most C++ code isn't prepared to handle these requirements properly. But the manpower is currently not in core development, but in base package/middleware development.
— Hooray (May 10th, 2014). Re: .
(powered by Instant-Cquotes)
 Cquote2.png
Cquote1.png timers are not all that bad given the design constraints of the FlightGear main loop - there are plenty other issues even apart from timers - and you really only need to look at comments from those core developers more familiar with Nasal internals to get a pretty complete picture - for instance, refer to the core developer comments quoted at: Initializing Nasal early

In summary, the main challenges Nasal is facing in FlightGear (and where it is causing problems in FG) are these:

  • executed in the main loop, so that GC overhead affects frame rate
  • it is added/initialized way too late, so that it is not available for certain purposes (think scripted initialization, benchmarking etc)
  • it cannot be tested in isolation, and also cannot be easily disabled completely
  • there is the hard-coded assumption that the interpreter sees a "full" fgfs session, including bindings for all subsystems, which is incompatible with the ongoing reset/re-init work, but also with the add-subsystem/remove-subsystem fgcommands
  • at run-time, it is next to impossible to clearly separate scripts based on their scope/purpose, i.e. aircraft scripts should not affect scenery/core scripts and vice versa - imagine selectively disabling certain scripts and restarting them (think firefox task manager showing JavaScript callbacks)
  • there is the hard-coded assumption that there will only ever be a single Nasal interpreter active at all times
  • equally, it is not easily possible to look behind the scenes to tell which scripts are "heavy" in terms of CPU/RAM utilization or "GC pressure" - we also cannot easily disable scripts entirely (programmatically)
  • there is no proper dependency resolution - neither in scripting space, nor in C/C++ space (Nasal bindings that depend on C++ subsystems being around)
  • no concept of "run-levels" (think headless, core, scenery, fdm, aircraft etc), so that the globals namespace is a huge mess
  • so far, timers and listeners are the only "building blocks" (hacks) to integrate scripted code with the main loop, which inevitably violate the encapsulation that SGSubsystem/SGSubsystemMgr could provide - equally, many virtual methods are not even implemented for important simulator events (pause, resume, save/load etc)
  • lack of generic building blocks (think abstract base classes/interfaces) to help enable/ensure that contributions are sufficiently general in nature, i.e. are not specific to a single aircraft/instrument, GUI dialog or use-case, and which support multiple independent instances "by design".
  • poor formalization of crucial simulator events at the SGSubsystem level (booting, pausing, resuming, saving/loading), which applies even moreso to Nasal code
  • no tracking of VM overhead in terms of Nasal footprint for different code, so that most people unfamiliar with SG/FG internals are basically unable to troubleshoot issues that might be related to Nasal
  • existing C++ code in SG/FG that is not properly exposed to scripting space, so that people re-implement FDM/AP logic in scripting space (for example)
  • where existing C++ code can be dynamically toggled on/off or customized, it can only deal with a single instance of the class and not allocate new objects on demand (think having more than a single FDM/AP object "live" at the same time) - that is a restriction due to the way the subsystemFactory works
  • Nasal's threading support is extremely rudimentary, i.e. lacking concurrency primitives that make "tasking" more straightforward for less experienced contributors (think design patterns), so that race conditions would inevitably occur sooner or later, even without looking at the FG side of things.
 Cquote2.png


Cquote1.png There is a lot of Nasal/Nasal GC bashing going on, but in most cases we've seen so far, it's really improper use of lower-level APIs like timers and listeners that are not properly managed by aircraft developers - which is unlikely to show up on powerful systems, unless you keep FG running for a long time - because the only thing that is really happening is that callbacks (=code) is triggered more often than necessary.

Imagine it like configuring your mobile phone to check your eMail inbox once per hour - in FlightGear, there is no notion of a "scheduler" to do this correctly, instead "timers" and "listeners" are used to associate callbacks (=code routines) with certain events, such as a timer expiring (e.g. after a few seconds) or a certain property changing (or any combination of these, i.e. by triggering listeners to be registered when a timer expires and vice versa). What is happening behind the scenes is a bit obscure, but that has nothing to do with Nasal or its garbage collection (GC) scheme, it has more to do with the low-level nature of the functions provided by core developers to "manage" such callbacks, i.e. the settimer() and setlistener() APIs are particularly tedious to manage properly - but the de facto practice is to use those as the main building blocks to write/integrate full subsystems into the FlightGear main loop - tiny coding errors may not have much of an effect, but under certain circumstances, those coding errors will add up (i.e. over time), so that the original "task" of checking your inbox once per hour, ends up being executed hundreds of times per second - the underlying code would still be correct though, it's just the event handling code that is not written correctly, which is often the case when using reset/re-init or when changing locations - because the simulator hasn't been designed, developed and maintained with these requirements in mind. Furthermore, there's not introspection facility provided that would allow people to look behind the scenes of the simulator to understand just how often a certain timer/listener is registered and triggered, so that the whole thing is extremely obscure - but like I said, this is not a problem inherent to Nasal coding, it also happens/happened at the C++ level - e.g. look at Torsten's fix for the effects code, which was also leaking/re-running listeners like cray (unnecessarily) - thus, it's redundant work that is getting re-scheduled to over and over again due to buggy code - the problem is not necessarily the buggy code though (writing broken code is part of the whole process), the real problem is that we don't provide any functionality that would allow people to look at where their resources (CPU, RAM, VRAM, GPU) are utilized, so that uninformed conclusions are drawn by some people (like "Nasal is bad ... and needs to be replaced with .... "). Thus, what you can do is to overload the corresponding APIs (extension functions) and treat the id() of the callback as the key for a hash lookup to gather data on how many timers/listeners are registered, and sample those over time.

You will almost certainly end up seeing dozens of unnecessary callbacks being invoked by certain code - even though that may not necessarily be restricted to Nasal code, like I said.
— Hooray (Jan 31st, 2016). Re: 777 freezes and FPS loss.
(powered by Instant-Cquotes)
 Cquote2.png

Goals

Threaded Subysystems

Cquote1.png I was hoping to use real threads for running FGPythonSys outside of the main thread, and child process forking and HLA/RTI communication on the C++ side for the aircraft subsystem. This would have given FGPythonSys and the aircraft their own embedded Python instance with GIL, so that their operation would be fully independent. The design would have really stress-tested FlightGear against racing conditions.
 Cquote2.png
Cquote1.png I also had the idea of running this subsystem in a thread. This avoids adding more strain to the main loop, making FGPythonSys even more inconspicuous. And it has downstream benefits, in that I'll be hitting many of the problems often discussed on this mailing list, and hitting them very hard. Asynchronous access to the navdb cache is one such problem. Racing is another - some parts of FlightGear uses SGThreads (such as parts of the sound system). With Python pushing on the code paths prone to racing, with easy to write tests to put huge amounts of strain on the system, I could help improve these parts. I see this as beneficial for the project, even if the Python initiative goes nowhere.
 Cquote2.png

Benefits for Nasal

Cquote1.png The developments I was hoping on working on in FGPythonSys could easily be ported to FGNasalSys. I am just not very familiar with the Nasal internals. For example I'll take back my statement about Nasal not being object oriented (thanks to the anonymous messenger for pointing out that blatant mistake). So from my perspective, I can personally bring developments to the table that could have beneficial side-effects. For example by helping Stuart with HLA and the property tree - which must change for HLA to be viable. Or advancing the subsystem infrastructure to support main thread, true threading and HLA modes. Putting FGPythonSys up as the first volunteer ensures that it experiences all the initial painful racing conditions, taking one for the team. I can only practically develop this via embedded Python, as that is my current expertise. But the developments could in the end be beneficial to Nasal as well.
 Cquote2.png

Message Passing APIs

Cquote1.png For example, it would be possibly to deprecate virtually all the custom Nasal APIs in favour of commands, but much harder to do that solely using properties. (Think about searching the NavDB or appending a waypoint to a flightplan). If we started moving in that direction, we don’t need to create separate APIs for multiple languages, they simply work using the properties + commands system, and the command system already has the exact set of property modification operations you proposed. I’m not set upon my message passing idea at all, but most solutions that don’t involve fine-grained locking of each property ultimately equate to message passing /anyway/; you have to queue up a sequence of changes to the property tree state, apply them in order and make the results visible. Given that, and the fact we already have the command system, my proposal is simply one way (and I think, a pragmatic one) to create such a setup using existing pieces available.
 Cquote2.png

Concerns and Issues

Dependencies

Cquote1.png Many posters were chiefly enthusiastic about Python because of the many libraries that can be used. However, I'd much like to avoid a situation like 'If you fly this plane, you in addition have to install Python libs X and Y, if you fly that plane it requires lib Z - oh, and that custom scenery uses lib Q for animation of jetways,...' - I trust you see the picture.
 Cquote2.png
Cquote1.png Like Thorsten I'm not keen in this. It would be a nice addition only if it was never used anywhere within the distribution. (FlightGear itself, FGData, FGAddon). I see some value in it for side projects but not for end users.
 Cquote2.png
Cquote1.png For one it add another huge dependency. Also, like Thorsten stated, there is a real possibility that a particular aircraft will end up requiring some obscure python module. If you add Python support by default you will see people using it for their aircraft.
 Cquote2.png
Cquote1.png I am happy with N different /optional/ languages being supported, but ideally through an RPC mechanism which is language neutral, hence my email about message-passing designs. I feel pretty strongly we do not want multiple /mandatory/ languages for core features of the simulator. In Qt we had a webkit dependency which required Ruby, Perl *and* Python to compile on Windows and the backlash from the developer / user community was extremely large.
 Cquote2.png

Security

Cquote1.png As discussed at https://wiki.python.org/moin/SandboxedPython , it's hard to embed Python without giving the scripts it runs full access to your system; hence, I wouldn't make this available to aircraft/scenery. (Nasal avoids this security problem by having its I/O functions only allow access to a limited range of files.) It could still be good to have available for local experimentation, but should be clearly labelled as insecure.
 Cquote2.png
Cquote1.png The security issue is that I expect FlightGear aircraft and scenery to be "content", i.e. safe to use even if I don't trust their authors with all my files, not "executables" (such as standalone-Python scripts), i.e. unrestricted so only to be installed from trusted sources. I agree it would be access as the FlightGear user and not as root, but that's already enough for the common home-user-targeted forms of malware. (And it might be remotely exploitable: last time I looked, Terrasync was un-authenticated, which is fine for a content-delivery channel but means it shouldn't be used for executables.) Inkscape, Gimp, etc only expose their scripting interface to plugins, not image files (i.e. the equivalent of giving FlightGear a Python interface but not allowing aircraft to use it, which I don't object to); Blender has an option to allow scripts in model files, but it is off by default (https://www.blender.org/manual/advanced/scripting/python/security.html).
 Cquote2.png
Cquote1.png I have toyed with the idea of embedding Python briefly in 2010. It can certainly be useful for a side project based on Flightgear, but James here has very good points and also Rebecca is right about security. Access to /home is a bad idea: it does not even have to be malicious, remeber the Steam bug which wiped the entire home directory a couple of years ago?
 Cquote2.png
Cquote1.png Python has too much introspection for such wrappers to work: open.__self__.real_open breaks it. (They never really worked in Nasal either (the old io.nas), which is why Nasal security is now done C++-side.)
 Cquote2.png
Cquote1.png I'd say that bringing in the Python toy is potentially harmful for the reaons already stated.
 Cquote2.png
Cquote1.png The only issue mentioned so far is trojan pegasuses. Specifically when FGPythonSys is used for content written by a user with malicious intent. This is also the only 'security' issue I can see as well, at least for now. This is a major issue.
 Cquote2.png

Main Loop Impact

Cquote1.png I like Python as a language and use it all the time, but in an embedded context it is quite large and the GIL makes using multiple Python instances on parallel threads tricky.
 Cquote2.png
Cquote1.png I wonder why a C++/subsystem integration of Python would be necessary. I am successfully using JavaScript via nodejs for a private project with FlightGear. Communication with FGFS runs via websocket and the http/fgcommand interface. I don't miss a single feature and everything works as one would expect. I would assume, the same technique would be possible with Python or any other scripting language that supports websockets, http-get, http-post and has some support for JSON (python has, ruby has, lua has). That being said, I don't see a benefit having a binary integration of Python in the flightgear core. Using python to get/set properties and run fg-commands is already possible without any c++ integration.
 Cquote2.png
Cquote1.png that Python integration is exactly the opposite way of what I would like the fg core to develop. Instead of adding just-another-feature we need to strip it down to getting a fast and constand fps rendering engine. Everything else needs to run outside the main loop and has to interact with the core by, say HLA/RTI or whatever IPC we have.
 Cquote2.png
Cquote1.png My major concern with your Python integration is that it introduces an unpredictable (w.r.t. timing) step in the main loop getting us on step further away from constant frame rates.
 Cquote2.png
Cquote1.png What scares me is that we could end up with lots of code in a scripting language that is essentially a performance dead end. And this when FG already is quite CPU bound and barely uses more than 1 CPU core. If Nasal should be replaced or augmented, then at least do it with a language and runtime that brings definite performance improvements and a multi threaded future.
 Cquote2.png

Impact on Nasal

Cquote1.png a different scripting language seems to be just that - a different language capable of doing the same things we can do right now in a different syntax. And in my view, that's not worth so much trouble. So maintaining a compile-time interface that is off by default for, say, research applications or people who want to use this for their own experiments sounds like a good idea to me. A situation in which different aircraft run on different scripting languages however does not.
 Cquote2.png
Cquote1.png Nasal has to be about the most straightforward scripting language I've ever had to deal with (including python, which I'd used before Nasal.) The fact that it's not really used outside of FG is completely irrelevant - the syntax is completely straightforward, there are thousands of example scripts to draw on and even if there are comparatively fewer people who are "experts" in it, every one of these virtually without exception will be knowledgeable about its use within FG. What use are a hundred thousand Python experts if they don't understand your problem? As others have already said, there are already many ways of interacting with the property tree with other languages; does anyone truly think that extending the actual core of FG in python is a good idea in any way?
 Cquote2.png

Deprecating Nasal?

Cquote1.png I also feel, wearing my ‘computer science’ hat, that a Nasal -> anything translator can be made robustly; we have an AST builder for the language already, the language is very regular, its types map to other languages directly, and we have the entire relevant corpus of code sitting in fgdata/ already; it’s straightforward to use that body of code to validate the translation. Hence my statement that we /could/ change the mandatory language, but we need to show some large benefits, and a migration path where we have both languages coexisting, and then prove we can machine translate, and then deprecate the old language, and finally remove support. What I don’t want is a situation where the Bombable script uses Lua, Advanced Weather uses Nasal, AI traffic uses Python, some particular aircraft uses Ruby and so on - that would be a disaster. As Thorsten said, Nasal isn’t great but we have it, it’s very small footprint and we have control over it, so we do need a hugely clear benefit to transition. If Python had trivial multi-thread safe interpreters with non-blocking garbage collection, that would be almost sufficient in my mind, but frustratingly it doesn’t.
 Cquote2.png
Cquote1.png I'm predicting you can't do it on the first go, but given enough effort, I suspect you can get there. So then - how does it look in practice once Nasal is deprecated? Right now I have 10.000 lines of AW which I know - with my comments in, with my formating conventions. I know where stuff is and can fix things quickly After your translator runs over it, I have 10.000 lines without my formating conventions in a language I'm not really familiar with. It runs - but my maintenance load has gone up by a factor 10. So it becomes essentially a froen feature. Unless I continue writing in Nasal and we just run the translator script every time something changes. So then I can maintain it - but everyone else who doesn't have the Nasal master file gets to see the machine-compiled Python version and has to dig into that... I don't know, doesn't sound too much fun to me either...
 Cquote2.png
Cquote1.png porting several hundred thousand lines of Nasal to something else, even if it is automated to a high degree, doesn't sound very attractive..
 Cquote2.png

High Level Architecture (HLA)

1rightarrow.png See High-Level Architecture for the main article about this subject.

Cquote1.png HLA is important here but not at the property level.
 Cquote2.png
Cquote1.png HLA is really intended to operate at a much higher level of granularity and pass objects around a distributed simulation. The use-case I'm working on at the moment passes an AI instance as an object (with position, velocity, accelerations, model to be displayed). How a HLA Federate maps that data internally is an implementation decision. There shouldn't be any race conditions because no assumptions are made about how the data is used or mapped. At present, I'm not thinking that HLA will be used as a mechanism to provide a shared property tree across multiple Federates. I think that's the wrong way to use HLA. Instead you make explicit decisions about the data models you will communicate. The other reason HLA isn't a good choice is to do with simulation clocking. I'm still getting my head around this properly, but the overall model is one where the simulation clock is advanced, and the instance the updates and publishes changes to the RTI that will be picked up by other instances at the next clock advance. (I think - I've still to work on a use-case where there is significant inter-dependence between instances).
 Cquote2.png

Python Federates

Cquote1.png even if Python isn't integrated with the FG core, there is plenty of opportunitie to use it within a FG-based simulation.
 Cquote2.png

Weather Simulation

Cquote1.png The weather module. That needs to provide wind and atmosphereic parameters for all simulation entities.
— Mathias Fröhlich (Jun 25th, 2011). Re: [Flightgear-devel] FlightGear Manager....
(powered by Instant-Cquotes)
 Cquote2.png
Cquote1.png A weather module running in a different process/thread/machine that computes positions for clouds that are consistenly displayed on each attached viewer. That being a module that is exchangable. The simple version just interploates metars like today, but more sphisticated versions might do a local weather simulation to get good results for thermals in some small area.
— Mathias Fröhlich (Jun 20th, 2012). Re: [Flightgear-devel] Slow frame rates.
(powered by Instant-Cquotes)
 Cquote2.png
Cquote1.png In the future, I hope to have a completely independent weather module using the HLA stuff that runs in an own process/thread. So, at that time you might be able to do more sophisticated stuff. May be it will then be possible to do a full cfd for subparts of the scene. That might be a good thing for a glider scene where you might want to have a more detailed fluid behaviour ...
— Mathias Fröhlich (Aug 2nd, 2011). Re: [Flightgear-devel] Future Weather System.
(powered by Instant-Cquotes)
 Cquote2.png
Cquote1.png I hope we get Mathias' HLA interface productive soon, so we can add a detailed weather simulation outside the main loop...
— Torsten (Dec 16th, 2011). Re: NOAA GFS .
(powered by Instant-Cquotes)
 Cquote2.png
Cquote1.png A shared Weather federate publishing both the environment for individual aircraft within the simulation, but also cloud positions so all aircraft had a consistent weather environment.
 Cquote2.png
Cquote1.png I'm hopeful that the work I'm doing on HLA will allow us to run Nasal in a separate thread from the FDM and display, so Nasal GC no-longer can impact frame-rates. It would also allow for writing a weather simulation completely external to the FlightGear instance, which could be quite neat.
— stuart (Nov 24th, 2015). Re: the real cost of the Nasal Garbage Collector.
(powered by Instant-Cquotes)
 Cquote2.png
Cquote1.png HLA being that different tasks are run by separate programs which all talk to a central controller unit... that will make the GUI interface (not the menu as some are want to call it) rendering one separate module while all the other stuff like AI craft and weather and such are in other modules which all talk to a central module... think of it like a central server with various clients getting and giving information while the GUI interface simply renders what it is told is available to render...
— wkitty42 (Jan 2nd, 2016). Re: 3.7 no smoke.
(powered by Instant-Cquotes)
 Cquote2.png
Cquote1.png Some of the stuff that I miss in nasal and i can easy do in python is adding an WebSocket client for example to connect to blitortung for real time lighting data. Or parsing Global Forecast System data for more realistic weather simulator using pygrib. (https://github.com/jswhit/pygrib)
— www2 (Dec 28th, 2015). Re: FGPython an propose for .
(powered by Instant-Cquotes)
 Cquote2.png

References

References
Retrieved from "https://wiki.flightgear.org/w/index.php?title=FGPythonSys&oldid=118867"