20,741
edits
Modernizing FlightGear Scripting: Difference between revisions
Modernizing FlightGear Scripting (view source)
Revision as of 16:41, 22 October 2016
, 22 October 2016→The current situation
| (8 intermediate revisions by the same user not shown) | |||
| Line 31: | Line 31: | ||
|date = Mar 5th, 2009 | |date = Mar 5th, 2009 | ||
|added = Mar 5th, 2009 | |added = Mar 5th, 2009 | ||
|script_version = 0.40 | |||
}}</ref> | |||
thinking purely on language syntax + interpreter engine footprint + seamless integration with flightgear + language capability/completeness: nasal may still be the overall winner. Nasal is brilliant in a number of ways. But to be fair, those are not the only considerations. Don't be too quick to discount the community of knowledge, the documentation, the library support, ease of installing new libs with new functionality ... there are many reasons to consider a choice. | |||
For example: I might take a slightly longer way home from work so I don't have to take a left turn at an uncontrolled intersection during a busy time at rush hour. Technically the first route is shorter, maybe even on average quicker, but perhaps I also consider stress levels and safety as part of the overall equation. That isn't a direct analogy to nasal vs. python, just pointing out that a person's final decision may weigh additional more subjective factors, and that a normal thing that we humans do.<ref>{{cite web | |||
|url = https://forum.flightgear.org/viewtopic.php?p=296312#p296312 | |||
|title = <nowiki> Re: Nasal must go </nowiki> | |||
|author = <nowiki> curt </nowiki> | |||
|date = Oct 8th, 2016 | |||
|added = Oct 8th, 2016 | |||
|script_version = 0.40 | |script_version = 0.40 | ||
}}</ref> | }}</ref> | ||
| Line 78: | Line 88: | ||
== Motivation == | == Motivation == | ||
It is also a question of resource optimiation. Mainstream scripting languages have lots of resources online, call it libraries, documentation, or best practices / know how in general. Nasal is naked in this regard. Any improvement to Nasal, either the language or the framework has to come from within the FG project itself, so in practice, it receives marginal attention.<ref>{{cite web | |||
|url = https://sourceforge.net/p/flightgear/mailman/message/34791061/ | |||
|title = <nowiki> Re: [Flightgear-devel] A FGPythonSys implementation: The embedded | |||
Python interpreter as a FlightGear subsystem. </nowiki> | |||
|author = <nowiki> Anton Gomez Alvedro </nowiki> | |||
|date = Jan 24th, 2016 | |||
|added = Jan 24th, 2016 | |||
|script_version = 0.40 | |||
}}</ref> | |||
Reinventing the wheel in a project that already relies so much on third party libraries simply makes no sense from an outsider's perspective, as it takes away precious and spare manpower from the crucial bits of the system.<ref>{{cite web | Reinventing the wheel in a project that already relies so much on third party libraries simply makes no sense from an outsider's perspective, as it takes away precious and spare manpower from the crucial bits of the system.<ref>{{cite web | ||
|url = https://sourceforge.net/p/flightgear/mailman/message/21740584/ | |url = https://sourceforge.net/p/flightgear/mailman/message/21740584/ | ||
| Line 108: | Line 129: | ||
In addition, documentation and end-user support for more established/mainstream languages is obviously much better than anything we have for the custom FlightGear/Nasal interpreter. | In addition, documentation and end-user support for more established/mainstream languages is obviously much better than anything we have for the custom FlightGear/Nasal interpreter. | ||
Many people argue that a more standard scripting language (like e.g. Python) would also attract more potential contributors to FlightGear, also the sizable community with 3rd party modules, means that new functionality can be much more easily implemented ("batteries included", and "no yak shaving" needed). | Many people argue that a more standard scripting language (like e.g. Python) would also attract more potential contributors to FlightGear, also the sizable community with 3rd party modules, means that new functionality can be much more easily implemented ("batteries included", and "no yak shaving" needed): | ||
* Python has much better, and much more, end-user documentation (in dozens of languages), which we don't have to write/maintain ourselves | |||
* Python has much better tooling to help with troubleshooting, debugging and profiling code (think IDEs, integrated editors for writing/debuggnig code) | |||
* Python also is being actively developed/maintained by an enormous community of contributors, which is to say we don't have to bother updating it, or fixing bugs in the runtime | |||
* Python is, and will continue to remain, relevant in the software engingeering industry even 30+ years from now (aka, it is more future-safe, and its success/availability is not tied to a single entity/use-case or application) | |||
* alternate/different GC implementations | |||
* JIT compilation/optimization opportunities | |||
* excellent support for threading and running code asynchronously (well, the GIL is there) | |||
* much better accessibility, and much lower barrier to entry: if I need help with Nasal, it's kinda problematic to expect the other party to have the runtime environment installed - whereas Python is a rather common thing these days, and it is kinda hard to imagine systems without a suitable Python implementation | |||
* FlightGear adopting industry solutions, and standards, is likely to become more relevant to professional players/companies/contributors (think being multi-platform, using OpenGL instead of DirectX, striving for OSG/HLA or CIGI adoption) | |||
* FlightGear supporting Python for embedded scripting/extension purposes is more likely to be appealing to educational setups already working with Python (think universities, college, CS/SE courses or aerospace programs) | |||
* With Python, when compared to Nasal, there is an "infinite" supply of people familiar with Python who use it professionally at their day jobs, many of whom are quite willing to use a language that is used in the real world, whereas learning a "niche" language is unlikely to be too appealing to people who need to make a living coding, simply because putting "Nasal experience" on your CV is unlikely to impress anyone (unless you happen to be an ENT doctor...) | |||
<ref>{{cite web | |||
|url = https://forum.flightgear.org/viewtopic.php?p=296287#p296287 | |||
|title = <nowiki> Re: Nasal must go </nowiki> | |||
|author = <nowiki> Hooray </nowiki> | |||
|date = Oct 8th, 2016 | |||
|added = Oct 8th, 2016 | |||
|script_version = 0.40 | |||
}}</ref> | |||
Thus, one of the key questions commonly posed is "''How coupled are Nasal and the scripting glue in FGFS ? Is there a clean break, or if not, can it be refactored without too much pain into something that would allow end dev users to use whatever scripting language they prefer, or have many modules already writen in, etc. If the coupling is not of the hair pulling type, it might be conceivable to integrate another scripting language alongside Nasal for starters, and in time, completely replace it if one is so inclined'' "<ref>{{cite web | Thus, one of the key questions commonly posed is "''How coupled are Nasal and the scripting glue in FGFS ? Is there a clean break, or if not, can it be refactored without too much pain into something that would allow end dev users to use whatever scripting language they prefer, or have many modules already writen in, etc. If the coupling is not of the hair pulling type, it might be conceivable to integrate another scripting language alongside Nasal for starters, and in time, completely replace it if one is so inclined'' "<ref>{{cite web | ||
| Line 145: | Line 188: | ||
== Concerns == | == Concerns == | ||
As discussed at https://wiki.python.org/moin/SandboxedPython , it's hard to embed Python without giving the scripts it runs full access to your system; hence, I wouldn't make this available to aircraft/scenery. (Nasal avoids this security problem by having its I/O functions only allow access to a limited range of files.) It could still be good to have available for local experimentation, but should be clearly labelled as insecure.<ref>{{cite web | |||
|url = https://sourceforge.net/p/flightgear/mailman/message/34788309/ | |||
|title = <nowiki> Re: [Flightgear-devel] A FGPythonSys implementation: The embedded | |||
Python interpreter as a FlightGear subsystem. </nowiki> | |||
|author = <nowiki> Rebecca N. Palmer </nowiki> | |||
|date = Jan 22nd, 2016 | |||
|added = Jan 22nd, 2016 | |||
|script_version = 0.40 | |||
}}</ref> | |||
It is hard to see a benefit having a binary integration of Python in the flightgear core. Using python to get/set properties and run fg-commands is already possible without any c++ integration.<ref>{{cite web | It is hard to see a benefit having a binary integration of Python in the flightgear core. Using python to get/set properties and run fg-commands is already possible without any c++ integration.<ref>{{cite web | ||
|url = https://sourceforge.net/p/flightgear/mailman/message/34792963/ | |url = https://sourceforge.net/p/flightgear/mailman/message/34792963/ | ||
| Line 206: | Line 260: | ||
|date = Jan 25th, 2016 | |date = Jan 25th, 2016 | ||
|added = Jan 25th, 2016 | |added = Jan 25th, 2016 | ||
|script_version = 0.40 | |||
}}</ref> | |||
FG intentionally should *not* have a proliferation of addons like FSX - we aim to merge into mainline what is reasonable to aviod having a plethora of mutually incompatible stuff like for other sims. So pointing out that there's no proliferation of addons is not a meaningful measure of success of failure.<ref>{{cite web | |||
|url = https://forum.flightgear.org/viewtopic.php?p=296221#p296221 | |||
|title = <nowiki> Re: Nasal must go </nowiki> | |||
|author = <nowiki> Thorsten </nowiki> | |||
|date = Oct 7th, 2016 | |||
|added = Oct 7th, 2016 | |||
|script_version = 0.40 | |script_version = 0.40 | ||
}}</ref> | }}</ref> | ||
== Challenges == | == Challenges == | ||
porting several hundred thousand lines of Nasal to something else, even if it is automated to a high degree, doesn't sound very attractive..<ref>{{cite web | |||
|url = https://sourceforge.net/p/flightgear/mailman/message/34791061/ | |||
|title = <nowiki> Re: [Flightgear-devel] A FGPythonSys implementation: The embedded | |||
Python interpreter as a FlightGear subsystem. </nowiki> | |||
|author = <nowiki> Anton Gomez Alvedro </nowiki> | |||
|date = Jan 24th, 2016 | |||
|added = Jan 24th, 2016 | |||
|script_version = 0.40 | |||
}}</ref> | |||
The paradigm of binding a C/C++ core to a scripting language has its own disadvantages, and that makes it very difficult to switch.<ref>{{cite web | The paradigm of binding a C/C++ core to a scripting language has its own disadvantages, and that makes it very difficult to switch.<ref>{{cite web | ||
|url = https://forum.flightgear.org/viewtopic.php?p=296201#p296201 | |url = https://forum.flightgear.org/viewtopic.php?p=296201#p296201 | ||
| Line 279: | Line 353: | ||
|date = Jan 23rd, 2016 | |date = Jan 23rd, 2016 | ||
|added = Jan 23rd, 2016 | |added = Jan 23rd, 2016 | ||
|script_version = 0.40 | |||
}}</ref> | |||
You can really only make progress on the Python front if you accept that Nasal is there to stay until Python works reasonably well without causing tons of regressions, absent that, there will be (at least) two co-existing scripting solutions, which is likely to complicate the situation even further - especially in the light of the mess that $FG_ROOT/Nasal and aircraft-side scripts have become, despite Nasal being an unpopular niche language. | |||
Otherwise, supporting Python would only make the shortcomings in FlightGear's legacy architecture even more prominent, quite possibly within just a few months, because Python has a much lower barrier to entry, so that people will be able to create all sorts of functionality, and in the process continue to clutter the main loop and make it even more non-deterministic than it is already. | |||
The first step really is coming up with a mode to entirely disable nasal using something like --disable-nasal, and the next step is to rework the Nasal initialiation sequence, to move away from the hard-coded assumption that the whole system is up and running and that all modules are always loaded - at that point, a non-Nasal mode can make actual progress, because FlightGear would no longer necessarily depend on Nasal - sooner or later, FGNasalSys could be reworked to support multiple instances, so that aircraft scripts can be treated, and executed, separately - i.e. separate from other simulator scripts (think GUI, scenery), and it would become '''much''' more straightforward to run certain parts on helper threads, or even introduce private property trees for certain purposes (think one private property tree for the FDM/AP, one for Canvas textures (MFDs)). | |||
All of this could greatly benefit from the Phi/mongoose groundwork that Torsten completed, because it would provide for a sane mechanism to get/set properties and run fgcommands, without requiring Nasal inevitably. | |||
However, we really don't want to have another scripting interpreter running inside the main loop/thread - that would only add even more work to our todo lists.<ref>{{cite web | |||
|url = https://forum.flightgear.org/viewtopic.php?p=296298#p296298 | |||
|title = <nowiki> Re: Nasal must go </nowiki> | |||
|author = <nowiki> Hooray </nowiki> | |||
|date = Oct 8th, 2016 | |||
|added = Oct 8th, 2016 | |||
|script_version = 0.40 | |script_version = 0.40 | ||
}}</ref> | }}</ref> | ||
== The current situation == | == The current situation == | ||
*If* you want embedded scripting with hooks inside flightgear, and you don't want nasal, then you are probably on your own with that. When we discussed this previously we decided that we didn't want to encourage multiple optional script engines because of the chances people will create aircraft with dependencies that no one else has and also the extra complexity of maintaining multiple script engines in the core code.<ref>{{cite web | |||
|url = https://forum.flightgear.org/viewtopic.php?p=296534#p296534 | |||
|title = <nowiki> Re: Why not use Erlang instead of </nowiki> | |||
|author = <nowiki> curt </nowiki> | |||
|date = Oct 11th, 2016 | |||
|added = Oct 11th, 2016 | |||
|script_version = 0.40 | |||
}}</ref> | |||
We have a single SGSubsystemMgr based FGNasalSys class inherited from SGSubsystem, which implements support for FlightGear scripting (add references/links below): | We have a single SGSubsystemMgr based FGNasalSys class inherited from SGSubsystem, which implements support for FlightGear scripting (add references/links below): | ||
| Line 392: | Line 491: | ||
|date = Oct 7th, 2016 | |date = Oct 7th, 2016 | ||
|added = Oct 7th, 2016 | |added = Oct 7th, 2016 | ||
|script_version = 0.40 | |||
}}</ref> | |||
We should probably implement multi-processing or multi-threading using message passing, and mirrors of the property tree. (And hence no concerns about locking at all!) | |||
Rough design is: | |||
* main thread has existing property tree | |||
* other threads have read only copy (mirror) of their parts property tree (subscription model, so only the parts they have requested) | |||
* new code synchronies property changes from the master to the copies once per frame or some other interval | |||
* other threads can ONLY interact with the main thread via the command API (which can be serialised into messages, and queued to be passed back to the main thread). | |||
(so on a non-main thread, when you try and do ‘setValue’ on a property, we queue a ‘property-assign’ command and send it to the main thread) | |||
The good thing about this is it works with multi-threading but also with multi-processing (using shared memory, named pipes or Unix domain sockets) so we could e.g. have the main simulation logic run as a separate process from multiple viewers. But we can run as a single process with threads using local memory to implement the message-passing of commands and have very little overhead. (Even though overhead of shared memory or Unix sockets will be virtually zero also). And of course if we use TCP sockets this is really distributable over a LAN (but not a WAN I guess) James intends to prototype this using some of the existing Network/IO protocols, eg the [[Telnet usage|telnet]] one and a binary equivalent. | |||
James mentioned it can be done with only small changes to SGPropertyNode, initially. What he would then suggest is, we implement scripting options such as Python by implementing a small library in that language, which talks the message-passing protocol. | |||
Again for a proof-of-concept this could be done using the [[Telnet usage|telnet]] or props protocols, and as more performance is needed we using a binary protocol and shared memory. | |||
We still need the discussion about which language is mandatory but using a different language for an optional component won’t impact the core code at all. | |||
Note this is totally orthogonal to the HLA work - HLA is a much higher level abstraction, we cannot use it to mirror the property tree. James' goal above is to explore a low-cost route to exploiting multi-core CPUs with the current codebase, in a way which keeps existing code working and fixes the locking / race bugs we already have with OSG touching properties at the same time as the main thread. So, in terms of any scripting system, James would suggest to think about it along these lines, and especially any problems you can foresee, because it’s much better that we agree what is practical and move together, than have three different attempts to fix or work-around the lack of locking in the property tree.<ref>{{cite web | |||
|url = https://sourceforge.net/p/flightgear/mailman/message/34790212/ | |||
|title = <nowiki> Re: [Flightgear-devel] A FGPythonSys implementation: The embedded | |||
Python interpreter as a FlightGear subsystem. </nowiki> | |||
|author = <nowiki> James Turner </nowiki> | |||
|date = Jan 23rd, 2016 | |||
|added = Jan 23rd, 2016 | |||
|script_version = 0.40 | |script_version = 0.40 | ||
}}</ref> | }}</ref> | ||