FGPythonSys: Difference between revisions
Jump to navigation
Jump to search
Line 206: | Line 206: | ||
=== Security === | === Security === | ||
{{FGCquote | |||
|1= The security issue is that I expect FlightGear aircraft and scenery to be "content", i.e. safe to use even if I don't trust their authors with all my files, not "executables" (such as standalone-Python scripts), i.e. unrestricted so only to be installed from trusted sources. I agree it would be access as the FlightGear user and not as root, but that's already enough for the common home-user-targeted forms of malware. (And it might be remotely exploitable: last time I looked, Terrasync was un-authenticated, which is fine for a content-delivery channel but means it shouldn't be used for executables.) Inkscape, Gimp, etc only expose their scripting interface to plugins, not image files (i.e. the equivalent of giving FlightGear a Python interface but not allowing aircraft to use it, which I don't object to); Blender has an option to allow scripts in model files, but it is off by default (https://www.blender.org/manual/advanced/scripting/python/security.html). | |||
|2= {{cite web | |||
| url = http://sourceforge.net/p/flightgear/mailman/message/34789666/ | |||
| title = <nowiki>Re: [Flightgear-devel] A FGPythonSys implementation: The embedded | |||
Python interpreter as a FlightGear subsystem.</nowiki> | |||
| author = <nowiki>Rebecca N. Palmer</nowiki> | |||
| date = Jan 23rd, 2016 | |||
| added = Jan 23rd, 2016 | |||
| script_version = 0.23 | |||
}} | |||
}} | |||
{{FGCquote | {{FGCquote | ||
|1= I have toyed with the idea of embedding Python briefly in 2010. It can certainly be useful for a side project based on Flightgear, but James here has very good points and also Rebecca is right about security. Access to /home is a bad idea: it does not even have to be malicious, remeber the Steam bug which wiped the entire home directory a couple of years ago? | |1= I have toyed with the idea of embedding Python briefly in 2010. It can certainly be useful for a side project based on Flightgear, but James here has very good points and also Rebecca is right about security. Access to /home is a bad idea: it does not even have to be malicious, remeber the Steam bug which wiped the entire home directory a couple of years ago? |
Revision as of 16:23, 26 January 2016
Started in | 01/2016 |
---|---|
Description | Python scripting support for FlightGear |
Maintainer(s) | bugman |
Contributor(s) | bugman (since 01/2016) |
Status | Under active development as of 01/2017 |
This article is a stub. You can help the wiki by expanding it. |
What is it
See Python in FlightGear for the main article about this subject. |
An experimental SGSubsystem integrating an embedded Python interpreter in FlightGear using the existing SGSubsystemMgr infrastructure (currently not merged with sg/fg next).
I am interested in experimenting with the FGPythonSys concept to break a number of FlightGear's current limitations. This probably has significant overlap with Stuart's HLA/RTI developments. I would like to try to experiment with the concept of subsystem creation. Specifically to add support for creating three different categories of subsystem: - Same thread spawning, i.e. the current design. - Subsystem thread spawning (for subsystems requiring very low latency). - Subsystem child process forking, using HLA/RTI for communications (for subsystems where latency and slow IPC is not an issue). — Edward d'Auvergne (Jan 25th, 2016). [Flightgear-devel] Designing a thread-safe property tree API (was
Re: A FGPythonSys implementation: ...).
(powered by Instant-Cquotes) |
I am happy to use FGPythonSys as a guinea pig here, as I would really like it run in its own dedicated thread. This is a good system for quickly writing many tests for catching sub-thread and child process racing conditions (this links back to my CppUnit test suite ideas). I would like to also expose subsystem creation - using all 3 of the above mechanisms - via a new Python 'subsystem' module (written purely in C++, including the HLA/RTI parts). The forked child process + HLA/RTI concept has a particularly high value with Python, as the child process will then have it's own Python global interpreter lock (GIL), isolated and independent from the FGPythonSys GIL. — Edward d'Auvergne (Jan 25th, 2016). [Flightgear-devel] Designing a thread-safe property tree API (was
Re: A FGPythonSys implementation: ...).
(powered by Instant-Cquotes) |
Vision
I would present this as a technology enabler, and one like no other. I envision implementing the following Python modules:
|
This Python API idea is to expose the lowest levels of the internal FlightGear C++ infrastructure, so that new FlightGear core code can be written in pure Python. One benefit that this experiment might lead to is to enable powerful code prototyping. A subsystem could be first written in Python, and then ported to C++. Or if the subsystem is fast enough, which in most situations will actually be the case, then it could remain in Python and be part of the FlightGear distribution (e.g. the code would be installed into /usr/lib/flightgear/python/, and not be part of FGData). By integrating Python into a subsystem as I have done, there is the benefit of very low latency and fast millisecond / kH code execution that higher level IPC mechanisms can never come close to matching (HLA/RTI, websocket, etc.). Therefore someone could write Python FG code that has the potential to run at a similar speed to core C++ code. The websocket mechanism also does not give access to the lowest levels of the infrastructure. So the greatest advantage is that extremely low-level core FlightGear code can be written in Python. — Edward d'Auvergne (Jan 25th, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
Aircraft Scripting
As for using this as an aircraft scripting language, then obviously we would present just a subset of the core FG functionality. An aircraft does not need access to simgear. I'll get to the important security issues soon. And, yes, this could possibly all be done with Nasal, but that's not the point. The Python language offers a high-level object oriented design, which Nasal does not, and is one of the most widely used languages for code prototyping. It may also prove to be much better in dealing with garbage collection, as a large number of people have worked hard over many, many years to make this part of Python extremely efficient (this is required because of Python's GIL). I also have a very deep knowledge of Python (I still have a functional copy of Python 1.0.1, the first version I coded for, on my system). Therefore I can construct powerful Python/C API interfaces very quickly, something I cannot do with Nasal. Anyway, I see great benefit in experimenting and enabling such a low-level API. What I don't see is a real disadvantage with this idea, and why this is proving to be so controversial. — Edward d'Auvergne (Jan 25th, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
Status
Motivation
The difference in speed between Python and C/C++ for complex operations is not that great. For more than 90% of the code in software, the language speed is not an issue. When it is, you write small Python modules in C to do the work. That could be done here, or the prototyping code all converted to C++. Anyway, orders of magnitude more important than the language speed differences is the design and algorithm. — Edward d'Auvergne (Jan 25th, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
a problem I'd like to solve for the FGPythonSys experiment. And that is to spawn independent subsystems from within Python. This itself has two purposes - to provide a Python interface to enable core code to be written in Python (strictly disabled for content developers), and to allow for a new, thread-safe mechanism to perform aircraft and other updates via Python. — Edward d'Auvergne (Jan 26th, 2016). Re: [Flightgear-devel] Designing a thread-safe property tree API
(was Re: A FGPythonSys implementation: ...).
(powered by Instant-Cquotes) |
With the new FGPythonSys, I could help work on making core infrastructure thread-safe right now without affecting the operation of the rest of the program. Placing FGPythonSys in its own subsystem thread would be the ultimate tool for hardening the core - it would highlight all the points in FlightGear that are prone to racing and locking. For example the global event manager and timer system with function call-backs is probably not thread-safe. And the property tree will clearly race all over the place. If there is interest, I can use these experiments to make these systems thread-aware and thread-safe, while maintaining full compatibility with all the other non-threaded subsystems. This might help eventually shift parts of Nasal, such as it's garbage collector, outside of the main thread (something that HLA/RTI does not appear to be suitable for). — Edward d'Auvergne (Jan 26th, 2016). Re: [Flightgear-devel] Designing a thread-safe property tree API
(was Re: A FGPythonSys implementation: ...).
(powered by Instant-Cquotes) |
Goals
Threaded Subysystems
I was hoping to use real threads for running FGPythonSys outside of the main thread, and child process forking and HLA/RTI communication on the C++ side for the aircraft subsystem. This would have given FGPythonSys and the aircraft their own embedded Python instance with GIL, so that their operation would be fully independent. The design would have really stress-tested FlightGear against racing conditions. — Edward d'Auvergne (Jan 26th, 2016). Re: [Flightgear-devel] Designing a thread-safe property tree API
(was Re: A FGPythonSys implementation: ...).
(powered by Instant-Cquotes) |
Benefits for Nasal
The developments I was hoping on working on in FGPythonSys could easily be ported to FGNasalSys. I am just not very familiar with the Nasal internals. For example I'll take back my statement about Nasal not being object oriented (thanks to the anonymous messenger for pointing out that blatant mistake). So from my perspective, I can personally bring developments to the table that could have beneficial side-effects. For example by helping Stuart with HLA and the property tree - which must change for HLA to be viable. Or advancing the subsystem infrastructure to support main thread, true threading and HLA modes. Putting FGPythonSys up as the first volunteer ensures that it experiences all the initial painful racing conditions, taking one for the team. I can only practically develop this via embedded Python, as that is my current expertise. But the developments could in the end be beneficial to Nasal as well. — Edward d'Auvergne (Jan 25th, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
Message Passing APIs
For example, it would be possibly to deprecate virtually all the custom Nasal APIs in favour of commands, but much harder to do that solely using properties. (Think about searching the NavDB or appending a waypoint to a flightplan). If we started moving in that direction, we don’t need to create separate APIs for multiple languages, they simply work using the properties + commands system, and the command system already has the exact set of property modification operations you proposed. I’m not set upon my message passing idea at all, but most solutions that don’t involve fine-grained locking of each property ultimately equate to message passing /anyway/; you have to queue up a sequence of changes to the property tree state, apply them in order and make the results visible. Given that, and the fact we already have the command system, my proposal is simply one way (and I think, a pragmatic one) to create such a setup using existing pieces available. — James Turner (Jan 25th, 2016). Re: [Flightgear-devel] Designing a thread-safe property tree API
(was Re: A FGPythonSys implementation: ...).
(powered by Instant-Cquotes) |
Concerns and Issues
Dependencies
For one it add another huge dependency. Also, like Thorsten stated, there is a real possibility that a particular aircraft will end up requiring some obscure python module. If you add Python support by default you will see people using it for their aircraft. — Erik Hofman (Jan 24th, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
I am happy with N different /optional/ languages being supported, but ideally through an RPC mechanism which is language neutral, hence my email about message-passing designs. I feel pretty strongly we do not want multiple /mandatory/ languages for core features of the simulator. In Qt we had a webkit dependency which required Ruby, Perl *and* Python to compile on Windows and the backlash from the developer / user community was extremely large. — James Turner (Jan 24th, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
Security
The security issue is that I expect FlightGear aircraft and scenery to be "content", i.e. safe to use even if I don't trust their authors with all my files, not "executables" (such as standalone-Python scripts), i.e. unrestricted so only to be installed from trusted sources. I agree it would be access as the FlightGear user and not as root, but that's already enough for the common home-user-targeted forms of malware. (And it might be remotely exploitable: last time I looked, Terrasync was un-authenticated, which is fine for a content-delivery channel but means it shouldn't be used for executables.) Inkscape, Gimp, etc only expose their scripting interface to plugins, not image files (i.e. the equivalent of giving FlightGear a Python interface but not allowing aircraft to use it, which I don't object to); Blender has an option to allow scripts in model files, but it is off by default (https://www.blender.org/manual/advanced/scripting/python/security.html). — Rebecca N. Palmer (Jan 23rd, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
I have toyed with the idea of embedding Python briefly in 2010. It can certainly be useful for a side project based on Flightgear, but James here has very good points and also Rebecca is right about security. Access to /home is a bad idea: it does not even have to be malicious, remeber the Steam bug which wiped the entire home directory a couple of years ago? — Adrian Musceac (Jan 24th, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
Python has too much introspection for such wrappers to work: open.__self__.real_open breaks it. (They never really worked in Nasal either (the old io.nas), which is why Nasal security is now done C++-side.) — Rebecca N. Palmer (Jan 24th, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
I'd say that bringing in the Python toy is potentially harmful for the reaons already stated. — Torsten Dreyer (Jan 25th, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
The only issue mentioned so far is trojan pegasuses. Specifically when FGPythonSys is used for content written by a user with malicious intent. This is also the only 'security' issue I can see as well, at least for now. This is a major issue. — Edward d'Auvergne (Jan 25th, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
Main Loop Impact
I like Python as a language and use it all the time, but in an embedded context it is quite large and the GIL makes using multiple Python instances on parallel threads tricky. — James Turner (Jan 23rd, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
I wonder why a C++/subsystem integration of Python would be necessary. I am successfully using JavaScript via nodejs for a private project with FlightGear. Communication with FGFS runs via websocket and the http/fgcommand interface. I don't miss a single feature and everything works as one would expect. I would assume, the same technique would be possible with Python or any other scripting language that supports websockets, http-get, http-post and has some support for JSON (python has, ruby has, lua has). That being said, I don't see a benefit having a binary integration of Python in the flightgear core. Using python to get/set properties and run fg-commands is already possible without any c++ integration. — Torsten Dreyer (Jan 25th, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
that Python integration is exactly the opposite way of what I would like the fg core to develop. Instead of adding just-another-feature we need to strip it down to getting a fast and constand fps rendering engine. Everything else needs to run outside the main loop and has to interact with the core by, say HLA/RTI or whatever IPC we have. — Torsten Dreyer (Jan 25th, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
My major concern with your Python integration is that it introduces an unpredictable (w.r.t. timing) step in the main loop getting us on step further away from constant frame rates. — Torsten Dreyer (Jan 26th, 2016). Re: [Flightgear-devel] Designing a thread-safe property tree API
(was Re: A FGPythonSys implementation: ...).
(powered by Instant-Cquotes) |
What scares me is that we could end up with lots of code in a scripting language that is essentially a performance dead end. And this when FG already is quite CPU bound and barely uses more than 1 CPU core. If Nasal should be replaced or augmented, then at least do it with a language and runtime that brings definite performance improvements and a multi threaded future. — Stefan Seifert (Jan 25th, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
Impact on Nasal
a different scripting language seems to be just that - a different language capable of doing the same things we can do right now in a different syntax. And in my view, that's not worth so much trouble. So maintaining a compile-time interface that is off by default for, say, research applications or people who want to use this for their own experiments sounds like a good idea to me. A situation in which different aircraft run on different scripting languages however does not. — Thorsten Renk (Jan 25th, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
Nasal has to be about the most straightforward scripting language I've ever had to deal with (including python, which I'd used before Nasal.) The fact that it's not really used outside of FG is completely irrelevant - the syntax is completely straightforward, there are thousands of example scripts to draw on and even if there are comparatively fewer people who are "experts" in it, every one of these virtually without exception will be knowledgeable about its use within FG. What use are a hundred thousand Python experts if they don't understand your problem? As others have already said, there are already many ways of interacting with the property tree with other languages; does anyone truly think that extending the actual core of FG in python is a good idea in any way? — AJ MacLeod (Jan 25th, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
Deprecating Nasal?
I also feel, wearing my ‘computer science’ hat, that a Nasal -> anything translator can be made robustly; we have an AST builder for the language already, the language is very regular, its types map to other languages directly, and we have the entire relevant corpus of code sitting in fgdata/ already; it’s straightforward to use that body of code to validate the translation. Hence my statement that we /could/ change the mandatory language, but we need to show some large benefits, and a migration path where we have both languages coexisting, and then prove we can machine translate, and then deprecate the old language, and finally remove support. What I don’t want is a situation where the Bombable script uses Lua, Advanced Weather uses Nasal, AI traffic uses Python, some particular aircraft uses Ruby and so on - that would be a disaster. As Thorsten said, Nasal isn’t great but we have it, it’s very small footprint and we have control over it, so we do need a hugely clear benefit to transition. If Python had trivial multi-thread safe interpreters with non-blocking garbage collection, that would be almost sufficient in my mind, but frustratingly it doesn’t. — James Turner (Jan 24th, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
I'm predicting you can't do it on the first go, but given enough effort, I suspect you can get there. So then - how does it look in practice once Nasal is deprecated? Right now I have 10.000 lines of AW which I know - with my comments in, with my formating conventions. I know where stuff is and can fix things quickly After your translator runs over it, I have 10.000 lines without my formating conventions in a language I'm not really familiar with. It runs - but my maintenance load has gone up by a factor 10. So it becomes essentially a froen feature. Unless I continue writing in Nasal and we just run the translator script every time something changes. So then I can maintain it - but everyone else who doesn't have the Nasal master file gets to see the machine-compiled Python version and has to dig into that... I don't know, doesn't sound too much fun to me either... — Thorsten Renk (Jan 25th, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
porting several hundred thousand lines of Nasal to something else, even if it is automated to a high degree, doesn't sound very attractive.. — Anton Gomez Alvedro (Jan 24th, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem..
(powered by Instant-Cquotes) |
High Level Architecture (HLA)
HLA is important here but not at the property level. — James Turner (Jan 25th, 2016). Re: [Flightgear-devel] Designing a thread-safe property tree API
(was Re: A FGPythonSys implementation: ...).
(powered by Instant-Cquotes) |
HLA is really intended to operate at a much higher level of granularity and pass objects around a distributed simulation. The use-case I'm working on at the moment passes an AI instance as an object (with position, velocity, accelerations, model to be displayed). How a HLA Federate maps that data internally is an implementation decision. There shouldn't be any race conditions because no assumptions are made about how the data is used or mapped. At present, I'm not thinking that HLA will be used as a mechanism to provide a shared property tree across multiple Federates. I think that's the wrong way to use HLA. Instead you make explicit decisions about the data models you will communicate. The other reason HLA isn't a good choice is to do with simulation clocking. I'm still getting my head around this properly, but the overall model is one where the simulation clock is advanced, and the instance the updates and publishes changes to the RTI that will be picked up by other instances at the next clock advance. (I think - I've still to work on a use-case where there is significant inter-dependence between instances). — Stuart Buchanan (Jan 26th, 2016). Re: [Flightgear-devel] Designing a thread-safe property tree API
(was Re: A FGPythonSys implementation: ...).
(powered by Instant-Cquotes) |
Python Federates
even if Python isn't integrated with the FG core, there is plenty of opportunitie to use it within a FG-based simulation. — Stuart Buchanan (Jan 26th, 2016). Re: [Flightgear-devel] Designing a thread-safe property tree API
(was Re: A FGPythonSys implementation: ...).
(powered by Instant-Cquotes) |