Nasal for C++ programmers

From FlightGear wiki
Jump to navigation Jump to search


Some words on Nasal for fellow C++ programmers

Compared to C++, there is really nothing "low quality" about Nasal per se: Nasal is just the "script glue" that connects different parts of the simulator: Many Nasal scripts leverage C++ code - and it is very easy to add new C++ code that can be called from Nasal. So this is a perfect option to combine the speed of C++ with the flexibility of scripting.

History has shown, that most code in FlightGear will eventually be made more configurable and more accessible, this usually happens in the same steps:

  • replacing static variables with variables stored in the property tree
  • using listeners and callbacks to get update notifications for important variables
  • adding new fgcommands
  • fully exposing a "control" interface by making it accessible in the property tree (see e.g. the AI/autopilot or canvas systems)
  • providing new scripting hooks using extension functions (see Howto:Extend Nasal)
  • providing a full OOP interface (using simgear/nasal/cppbind) (see $FG_SRC/Scripting/NasalPositioned.?xx)

Even if you should know C or C++ already, Nasal probably remains the most accessible and the most powerful method for customizing the simulator in the beginning, simply because it is extremely easy and fast to get started, you don't need an "integrated development environment", you don't need to install compilers and you don't need to satisfy any 3rd party dependencies; bottom line being: if you can run FlightGear, you can also run Nasal and directly create new features.

In addition, Nasal code is fairly abstract code, too. Once you start looking at some existing Nasal scripts, you will see that it is also fairly high level code, much more high level than C++ or C- so Nasal has a much higher density of code, too. Nasal's role in FlightGear really is like JavaScript's role in Firefox or Chrome, where it is also used for many/most core-related logics (CSS/XUL).

Nasal is an extension language, a scripting language that is much more high level than C++ and specifically meant to be used for extending a host application (FlightGear). C++ on the other hand is a systems programming language (most APIs in Nasal are FlightGear specific).

Nasal's power is due to the fact that it is not as low level as C++ code, and that it is very easy for non-programmers to get started adding custom features to FlightGear, without requiring a background in software development or programming.

Aircraft developers or other base package contributors don't need to be familiar with C++ programming, they don't even need an IDE or a working build environment. They can just write scripts in a text editor and drop them into the base package, and things will simply work. That is because FlightGear is the runtime environment for Nasal scripts.

Obviously, an experienced programmer can make use of more complex Nasal features, and come up with much more optimized code and sophisticated features.

Basically, there really is nothing in either language that you cannot also do in the other language - ever heard of "turing completeness"?

Of course, some things are better handled in C++ space, while others are better (=more easily) done in Nasal, usually you'll want to implement performance-critical stuff in C/C++ and then make it accessible to scripting space.

It really boils down to what needs to be done, usually new features can be implemented (or at least prototyped) with just some Nasal code, sometimes new Nasal functions may be required to interface to the rest of FlightGear.

Some words on plugins

Experienced C++ developers may be looking for an SDK to create plugins for FlightGear. This is a discussion that has been brought up frequently by new FlightGear users who would like to create new features/modules for FlightGear.

FlightGear doesn't have such an SDK, and the general consensus is that FlightGear doesn't need an SDK for creating binary plugins. FlightGear is an open source flight simulator, developers are invited to directly contribute to FlightGear itself.

While adding support for binary plugins is of course possible, and not even that complicated, it would be a pretty bad idea for a number of reasons:

FlightGear supports a fairly powerful (and high level) scripting language to provide extensibility - dynamically linking to binary plugins (DSOs, DLLs) however is highly platform specific and has also many other disadvantages (e.g. mutual library dependencies and versioning requirements).

Not to mention, that plugin developers need to be able to build/compile their modules before using them. Nasal scripts on the other hand "just work" by dropping them into the base package folder, in its current form, there's nothing platform-specific about Nasal code - if it runs on one machine, it will also run another - regardless of your CPU, GPU or operating system.

Instead, FlightGear is usually extended by either using its support for Nasal scripting or by adding new C++ code to the source tree, or a combination of both approaches (so that Nasal scripts can call C++ code).

Occasionally, plugin developers for commercial flight simulators such as MSFS or X-Plane highlight how important such a plugin system is. But the truth is, it is only since fairly recently, that there are discussions going on in the X-Plane community to also add scripting support to their simulator, ironically by the use of a python plugin specifically for scripting:

Cquote1.png What we need is a scripting system. The scripting system would provide a relatively simple text-file syntax to do simple scripting of systems and instruments for airplanes.
— X-Plane Scenery Blog[1]
Cquote2.png

Now let's look at this decision: They have like how many years experience working with binary plugins? How come they still intend to support scripting, probably because it's the better solution!

So, in this aspect, FlightGear is really years ahead of the other two large flight simulators, both of which provide fairly limited script-ability.

Also, you cannot circumvent the requirements of the GPL 2, so any binary modules (plugins) linked into GPL'ed software, need to be GPL'ed too - which requires the corresponding source code to be also distributed alongside with any binary releases.

If you want to add new features to FlightGear, the usual way is to add a new subsystem - for an example of how this is done in C++, you may want to search the FG forums for the discussion on "ridge lift" that took place a while ago which shows how the fellow forum member "WooT" implemented support for ridge lift by adding a new subsystem to FlightGear, see these threads for details:

Related discussions:

Performance

Obviously, C++ code will usually be faster than the corresponding Nasal code. But, while performance is not a design goal, Nasal isn't especially slow either. For example, early benchmarks of the garbage collector showed it as faster than perl's reference counter, and its number crunching performance is on par with python. But in all cases, simplicity, transparency and a sane feature set are preferred over speed in Nasal.

Nasal was specifically designed for use as an extension language in an larger project such as FlightGear. The problem with many otherwise excellent languages in this environment is that they are huge. Perl and python are great, but enormous. Even their "core" interpreters and library code are larger than most projects that require an embedded language. They cannot be readily shipped with their host application and need to be installed system-wide. This is a pain and a compatibility hassle.

The real goal with Nasal is to have a language that supports most "normal" programming idioms (objects, functions, arrays, hashes) while avoiding the bloat that comes from "platform" scripting languages like perl, python, ruby and php.


Garbage collection

Nasal garbage collects runtime storage, so the programmer need not worry about manual allocation, or even circular references. The current implementation is a simple mark/sweep collector, which should be acceptable for most applications. Future enhancements will include a "return early" capability for latency-critical applications. The collector can be instructed to return after a certain maximum delay, and be restarted later.


As far as speed goes, the last time any benchmarking for Nasal was done, it was about as fast as Perl 5 or Python 2.2 at most things. It's garbage collector was faster, its symbol lookup about the same or slightly faster, and its bytecode interpreter somewhat slower.

Meanwhile, the GC has been identified as a potential bottleneck in FlightGear, see How the Nasal GC works for details.

Thread safety

Unlike almost all other script interpreters (and unlike the FlightGear/Nasal interface itself currently) , Nasal is thread-safe and scalable when called from multiple CPU threads (as opposed to the user space interpreter threads implemented by Ruby or the GIL used by Python for example).

No special treatment is required (as for perl, which clones a separate interpreter with separate data for each thread and uses locking around specifically-designated shared data) and the threads can be scheduled simultaneously. There is no global lock on the interpreter, as used by Python or Lua. The only limit on scalability is garbage collection, which must block all interpreter threads before running.

When running threaded code, Nasal provides "minimal thread safety", meaning that the interpreter itself can be safely called from multiple CPU threads without risk of corrupting or deadlocking the interpreter internals. Multithreaded operations are therefore "safe", although they are not guaranteed to be atomic. In particular, poorly synchronized insertions into containers can "drop" objects into oblivion (which is OK from an interpreter stability standpoint, since the GC will clean them up normally). Nasal itself provides no synchronization primitives to address this; thread architecture is a "top-level" design job, and Nasal is intended to be an extension language in a larger project. Choice of synchronization mechanisms is going to be highly application dependent.

Exception handling

Like python, nasal supports exception handling as a first-class language feature, with built-in runtime-inspectable stack trace. Rather like perl, however, there is no special "try" syntax for exception handling, nor inheritance-based catching semantics. Instead, you use the call() builtin to invoke a function object and inspect the results to determine what error was thrown (either with the die() builtin or via an internal runtime error) and what the stack trace looked like. Elaborate exception handling isn't really appropriate for embedded scripting languages.

High level programming

Thus, programmers already familiar with C++ shouldn't just disregard Nasal as a "toy" that doesn't seem suitable for serious development: some of the more complex Nasal scripts can literally make one's head spin around and it would quite obviously take much more C++ or Java code to implement the same features, while sacrificing all the flexibility and power that a scripting language offers.

Some features can certainly be more easily implemented in Nasal space, than in C++ space. Often, the Nasal solution is "on par" with similar solutions in C++.

Accessibility

For instance, Nasal code cannot only be easily run and contributed by all users, but it can also be easily reused and maintained by other users. This means, that given the number of active C++ developers, compared to the number of base package contributors, your code is more likely to be actively maintained by fellow users if it is written in Nasal.

In other words, if there are some experimental features you'd like to explore, Nasal is an excellent way to ensure that other FlightGear users can easily test your new features. This could be witnessed during the development of the local weather system or the bombable addon,too.

This is in stark contrast to features developed solely in C++ space, because these can usually only be tested by people able to build FlightGear from source, especially if your code isn't yet in the main repository, where it would eventually be available in the form of a binary snapshot.

Obviously, none of this is to say that Nasal is the perfect solution for any problem, there are many things for which Nasal isn't necessarily a perfect choice, such as low level code for example (i.e. rendering).

On the other hand, Nasal really is a powerful tool in FlightGear, and if you find that something should, but currently cannot, be done in Nasal space, it is extremely easy to add support for new features to the Nasal engine using extension functions or property listeners to trigger C/C++ code.

Shouldn't we favor C++ over Nasal?

Performance-critical code should definitely be implemented in C++ space. Still, there are several options, such as dedicated C++ subsystems (SGSubsystem) or Nasal extension functions.

Please, if you do know C++ and the SG/FG code bases, you are obviously encouraged to contribute directly to the core source code - nobody is trying to keep people from contributing there - see Howto:Start core development.

That said, among some core developers, there's apparently the misconception that other FlightGear contributors increasingly favor Nasal over C++.

Obviously, some core developers are particularly concerned about the amount of Nasal code added to FlightGear (i.e. the base package) recently.

So to put things a little into perspective, and trying to explain the current situation:

  • We must not miss the fact that these are complaints about the plethora of Nasal code added to FG by non-core developers, these are usually people who don't know C++, who don't know the FG/SG code bases and who don't know how to build FG from source or at least who don't regularly rebuild a custom/patched binary from source, and who don't have commit privileges to directly contribute to FlightGear.
  • It's not that C++/core developers suddenly decided to stop writing C++ code and instead use Nasal, Rather, the opposite is true, there are still a number of core developers who refuse to use Nasal at all and stated so publicly.
  • So let's face it: How many "qualified" Nasal coders can you name? How many Nasal developer are even able to program in C++, are familiar with the STL and Boost? How many Nasal coders would be potential C++ core developers? And then, how many of them are able to use git and gitorious? How many are actually able to build FG from source?
  • For example, up to a certain point, the local weather system has been entirely developed without ever building FG from source, and without using git.
  • While writing Nasal code is obviously much simpler than writing C++ code, writing really good Nasal code is still difficult - on the other hand, writing really bad code in Nasal is much more difficult than in C++.
  • Just because someone is able to write a script in a dynamic, untyped, garbage-collected programming language doesn't automatically mean that they are able to write quality C++ code that can be added to the main code base, using dependencies like the STL, Boost, OpenGL, OSG etc.
  • The truth is, there's TONS of Nasal code in the base package that has an impact on performance, because of the way it is written, not primarily because of the well-known GC issues.
  • Often, most high quality examples of Nasal code were written by people who also happened to be core developers.
  • So it's way too simple to say that forum users are writing Nasal because they feel it's "just better". And, clearly, nobody on the forum wants to replace fgfs.exe with "nasal.exe"[1]. It's really just polemics to suggest that forum users feel "core=bad" and "nasal=good", forum users tend to contribute in the form of Nasal code, because the barrier to entry is so much lower, and because their work can be deployed much more easily.
  • For forum users, it all boils down to an accessibility issue: Nasal is obviously much more accessible than C++ coding, not just for the developer of a feature, but also for people wanting to use the feature.
  • Yes, it's true, Nasal code makes the main loop non-deterministic, i.e. because of the garbage collector - but there are people currently working on improving the GC: How the Nasal GC works.
  • On the other hand, we also have tons of C++ code which is not using smartpointers and yes, which is leaking memory - in other words which is "non deterministic", too (definitely from an embedded developer's perspective).
  • Now "fixing" non-deterministic Nasal code involves re-implementing a single self-contained component (a single C file actually, with ~320 lines of code), the Nasal GC - or even just adding support for an existing GC implementation. Which is something that is currently being worked on by another contributor, so that the GC impact will eventually be not as severe probably.
  • Making the rest of FG stop leaking memory involves much more work unfortunately.
  • And then, we have threading-related segfaults because of C++ code, where running the same binary with the same arguments sometimes works, and sometimes simply doesn't, and crashes FG.
  • Really, writing badly performing C++ code is much easier than writing bad Nasal code. There are only a handful of ways to crash FG from Nasal.
  • And writing/reviewing Nasal code doesn't involve the same skills that C++ programming requires.
  • Nasal code doesn't need to be reviewed and committed by core developers - it can be peer-reviewed by other Nasal programmers, which is often happening on the forums.
  • On the other hand, we've seen quality C++ code submissions which ended up not being committed or even reviewed for months (or even years)
  • Deploying contributions written in Nasal is MUCH simpler, and doesn't necessarily involve any core developers!
  • When compared to Nasal, we have less people in the FG communtiy who know how to program C++ and who know how to build FG from source, that's what it all boils down to, really.
  • We cannot possibly replace or rewrite the sheer amount of scripted code we have, and looking at all the features implemented in Nasal, most of them cannot as easily, or simply shouldn't be implemented in C++ at all. After all, FG is all about being open and extendable for end-users.
  • It's true, we get to see an increasing amount of features being developed in the base package. Which is a good thing because not every conceivable feature can be developed by core developers in C++ space. Many of these features were developed in Nasal because it was possible and simple.
  • Contributing to the C++ code, simply isn't easy. Which is not meant to sound negative: it's a long-standing issue. Also, anything coded in C++ is obviously not as flexible and configurable, not without making excessive use of properties and XML config files, and then there are still ugly corner cases like tied properties or properties only read once during startup, or tags that only support static value and no properties at all.
  • The other problem is that there is simply a shortage of active C++ developers familiar with the project and able to build from source.
  • So, it is MUCH easier to develop MOST end users features in scripting space than in C++. That's one part of the reason why we sometimes get to see tons of new features showing up in the base package, while only very few new features show up in the SG/FG repositories in the same month.
  • Anybody complaining about the degree of Nasal additions to FlightGear, needs to keep in mind that even developing just a single feature in C++ space takes often much longer (and involves more people, if they don't happen to be core developers), this applies especially to inexperienced programmers.
  • So whenever somebody is complaining about the plethora of scripted code in Nasal, it is a good idea to keep the user (non core developer) perspective in mind: people are trying to bring changes to FlightGear, to contribute in a meaningful way, without blocking each other and without facing a steep learning curve, without being limited by the number of releases per year or by core developers having some time to mentor them.
  • Nasal provides a very real chance for people to innovate and develop features without depending on core developers and without having to be able to build FG from source
  • We have tons of features that we would not have without Nasal in the first place, including the local weather package and the bombable addon - these are features that probably would have never been developed by core developers. Frankly, because C++ is just too low level a language for such things and because features implemented in C++ space often end up there because someone cared enough to implement them in the first place, not because some end user "requested" them.
  • if the primary concern is really shifting the development focus to increased use of C++ instead of Nasal, then, it needs to be made much easier for new developers to get involved, which includes reviewing patches and merge requests. At the moment, the project simply isn't there yet - which mostly boils down to a lack manpower obviously.
  • Also, what's the alternative here? That people stop developing their own features just because we don't have enough active core developers who can provide help and support to newcomers and actually mentor them to a degree such that their patches can be reviewed and committed?
  • This is not a problem about Nasal at all, it's a manpower and accessibility problem: due to FG's flexible design we have more and more users who want to add to it and extend/improve FlightGear. This however isn't particularly straightforward for people without solid multiplatform, C++/OOP, programming experience - they have to face an extremely steep learning curve. Nasal simply provides the lowest entry barrier here.


Seriously, people who tend to write code in Nasal rather than in C++, usually do so for a reason - with Nasal we have:

  • pretty good documentation
  • a fair number of beginner tutorials
  • lots of examples and snippets
  • people able and willing to help
  • dedicated sub forum
  • experienced Nasal coders mentoring new contributors
  • a fully working development environment, i.e. no need to set up a separate build environment
  • no tedious merge requests necessary, or long code reviews
  • no need to wait for a new release to use and distribute a new feature (remember, just 2 releases per year)

In other words, if you are a C++ core developer who wants to see more C++ code added to FG instead of Nasal code, try to beat that! ;-)

So a user might argue that, more or less active core/C++ developers are complaining about features getting implemented in scripting space without volunteering to implement them in C++ space instead.

Given the shortage of active C++ developers, and an increasing number of people being able to write Nasal code (and develop moderately complex features in scripting space) it isn't really surprising that we are seeing so many features being implemented in Nasal currently.

Really, what we've been seeing is a simple form of evolution where the project dynamics adapted to the situation at hand, by recognizing a lack of core developers and shifting the focus to where the manpower is, which clearly is not the C++ source code unfortunately ...

  1. Benjamin Supnik (14 February 2009). Scripting: A Line In the Sand.