Howto talk:Coding a simple Nasal Framework

Todo: Considerations for creating Canvas Widgets

• benefits of creating widgets
• supporting multiple instances
• creating multiple windows of a widget
• creating a window/dialog with multiple widgets (tabs?)
• all Canvas-based instruments/MFDs should support GUI-based viewing, without requiring a cockpit/aircraft-even if it's just for testing/development purposes (profiling)
• every widget must support rendering to a customizable context/canvas, so that recursion is fully supported, i.e. dialogs showing instruments with GUI elements

--Hooray (talk) 14:25, 29 June 2014 (UTC)

Scripted Frameworks vs. C++ development

I feel that the whole subject of creating a Nasal framework could become an exercise in futility, and should even be discouraged in favour of writing code within the core framework. Nasal simply does not have the suitable debugging tools which could make it a big winner. A large piece of code could easily become unmaintainable in case the original developer left. Case in point: local weather.

Adrian (talk)

I guess it's a matter of having to make concessions - in an ideal world, you'd be correct obviously. But given the state of affairs, manpower is not where core development is - but where fgdata contributions happen. The degree of Canvas-based developments and related novelties has meanwhile exceeded any related core developments (except for Canvas itself), including stuff like the hard-coded Map dialog, wxradar, agradar, groundradar, kln89 etc - and that was foreseeable even within a few months after the most basic drawing primitives were added and supported.

But what's been happening is a lot of unprecedented "glass" development in various areas, that was basically completely chaotic, disorganized - without any collaboration and coordination taking place. The few cases where this was not the case, we spent a ton of time doing behind-the-scenes networking. But otherwise, we're seeing a number of efforts, with very desirable features, being developed by people who have no clue/interest about collaborating properly, in order not to write redundant -and usually even incompatible/conflicting- code.

However, I am not sure if you've even played with Canvas recently, or if you're aware of the underlying C++ code, and the deprecated code that is made obsolete by Canvas.

But it is obvious that without any further involvement/direction from our side, all this fgdata development is going to continue obviously, without any significant degree of collaboration/coordination going on.

So coming up with frameworks in scripting space is the only sane, and scaleable, way to deal with this - absent some major change in the FlightGear development philosophy.

Also, what you say is not quite spot-on when it comes to "the core framework" though: What we'll certainly need to explore increasingly is extending the Canvas system itself through extended/added elements and primitives - and that's been happening already, Tom has added a huge number of novelties recently - but we cannot be expected to port all existing code that's being written by dozens of aircraft developers and other Canvas adopters.

Thus, localizing and encapsulating functionality in terms of having dedicated frameworks is the right thing to do - because it allows us to define boundaries, while phasing out/re-implementing more native hooks over time, i.e. more optimized code.

But as long as people come up with their own code doing all sorts of things in a heavily redundant fashion, they cannot benefit from such optimizations automatically, because all THEIR existing code will need to be manually ported - which is exactly the whole point of having a framework-centric development model.

If you've been following my Canvas related postings on the forum, you may have seen that I started exploring re-implementing certain parts of MapStructure in C++ space, which is kinda promising - and all code/aircraft/dialogs using the MapStructure framework (such as Gijs' ND) will automatically benefit from those changes if they're merged, because the key functionality is well-encapsulated there.

Which also kinda challenges your whole point about maintenance issues, because frameworks are typically just 500-1000 lines, while separating front-end and back-en code.

Like Thorsten suggested himself, LW is far from being a good example here - despite your point being totally valid (I kept saying that for years ...), but the main challenge there is not "framework-centric" development, but the whole development methodology adopted there - Thorsten recently mentioned that he thinks it could be re-architected within 4-6 weeks by a team of skilled Nasal coders - so we've had the same discussion behind the scenes. But Thorsten also said that the main issue is that he wasn't familiar with certain programming techniques when he started LW, so it very much "evolved" over time, and wasn't refactored in any major way - and Thorsten kept saying that "it works", so there's not much motivation to do something about it, which I kinda share - I am also more interested in other things, despite having tried cleaning up a few things there a while ago.

Overall, when it comes to major Nasal-based contributions like Advanced Weather (or Bombable) the main challenge is not the implementation language, but rather the underlying expertise, i.e. maths, physics, AI, combat etc - the language part is trivial, and good and maintainable Nasal code can definitely be written, just look at any code committed by AndersG, TheTom or Philosopher - the real challenge is that there's no major barrier to entry when it comes to writing Nasal code, and we're seeing a lot of mediocre code committed, often in terms of performance - but almost certainly in structural terms, i.e. stuff like code organization/reuse and refactoring/generalization. Which kinda comes with the territory, i.e. $FG_ROOT/Nasal not being reviewed/maintained any longer. Nasal is not the reason for this, it's just making the underlying problem more prominent, because we 1) have a lot of fgdata-level coding going on and 2) because we're not reviewing/refactoring stuff properly.

Like I said, the core Canvas system will certainly be extended and refined over time, but there's no sane way for us to deal with existing code, unless people adopt a framework-centric development model that would allow us to encapsulate and localize functionality - once that is the case, refining frameworks, re-implementing them and even phasing them out is straightforward - but as long as people continue with the current "method", we'll end up with a huge mess with Nasal code using different parts of the Canvas system is hugely inconsistent ways, without having anybody willing to do the ugly refactoring/porting work to use more recent Canvas APIs. And frankly, I am not going to volunteer cleaning up all that mess - it's like ThorstenB once said about Advanced Weather: it's simply too much to handle for a single person. Then again, we cannot expect non-coders to adopt and honor important concepts like "code reuse", "constant refactoring" and "encapsulation" without also providing working examples. As could be seen in the last few days, doing that actually scales very well - just look at all of omega95's recent postings in the canvas forum.

Extending the Canvas system is happening, and will continue to happen, but it's a process that will take months, and probably even years - until then, it makes sense to encapsulate common functionality in terms of frameworks, and use that come up with requirements for the Canvas system itself. Otherwise, we'd need a time machine ...

Now, debugging and profiling Nasal code is a different issue - and one that's being worked on, Philosopher has created several very promising patches, some of which we've been running to test/profile and debug code - none of this has been merged yet, but it's on the way.--Hooray (talk) 20:00, 24 July 2014 (UTC)

I certainly don't disagree that recent Canvas efforts have become a lot more organised and this has affected the quality of the code in a good way, but the fact still remains that there are too few useful tools to aid Nasal development, while there certainly are a ton for C++ frameworks. Also, while C++ is less forgiving than Nasal, it is also easier to maintain and debug using established techniques. I feel that the project would benefit a lot by encouraging existing contributors to move up to the core code base and have access to a plethora of data that otherwise would be wrapped in layers which don't contribute much to speed and ease of debugging. Of course, Nasal has an important place, but I think developing a large platform strictly within Nasal means asking for trouble in the future.

Adrian (talk) 20:40, 24 July 2014 (UTC)

I think the whole debate is kinda moot, because we're talking about a "perfect scenario" here - obviously, FlightGear would be in a great shape, if there was more C++ code doing certain things, instead of having mediocre workarounds in Nasal space for common use-cases, contributed by non-developers more often than not.

But the situation is quite the opposite, pace of core development cannot even be compared to the degree of base package development going on facilitated through Nasal/Canvas. And yeah, there are exceptionally awesome tools available for C++ code - but then again, how many of those do you currently see being applied to the existing FlightGear code base ?

Frankly, how much of it do you even consider being maintained ? Let's face it, there's a ton of legacy C++ code in both SG and FG that isn't even being refactored/maintained, despite great tools being available, no matter if it's clang, pork/oink or profiling/debugging tools like valgrind/gdb etc. Thus, I find the whole debate kinda theoretical in nature - because we already have a huge C++ code base that is nowhere close to being regularly maintained - in fact, Canvas is basically the primary counter-example for this currently.

Which kinda proves the point that having good tools/diagnostics is one thing, but not having the manpower/people to apply them is a very real bottleneck still. As I mentioned on the forum a while ago, Unifying the 2D rendering backend via canvas is going to be one of those few refactoring efforts where we're getting rid of a ton of legacy C++ code that hasn't been touched in years (usually, 3-5+ years: e.g. HUD, 2D panels, GUI, hard-coded instruments etc).

Also, just look at the situation we're facing when it comes to Reset & re-init, FlightGear Sessions or Initializing Nasal early - all of these are efforts that are significantly complicated by the fact that existing C++ APIs were never fully/widely adopted by other core developers, including core APIs like SGSubsystem itself introduced over a decade ago. We're currently seeing this in the scope of Nasal/Canvas frameworks, too.

David Megginson and a few others were extremely visionary back then when they introduced APIs for subsystems, but the way C++ core development happened, it took more than half a decade before even the SGPropertyChangeListener API started being adopted, all this despite having excellent reasons, tools and APIs available.

So it's not exactly like having frameworks is competing with C++ development at all - quite the opposite, people have been using Nasal, and Canvas, to augment C++ core development, especially in under-developed areas, for which we don't have any dedicated developers/maintainers.

I don't disagree with your sentiment about "encouraging C++ level contributions", but again - you're talking about a "perfect scenario", from which we're light-years away. People are using Nasal for a reason: there are very real bottlenecks and barriers to entry involved. When it comes to Nasal/Canvas, there's no "exclusive" design that makes the whole thing Nasal specific - it all works just across the property tree, Nasal is basically just the "glue" that ties it all together, but people could also be using python, perl, lua or even C++ to accomplish the same thing. The reason that we're developing certain things in Nasal/XML is simply that Canvas is not about a single use-case like a PFD, ND, CDU or EFB etc -it's agnostic to the way people are using it, the most specific use-case that is being explicitly added meanwhile is the Canvas GUI built on top of Canvas Widgets - but otherwise, it's technically the correct thing to do. For exactly all the same reasons that FireFox/Chrome are built around JavaScript and XUL/XML, too - including the GUI itself.

The primary challenge arising from contributions like LW/AW is the way they're structured and developed - i.e. Nasal is a much higher, and simpler, language than C/C++ - yet, people tend to use it like assembly code, without using available programming concepts like OO to structure their modules, which is why we end up with huge un-maintainable piece of spaghetti code that nobody with a coding background wants to touch. But you can find similar examples in existing C++ code, too - the whole init process is some of the worst mess in FlightGear, and that's been the case for years:

The original reset was a simple kludge that worked fine for a simple program; now that FlightGear is a lot more sophisticated, we need to refactor a bit rather than just holding things together with scotch tape and bubble gum. The problem is that right now we have two different approaches - a lot of stuff is handled cleanly in individual modules' update() methods, but a lot of stuff is still handled by hundreds of lines of BASIC-like, hard-to-read spaghetti code in main.cxx and fg_init.cxx (forget OO, it's not even functional programming). That was the right idea to get started - I'm not a fan of wasting time on an initial OO design that you won't end up using -- but now we need to finish cleaning it up. We need to make this a high priority. — David Megginson

Personally, I am not convinced that a weather subsystem even belongs into the main loop - and I would also like to see more C++ development going on, but as you know perfectly well, getting C++ code reviewed/discussed and committed is a painful and tedious process - simply due to the lack of manpower, which really is what it all boils down to: We're all motivated by our own little pet projects, we don't want to deal with chores like reviewing others' code, let alone work out a way to collaborate with them and their conflicting ideas - which applies not just to C++ development, but also Nasal. In the case of Canvas, debating the merits of C++ vs. Nasal use is kinda moot, because it's being primarily developed in C++ space - only stuff that isn't performance-critical is exposed to scripting space, usually for the sake of flexibility.

The Nasal/Canvas modules/bindings represent just a tiny fraction of the code involved here, and the challenge is elsewhere, usually in code written by aircraft developers, and not the actual Nasal/Canvas modules providing the APIs or "frameworks". We could go and kill of parts of the MapStructure framework tomorrow (or even all of it), and we would not need to touch a single aircraft/GUI dialog to ensure that things will continue to work - which doesn't apply to efforts pursued by people who don't follow a framework-centric approach.

Note that this has nothing to do with Nasal, it's just about design philosophies. We would be facing the same challenge if someone were to provide Python bindings, for which great tools are available - as long as people don't think in terms of long-term maintenance, they'll continue to make the same mistakes - regardless of the language being used.

As you may know, a few core developers even contemplated and discussed re-implementing existing instruments/HUDs on top of Canvas in C++ space, which would take away all the flexibility that people are currently embracing and enjoying-very much alike to the situation with generic/XML-configurable protocols vs. hard-coded I/O protocols, that usually haven't been maintained in years either (think MP). Note that this doesn't even involve Nasal at all - it's primarily an accessibility issue, where Nasal shines obviously in comparison to legacy C++ code that's not being maintained, but otherwise, this goes to show that Nasal/Canvas is not the problem, it just magnifies the issue by acting as a "booster" and technology-enabler that people are using without requiring a lot of expertise. --Hooray (talk) 21:16, 24 July 2014 (UTC)