Hackathon Proposal:Nasal GC Inspection

From FlightGear wiki
Jump to navigation Jump to search
Screen shot showing a Canvas GUI dialog rendering the new Property browser dialog based on porting the legacy PUI widget, showing GC stats
Cquote1.png I noticed a lot of stuttering and frame rate which with regular pattern was going from 30/40ms to almost 1 second. I switched off WS3.0 and the situation is better, but still I can see stutters.
— Salvatore De Astis[1]
(powered by Instant-Cquotes)
Cquote1.png The jitters/stutters are almost certainly a separate issue, most likely Nasal garbage collection.
— Stuart[2]
(powered by Instant-Cquotes)

Cquote1.png It's my understanding that this behaviour is exactly what most people are seeing of FG these days. The FGUK Saturday evening multiplayer flights are normally filled with people getting the random stutters and causing issues when trying any sort of formation flying (especially when it's the leader)

It's been how FG has been for a lot of users for several years at least I'd say. Not so much a problem when doing generic GA or Airliner operations but it can be an issue if you're flying something with higher performance or to tighter tolerances.

— Alex Barrett[3]
(powered by Instant-Cquotes)

Title: Nasal Garbage Collection Inspection

Potential mentors: Richard Harrison, James Turner
Old PUI dialog showing the Property browser in a custom SG/FG build that exposes Nasal internals via the fgfs property tree (listeners, timers and Nasal GC info)
Nasal, and particularly its garbage collector (GC), is known for having a certain impact on the frame rate, and frame spacing - however, there are performance issues other than Nasal code, and some Nasal related issues (e.g. poorly-written code) may contribute to masking such issues - so far, it's been impossible to easily disable Nasal (the scripting subsystem in FlightGear) entirely.

FlightGear's garbage collector This is a link to a Wikipedia article uses a so called mark/sweep collector This is a link to a Wikipedia article.

Nasal GC internals visualized by plotting fps, frame spacing and GC duration using gnuplot

The jitter/stuttering is caused by Nasal's "garbageCollect" method. Simple test: add a printf (apply attached "patch" to simgear) and you should see the "stutter" is synchronized with it. The garbage collector does a complete scan of all Nasal objects to detect and remove unreachable elements. So, the more Nasal data elements we have, the worse the jitter gets. Large Nasal data structures will eventually break every CPU. And since it's done in the context of normal Nasal calls, and not in a separate background thread, it directly affects the duration of our main update loop - hence frame rate. Not so nice.[4]

The event manager handles timers, and that's almost exclusively used by Nasal. Almost all the Nasal code runs in timers (except for property listeners). So read "events" as the total execution time for Nasal (timers). The timing data shown for the subsystem "nasal" only refers to the execution time of Nasal internal house keeping (i.e. some garbage collection), but not to the execution of actual Nasal code. Indeed misleading.[5]

Interested Parties: please add yourself if you are interested in working on this
Status: RFC

GC stuttering can be seen when using Nasal heavy features, like the FG1000.

nvidia nsight showing fgfs (Minimal Startup Profile, no scenery, with all Draw masks set), just running the FG1000 without PUI
Screen shot showing a diagram created with gnuplot to visualize the relationship between frame rate, frame spacing and GC induced pauses (exemplified by opening the FG1000 at ~ 03:00) - this causes a ton of GC pressure, and ends up running the GC more frequently than before
The current Nasal GC integration in FGNasalSys is leaking listeners, as well as references and objects, and may trigger callbacks to be invoked unnecessarily

frame stuttering definitely happens for me, a pause of anything between 20 to 50ms every time the current GC is invoked. I've timed this in the C++ and the values (for the F-15 are in the table below). The main loop has to waits for this so it definitely happens. The time taken will vary with CPU, the measurements below were taken with i7-4790k, 24gb RAM, R9-290

average msec 	24.65
max 	45.349
min 	1.488

The claim isn't hard to solidify - I've measured it using high resolution timers; however I accept that it might be hard to perceive; and doesn't affect Nasal that is coded to avoid GC (either using a lot of globals or being careful about local variables) Fly low and fast in the F-15 and look out of the side window (high zoom level makes it more apparent).[6]


1rightarrow.png See How the Nasal GC works for the main article about this subject.

see simgear/simgear/nasal/gc.c

Ideas: At some point in the future, the Nasal runtime would benefit from being upgraded with a concurrent and perhaps parallel garbage collector. The current stop-the-(Nasal)-world GC may inflict long delays in the simulation main loop.

For the time being, it would be a good idea to provide options to disable Nasal entirely, and/or the GC itself, so that people can better identify/track Nasal related issues.

Several years ago, ThorstenB provided patches that dumped some GC internals to the console (see [1] and [2] ) - that work could be adapted and generalized to publish the same stats to the property tree.


More recently, Richard H. added suppport for an experimental threaded GC (see SimGear commit 92a3c8bbd823ce2aee5cbe434cc0ea5c03c47f71 meanwhile reverted), those patches also included additional stats that could be enabled using preprocessor macros. In conjunction, these two patches could be used to "publish" the same stats at runtime in the property tree, possibly GC globals, but also per memory pool.

That way, the built-in XML logging system could be used to dump GC stats to an XML file for later evaluation. Also, another possibility would be to visualize GC stats using the Canvas system (via plotting).

Alternatively, a gnuplot file could be created to visualize GC stats using gnuplot.

diagram showing how a naive threaded approach to free dead blocks can reduce the total workload for the garbage collector

Another idea would be hooking up these properties to the built-in on-screen OSG stats: Extend FGStatsHandler to expose Nasal GC stats via osg::StatsHandler Not done Not done

Cquote1.png The built-in osgviewer stats can be extended with custom stats, that works by subclassing osg::StatsHandler, this is already done in $FG_SRC/src/Viewer/FGEventHandler.hxx#l14 (commit 130f581b18711b63138580d306a39c27b891178c)

The class can be extended to add your own stats via osgViewer::StatsHandler::addUserStatsLine()

You can even register totally custom stats via osg::Stats

The main suspects in this case are probably 1) scenery, 2) cockpit, 3) Nasal (GC - bottleneck() function in simgear/nasal )
For the Nasal stats, you'll probably want to register them in FGNasalSys::FGNasalSys() by accessing globals->viewer->
That should give you a better idea about what's going on there, and it's been suggested by core developers, too:

http://www.mail-archive.com/flightgear- ... 37823.html

Cquote1.png Another goal is to add more node bits (and a GUI dialog to control them) so

various categories of objects can be disabled during the update pass. This will
mean the direct hit of, say, AI models vs particles vs random trees can be
measured. Of course it won't account for resources (memory, textures) burned by
such things, but would still help different people identify slowness on their


— Hooray (Sun Feb 23). Re: New Aircraft: the Extra500.
(powered by Instant-Cquotes)
Required skills: Property tree, PropertyList XML File, Nasal, Canvas, Timers, Listeners
Learning Opportunities: