Howto:Use the system monitor: Difference between revisions

From FlightGear wiki
Jump to navigation Jump to search
(Disabled some "Caution" text that i think was distracting from the main content.)
 
(34 intermediate revisions by 7 users not shown)
Line 1: Line 1:
{{Portability Navbar}}
<!--
== Terms ==
{{FGCquote
|1= those systems are not "processes", but just "loops" - i.e. whatever the thing is called in the list, you should append "update-loop" to it - and some names are still misleading (events=Nasal).
|2= {{cite web
  | url    = {{forum url|p=264426}}
  | title  = <nowiki>Re: Su-15</nowiki>
  | author = <nowiki>Hooray</nowiki>
  | date  = Nov 13th, 2015
  }}
}}
{{FGCquote
|1= FG is basically using cooperative multi-tasking-imagine it like a team of people (15+), all of whom need to finish their work and hand it over to the next person, so that a single frame can be produced - with some people (subsystems) being allowed to redo/check their work several times in a row (fdm, autopilot), while others will only get a single opportunity to update their internal state.
Finally, it is this "team of people" that will produce a single frame - so whenever "someone" is too slow, it will end up causing stuttering. No matter the reason, scenery/terrain, Nasal etc - at some point, another subsystem will have to wait, which means fewer frames created per second.
|2= {{cite web
  | url    = {{forum url|p=264313}}
  | title  = <nowiki>Re: Su-15</nowiki>
  | author = <nowiki>Hooray</nowiki>
  | date  = Nov 13th, 2015
  }}
}}
-->
== The system monitor ==
== The system monitor ==


[[File:System-performance-monitor-fg260.png|400px|thumb|Screen shot showing the system performance monitor available in FG 2.6.0+]]
[[File:System-performance-monitor-fg260.png|400px|thumb|Screen shot showing the system performance monitor available in FG 2.6.0+]]
<!--
{{Caution|
{{FGCquote
|1= the built-in performance monitor, and especially the GUI (it being used in PUI/PLIB)- is adding to the performance overhead you're seeing
|2= {{cite web
  | url    = {{forum url|p=243288}}
  | title  = <nowiki>Re: FlightGear permanently damages the performance of comput</nowiki>
  | author = <nowiki>Hooray</nowiki>
  | date  = May 18th, 2015
  }}
}}
{{FGCquote
|1= you will want to use the http/telnet interfaces to watch the performance stats, because the GUI dialog using PUI/Nasal is dead-slow, and even adding to the overhead you are seeing
|2= {{cite web
  | url    = {{forum url|p=264313}}
  | title  = <nowiki>Re: Su-15</nowiki>
  | author = <nowiki>Hooray</nowiki>
  | date  = Nov 13th, 2015
  }}
{{FGCquote
|1= "events" is the subsystem that triggers/runs Nasal timers, and any Nasal code (anywhere), may trigger the GC (garbage collector) - to see if that's the case, you can use the "GC tracker" patch in the sigar code we have.
|2= {{cite web
  | url    = {{forum url|p=258607}}
  | title  = <nowiki>Re: How to tell if you are CPU or GPU limited (split)</nowiki>
  | author = <nowiki>Hooray</nowiki>
  | date  = Sep 23rd, 2015
  }}
}}
}}
{{FGCquote
|1= You can get pretty accurate stats using the performance monitor - it will write everything to the property tree. To see for yourself, open the performance monitor dialog and then use the property browser to watch the performance monitor branch in the property tree, you can learn more by inspecting $FG_ROOT/Nasal/performance_monitor/monitor.nas
As you can see there, the switch for enabling the monitor is /sim/performance-monitor/enabled (bool)
This will work regardless of the PUI dialog, and you can view/access all stats (for each subsystem!) without showing the PUI dialog (=better performance)
For a list of subsystems, go to /sim/performance-monitor/subsystems
Note that you can also use the http (browser) or telnet protocols to do this while extending the logging protocol
|2= {{cite web
  | url    = {{forum url|p=258579}}
  | title  = <nowiki>Re: How to tell if you are CPU or GPU limited (split)</nowiki>
  | author = <nowiki>Hooray</nowiki>
  | date  = Sep 23rd, 2015
  }}
}}
}}
-->


In FlightGear >= 2.6.0, use the menu option '''Debug''' => '''Monitor System Performance''' to analyze stutters.
In FlightGear >= 2.6.0, use the menu option '''Debug''' => '''Monitor System Performance''' to analyze stutters.
Line 11: Line 85:
Also, imagine a system producing 100 frames in 0.5 seconds, then blocking completely for another 0.5 seconds:
Also, imagine a system producing 100 frames in 0.5 seconds, then blocking completely for another 0.5 seconds:
The fps display would still show "100fps", which seems great. But the 0.5 second stutter cause the visual performance to be terrible.  
The fps display would still show "100fps", which seems great. But the 0.5 second stutter cause the visual performance to be terrible.  
When evaluating simulation performance, don't get fooled by the frame rate. What's really important to us is the "worst case frame latency". Even if the system is producing a huge average of 100 frames per second, it can still look absolutely crappy, if only a single frame took more than 20-30ms, since that's immediately visible to the human eye. So, we're building a real-time system here, and 20-30ms is our timing constraint.


That's why it is preferable to display "(worst-case) frame spacing" instead of fps (View => Display Options => Show frame spacing). The frame spacing for the previous example would show "500ms", while a system producing 100 frames with perfectly even spacing would show "10ms".
That's why it is preferable to display "(worst-case) frame spacing" instead of fps (View => Display Options => Show frame spacing). The frame spacing for the previous example would show "500ms", while a system producing 100 frames with perfectly even spacing would show "10ms".


So, frame spacing is a much better property to judge visual quality than just watching fps.
So, frame spacing is a much better property to judge visual quality than just watching fps.
The new GUI dialog is available in the menu: "'''Debug'''" => "'''Monitor system performance'''".


* Only statistics on our "subsystems" are shown. That's the FG modules running in our single-threaded main loop. Everything outside the main loop isn't shown, i.e. there are no statistics on the GPU or OSG rendering threads (use the OSG statistics screen for these).
* Only statistics on our "subsystems" are shown. That's the FG modules running in our single-threaded main loop. Everything outside the main loop isn't shown, i.e. there are no statistics on the GPU or OSG rendering threads (use the OSG statistics screen for these).
Line 24: Line 96:
* Title bar shows average and worst case frame rate. In a perfect world, all frames should be spaced evenly, so both values should be almost identical. When frames are produced unevenly, then a larger difference between the worst and average frame rate is visible. Needless to say, we should try to get FG to deliver frames evenly, so movements look smooth.
* Title bar shows average and worst case frame rate. In a perfect world, all frames should be spaced evenly, so both values should be almost identical. When frames are produced unevenly, then a larger difference between the worst and average frame rate is visible. Needless to say, we should try to get FG to deliver frames evenly, so movements look smooth.


* Main table shows statistics on the individual subsystems. Helps to identify how much tme certain systems currently consume, or which are responsible for jitters resulting in uneven frame rates.
* Main table shows statistics on the individual subsystems. Helps to identify how much time certain systems currently consume, or which are responsible for jitters resulting in uneven frame rates.


* A bit background on the FG subsystems may be necessary though to really judge what's going on:
* A bit background on the FG subsystems may be necessary though to really judge what's going on:
** For example, you'll see the "nasal" subsystem consuming almost no time at all, so it looks great. However, almost all the nasal code runs in timers, and timers are driven by the "events" subsystem. So, to judge Nasal performance, you'll mainly need to look at "events" (and yes, you'll see time being consumed and jitters being produced there, which is related due to Nasal's current MARK/SWEEP Garbage Collector implementation, i.e. see: [[Improving Nasal]]).
** For example, you'll see the "nasal" subsystem consuming almost no time at all, so it looks great. However, almost all the nasal code runs in timers, and timers are driven by the "events" subsystem. So, to judge Nasal performance, you'll mainly need to look at "events" (and yes, you'll see time being consumed and jitters being produced there, which is related due to Nasal's current MARK/SWEEP Garbage Collector implementation, i.e. see: [[Improving Nasal]]).
** the props (property tree) system make invoke Nasal callbacks via listeners
** The "tile-manager" an "ai-model" subsystems may also contain time consumed by the Nasal engine, since each scenery tile and each AI/MP aircraft may trigger a Nasal script ("load"/"unload" hooks).
** The "tile-manager" an "ai-model" subsystems may also contain time consumed by the Nasal engine, since each scenery tile and each AI/MP aircraft may trigger a Nasal script ("load"/"unload" hooks).
** The GUI system is another subsystem which may trigger Nasal hooks
** The GUI system is another subsystem which may trigger Nasal hooks
** While it is true that the current Nasal mark/sweep garbage collector has been shown to cause occasional stuttering under certain circumstances, the system monitor in its current form is '''not''' suitable to tell you the difference between a badly coded "rogue" script causing stutter or the stuttering being really caused by the Nasal GC. Also, it is worth noting that following good coding practices may actually help decrease the overall pressure on the GC, i.e. by keeping the number of references to be traversed low and pretty constant, but also by not calling unneeded code too frequently using timers or listeners.


* At the moment, it is unfortunately not yet possible to get timing statistics for individual Nasal scripts or sub modules, but this is something that has been requested by a number of users and so we are looking into making this possible eventually.
* At the moment, it is unfortunately not yet possible to get timing statistics for individual Nasal scripts or sub modules, but this is something that has been requested by a number of users and so we are looking into making this possible eventually.
Line 36: Line 110:


Anyway, this alone isn't improving anything yet, but having a GUI to conveniently monitor performance will hopefully help us to see where exactly we're having issues - and help us to improve...
Anyway, this alone isn't improving anything yet, but having a GUI to conveniently monitor performance will hopefully help us to see where exactly we're having issues - and help us to improve...
== Discussed Enhancements ==
* decouple Nasal events from the "events" subsystem and use a new "events-nasal" instead {{forum link|p=164715}}
* show average frame-spacing effect per subsystem as a percentage
* show memory consumption per subsystem (overload the new operator)
* plot graphs for the various metrics using the new [[Canvas]] system
* track Nasal GC pressure (invocations) and total memory allocated [http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg31762.html]


= Debugging stuttering =
= Debugging stuttering =


To put it simple, vertex shader performance is affected by the amount of detail visible in the geometry of scene (number of vertices visible), and of course shader complexity itself, while fragment performance is affected by screen resolution(or widow size if you run it windowed) and shader complexity as well. They both affect framerate in the end, since the frame needs to be finished before being put up for display.
To put it simple, vertex shader performance is affected by the amount of detail visible in the geometry of scene (number of vertices visible), and of course shader complexity itself, while fragment performance is affected by screen resolution(or window size if you run it windowed) and shader complexity as well. They both affect framerate in the end, since the frame needs to be finished before being put up for display.
Frame delay might be affected by a lot of other stuff, but it's a better measure of smoothness since it displays the "longest" time it had to wait for a frame in a certain amount of time.
Frame delay might be affected by a lot of other stuff, but it's a better measure of smoothness since it displays the "longest" time it had to wait for a frame in a certain amount of time.
There might be some performance lost in high detail scenes, due to the culling (visibility check, and hiding of faces facing away from the camera, or being behind other faces, or too small to display (openscenegraph is smart about that one and it won't "display" faces/parts of objects that would end up too small)) that needs to be done on the scene before it goes through the shaders.
There might be some performance lost in high detail scenes, due to the culling (visibility check, and hiding of faces facing away from the camera, or being behind other faces, or too small to display (openscenegraph is smart about that one and it won't "display" faces/parts of objects that would end up too small)) that needs to be done on the scene before it goes through the shaders.
Line 45: Line 126:
To test if it's a graphics performance issue, you'd do a couple of tests.:
To test if it's a graphics performance issue, you'd do a couple of tests.:


First load up the ufo, oevr detailed scenery, remain static, don't move the view around. Check framerate and frame delay. If they're consistent with each other it might be a graphics performance issue, if not you can be sure there's some other subsystem that does background processing and needs to finish it's stuff.
First load up the ufo, over detailed scenery, remain static, don't move the view around. Check framerate and frame delay. If they're consistent with each other it might be a graphics performance issue, if not you can be sure there's some other subsystem that does background processing and needs to finish its stuff.


Move the view around, do you get stutters? If you do, then there's some pressure on your video card's RAM. As it needs to load the new geometry and textures that come into view, it might not have enough space left, so it might need to offload the "old" ones. Even like this framerate should stabilise once you stop moving the view around, or once the rate of movement is slow enough so that the video card can keep up.
Move the view around, do you get stutters? If you do, then there's some pressure on your video card's RAM. As it needs to load the new geometry and textures that come into view, it might not have enough space left, so it might need to offload the "old" ones. Even like this framerate should stabilise once you stop moving the view around, or once the rate of movement is slow enough so that the video card can keep up.
Line 54: Line 135:


In short: if the framerate '''varies along with the Frame Delay''', then it's a graphics performance issue. If framerate is pretty stable, and Frame Delay '''varies more or less wildly''' I'd suspect some other subsystem being the culprit and causing more or less periodic delays as it needs to do it's stuff.
In short: if the framerate '''varies along with the Frame Delay''', then it's a graphics performance issue. If framerate is pretty stable, and Frame Delay '''varies more or less wildly''' I'd suspect some other subsystem being the culprit and causing more or less periodic delays as it needs to do it's stuff.
== Profiling ==
For developers, and people building from source, FlightGear versions >= 2.9 provide real profiling support using Google PerfTools, for details please see [[Built-in Profiler]].


= Interpreting the OSG on screen stats =
= Interpreting the OSG on screen stats =
 
{{Main article|OSG on screen stats}}
[[Image:OSG-OnScreen-Stats.png|thumb|400px|Screenshot showing the OSG on screen stats]]
[[Category:Howto]]
 
[[Category:Scenery enhancement]]
You can also try to play with osg's frame statistics, you can switch that on from the debug menu.
 
Being CPU and draw limited means the yellow bar is long. And the blue one grows when cull happens
to be a problem.
 
A simple help is to switch on the onscreen stats in the viewer or flightgear  and see if the yellow bar is longer than the orange one. A yellow bar longer than the orange one means that the cpu is not able to saturate the gpu.
Again, beware this does in no way mean that you should write complicated shaders to saturate the gpu! This rather means that the geometry setup/state change time - the yellow one - must decrease!
 
Once your orange bar is longer than the yellow one, you can start to care for
the shaders and their execution. When thinking about shaders, keep in mind that different GPU's perform very
different on the same shader.
 
And regarding all that, even if your box already is gpu bound this does not mean that other driver/hardware combinations are gpu bound too. Always: As much as needed and as few as possible.
 
== References ==
* http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg35202.html
* http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg35917.html

Latest revision as of 07:13, 16 October 2020


The system monitor

Screen shot showing the system performance monitor available in FG 2.6.0+


In FlightGear >= 2.6.0, use the menu option Debug => Monitor System Performance to analyze stutters.

In a perfect world, min/max/mean values should be almost identical for every subsystem - and standard deviation should be almost 0.

Larger differences / high standard deviations result in a sloppy simulation and stuttering movements, though they'll hardly influence the frames-per-second value at all.

Also, imagine a system producing 100 frames in 0.5 seconds, then blocking completely for another 0.5 seconds: The fps display would still show "100fps", which seems great. But the 0.5 second stutter cause the visual performance to be terrible.

When evaluating simulation performance, don't get fooled by the frame rate. What's really important to us is the "worst case frame latency". Even if the system is producing a huge average of 100 frames per second, it can still look absolutely crappy, if only a single frame took more than 20-30ms, since that's immediately visible to the human eye. So, we're building a real-time system here, and 20-30ms is our timing constraint.

That's why it is preferable to display "(worst-case) frame spacing" instead of fps (View => Display Options => Show frame spacing). The frame spacing for the previous example would show "500ms", while a system producing 100 frames with perfectly even spacing would show "10ms".

So, frame spacing is a much better property to judge visual quality than just watching fps.

  • Only statistics on our "subsystems" are shown. That's the FG modules running in our single-threaded main loop. Everything outside the main loop isn't shown, i.e. there are no statistics on the GPU or OSG rendering threads (use the OSG statistics screen for these).
  • Title bar shows average and worst case frame rate. In a perfect world, all frames should be spaced evenly, so both values should be almost identical. When frames are produced unevenly, then a larger difference between the worst and average frame rate is visible. Needless to say, we should try to get FG to deliver frames evenly, so movements look smooth.
  • Main table shows statistics on the individual subsystems. Helps to identify how much time certain systems currently consume, or which are responsible for jitters resulting in uneven frame rates.
  • A bit background on the FG subsystems may be necessary though to really judge what's going on:
    • For example, you'll see the "nasal" subsystem consuming almost no time at all, so it looks great. However, almost all the nasal code runs in timers, and timers are driven by the "events" subsystem. So, to judge Nasal performance, you'll mainly need to look at "events" (and yes, you'll see time being consumed and jitters being produced there, which is related due to Nasal's current MARK/SWEEP Garbage Collector implementation, i.e. see: Improving Nasal).
    • the props (property tree) system make invoke Nasal callbacks via listeners
    • The "tile-manager" an "ai-model" subsystems may also contain time consumed by the Nasal engine, since each scenery tile and each AI/MP aircraft may trigger a Nasal script ("load"/"unload" hooks).
    • The GUI system is another subsystem which may trigger Nasal hooks
    • While it is true that the current Nasal mark/sweep garbage collector has been shown to cause occasional stuttering under certain circumstances, the system monitor in its current form is not suitable to tell you the difference between a badly coded "rogue" script causing stutter or the stuttering being really caused by the Nasal GC. Also, it is worth noting that following good coding practices may actually help decrease the overall pressure on the GC, i.e. by keeping the number of references to be traversed low and pretty constant, but also by not calling unneeded code too frequently using timers or listeners.
  • At the moment, it is unfortunately not yet possible to get timing statistics for individual Nasal scripts or sub modules, but this is something that has been requested by a number of users and so we are looking into making this possible eventually.
  • Statistics data is only collected while the dialog is open (or you manually set /sim/performance-monitor/enabled).

Anyway, this alone isn't improving anything yet, but having a GUI to conveniently monitor performance will hopefully help us to see where exactly we're having issues - and help us to improve...

Discussed Enhancements

  • decouple Nasal events from the "events" subsystem and use a new "events-nasal" instead [2] This is a link to the FlightGear forum.
  • show average frame-spacing effect per subsystem as a percentage
  • show memory consumption per subsystem (overload the new operator)
  • plot graphs for the various metrics using the new Canvas system
  • track Nasal GC pressure (invocations) and total memory allocated [3]

Debugging stuttering

To put it simple, vertex shader performance is affected by the amount of detail visible in the geometry of scene (number of vertices visible), and of course shader complexity itself, while fragment performance is affected by screen resolution(or window size if you run it windowed) and shader complexity as well. They both affect framerate in the end, since the frame needs to be finished before being put up for display. Frame delay might be affected by a lot of other stuff, but it's a better measure of smoothness since it displays the "longest" time it had to wait for a frame in a certain amount of time. There might be some performance lost in high detail scenes, due to the culling (visibility check, and hiding of faces facing away from the camera, or being behind other faces, or too small to display (openscenegraph is smart about that one and it won't "display" faces/parts of objects that would end up too small)) that needs to be done on the scene before it goes through the shaders.

To test if it's a graphics performance issue, you'd do a couple of tests.:

First load up the ufo, over detailed scenery, remain static, don't move the view around. Check framerate and frame delay. If they're consistent with each other it might be a graphics performance issue, if not you can be sure there's some other subsystem that does background processing and needs to finish its stuff.

Move the view around, do you get stutters? If you do, then there's some pressure on your video card's RAM. As it needs to load the new geometry and textures that come into view, it might not have enough space left, so it might need to offload the "old" ones. Even like this framerate should stabilise once you stop moving the view around, or once the rate of movement is slow enough so that the video card can keep up.

Fly straight and level, at reasonable speeds, keeping the view fixed. Do you get stutters? Is framerate consistent with frame delay? If it is, and you get low fps, again it might be a graphics issue, as new random objects come into view and/or the trees shader creates new ones in the distance. If not I's still suspect another susbsytem being the culprit.

You can repeat these with some aircraft, see if and how things change.

In short: if the framerate varies along with the Frame Delay, then it's a graphics performance issue. If framerate is pretty stable, and Frame Delay varies more or less wildly I'd suspect some other subsystem being the culprit and causing more or less periodic delays as it needs to do it's stuff.

Profiling

For developers, and people building from source, FlightGear versions >= 2.9 provide real profiling support using Google PerfTools, for details please see Built-in Profiler.

Interpreting the OSG on screen stats

1rightarrow.png See OSG on screen stats for the main article about this subject.