Troubleshooting crashes: Difference between revisions

From FlightGear wiki
Jump to navigation Jump to search
No edit summary
m (Switch to the {{forum url}} and {{forum link}} templates for all forum links.)
(39 intermediate revisions by 8 users not shown)
Line 1: Line 1:
'''Targeted FlightGear versions: 2.6, 2.8, 2.10, 2.12, 3.0+ '''
{{Troubleshooting}}
{{note|'''This article is valid for FlightGear versions: 2.6, 2.8, 2.10, 2.12, 3.0+'''. Help us in keeping this article up to date! If you're facing some crash not listed here and manage to solve it, '''please add it to a proper section''', so that other can benefit from your experience!}}


== Motivation ==
Crashes are never easy to troubleshoot. It's very likely that you'll need to experiment a bit to find out the cause, or at least to provide useful feedback. In fact issues depend a lot on things like hardware components, driver support and so on. Developers can't work on your machine, so you'll need to provide them with some specific information. This article will guide you in that.
This article should be suitable for pretty much any OSG-based version beyond 2.6/2.8 - the 2.10 reference/disclaimer below is just about a specific GLSL issue that got introduced (and now fixed in 2.11+) in FG 2.10 in particular, the point of the whole write-up is to provide a dedicated place for people to go with these sorts of problems, and to reduce the workload on our part obviously - and provide some guidance to end-users facing such issues, allowing them to experiment a little and provide more specific feedback. The idea is to keep it up-to-date for upcoming FG versions, too - including 2.12 and beyond.
In other words, any and all help is appreciated, so please feel free to get involved and contribute to the article by adding your own findings, suggestions and techniques. Your help is really appreciated!


If you should find yourself having to disable additional settings in order to further maximize FlightGear performance, please do add your findings to this article, so that others can benefit.  
If not only FlightGear, but the whole system crashes, or freezes, please refer to [[System Crashes]].


Our hope is that we can come up with a fairly safe subset of FlightGear settings that should work for 99% of our users, even on  moderately old hardware. This would enable us to change the FlightGear defaults accordingly, and use these settings as a safe fallback alternative - so that FlightGear should no longer just crash for people without certain hardware/features.  
If in the end you like bug hunting, please consider helping in [[Volunteer#Pre-Release Testers|testing future releases]].


There's a trend to expose more and more information via the property tree, to make the simulator increasinly runtime-configurable, and to allow subsystems to be re-initialized at runtime, so being able to start up FlightGear in some form of "safe" mode is generally a good thing and useful for anybody helping with troubleshooting on the forums, because it helps us exclude many potential problems. In the long run, such a profile could also be used as the foundation for a [[FlightGear Benchmark]] or some scripted regression suite to run FlightGear in a [[FlightGear Headless|headless]] mode to help with release preparations.
== Known problems ==
Before diving into the real troubleshooting, first try to see if one of the following is your case. This could be a huge time saver!


== The Issue with 2.10 ==
'''If your problem is not listed here, please consider adding it after you have solved it! Others like you will be grateful!'''
<font color="red">'''Note:''' FlightGear 2.10 users seeing crashes during simulator startup (without seeing the actual aircraft/scenery) with very old graphics cards that lack GLSL (shader) support should know that there used to be a bug in 2.10 that broke support for graphics cards without any shader support. In other words, if you are able to run other OpenGL software, but cannot even display the about dialog in FG 2.10 using the minimal startup profile detailed below, it is almost certainly your graphics card's lack of shader support causing the issue, because FlightGear 2.10 tries to copy graphics card information to the property tree for debugging purposes, to allow end-users to provide better troubleshooting reports - ironically, this very feature introduced crashes for all 2.10 users without any GLSL support. This issue has been fixed in FlightGear 2.11+ </font>


== Crashes in general ==
=== Optimus/Hybrid graphics technologies for laptops ===
If FlightGear crashes (or looks/performs very badly/slowly), it's possible that this is due a number of reasons, such as limited hardware resources (CPU, GPU, RAM etc) or insufficient driver support - so crashes are not necessarily due to a faulty program. FlightGear does work for hundreds of forum users, and usually without any form of sophisticated prior setup or configuration. So if something doesn't work "out of the box" it's usually some local or system-specific issue. In general, you are unlikely to encounter any major bugs when using a release - if you don't mind having to deal with bugs and technical issues, we'd like to invite you to help us test prereleases (so called "release candidates"/RCs) and nightly builds - or directly build from git.
If FlightGear crashes with
Unknown exception in the main loop. Aborting...
Possible cause: Not enough space
and you happen to have a laptop with Optimus Technology (NVidia) or Hybrid Graphics technology (ATI) onboard, you'll find some help at [[Troubleshooting performance issues#Operating system and driver setup]]. (Note that the problem is treated there because it's mostly cause of low performance.)


== Possible causes ==
=== World Scenery 2.0 and 32-bit machines ===
It's just as well possible that the FlightGear default settings are not supported with your hardware/drivers, so that the process will be killed by the OS because it's trying to do something not supported by your computer. If you are experiencing full system crashes, you should see [[System Crashes]] instead.
[[World Scenery]] 2.0 (the one with [[CORINE]] and [[OpenStreetMap]] data) highly raised memory requirements with respect to the previous version. Basically, with 4GB of RAM once you could safely use a visibility of a hundred kilometers; with Scenery 2.0, 4GB is barely enough for 10 km. A 32-bit machine can't use more than that memory, and simply kills FlightGear when it asks for more. To verify this, reduce your visibility (in-sim with ''View > Level of Detail adjustment'' and set ''Bare'' to 5000 m) and see if it stops crashing. You can also check memory consumption with the tools provided by your OS.


By default, FlightGear 2.6+ will assume a certain runtime environment and certain GPU features which may not be suppported by most older cards (e.g. shader support), such as GeForce 6/7 generation hardware and most older hardware  (older than 4-5 years), so that it can be expected that FG will probably not work on such computers "out of the box" and may get terminated by your operating system eventually, i.e. the default startup settings will need to be customized accordingly, to disable all unsupported default features. See [[Settings for slower graphics cards]], [[Problematic Video Cards]] and [[FlightGear Hardware Recommendations]] for more details.
Sadly, the solution is keeping a low visibility, or moving to a 64-bit machine with at least 8GB of RAM, which can provide even 45km of visibility. If you're feeling "hackish", you can try the solution reported at [[Unexpected Crashes with Scenery ver.2]].


Also, make sure you've got the newest driver for your video card installed and running without errors. A suggestion for a thorough updating procedure can be found [[About graphic drivers|here]].
=== Old graphics cards and FlightGear 2.10 ===
If you're using FlightGear 2.10, you're seeing crashes during simulator startup (without seeing the actual aircraft/scenery), and you're using an old enough graphics card, it is possible that your crash is due to a bug in that FG version and to your card lacking [[Shader|GLSL]] support. This bug would prevent using FG even with shaders off! Ironically, this bug comes from a feature introduced to help troubleshooting bugs.


Before proceeding any further, please make sure that you are actually able to run other OpenGL-based software/simulators/games, unrelated to FlightGear!
So, if you're on 2.10 and think your card could be old enough, try the minimal startup configuration below: if it keeps crashing, that's definitely it. The bug was solved immediately later (ver. 2.11)


The startup profile detailed below disables tons of features that are known to have a massive impact on frame rate and overall performance by making FG look plain bad and crude, as such, it gives you a rough idea of the maximum performance fgfs is able to achieve on your system - which in turn, allows you then to re-enable individual settings one-by-one, and see their impact on framerate/performance - so that you can determine which settings are particularly problematic - so that you can decide to come up with a custom profile to make the 60-100 fps goal for example - as you can see, on a 2009-era laptop, you could hope for ~790 fps and 3ms of frame spacing when in the "ugly" mode - so by hand-picking my settings and tweaking them as needed, you'll be getting beyond 50 fps, even with 75-85% of eye candy enabled, with fairly moderate GPU hardware
== Troubleshooting your crash ==
FlightGear crashes are due to a number of reasons, not necessarily to bugs in its code. After all, it works "out of the box" for most users, so if something's broken it's often a system-specific issue, and yet again, it could be a system configuration that the developers couldn't test - [[Volunteer#Pre-Release Testers|you can help with that]]!)


== Running FlightGear with minimal settings ==
It's just as well possible that the FlightGear default settings are not supported with your hardware/drivers. Since version 2.6, FlightGear will assume support for certain features that hardware older than 4-5 years might not give (especially graphics cards), so the default settings need to be customized to disable all unsupported features. See [[Troubleshooting performance issues]], [[Problematic Video Cards]] and [[FlightGear Hardware Recommendations]] for more details.
In such cases, it is important to start up FlightGear with minimum settings, and start re-enabling features step by step to see what settings are supported by your hardware. You may want to [[Howto:Improve_framerates#Debugging.2Flog level|raise the FlightGear log level]], so that you get to see where the program stops starting/working, and what it was doing before it got terminated.


The following settings are really just intended to provide a '''minimal startup profile''', with ALL eye candy disabled by default - so that FG doesn't crash due to lack of certain features (OpenGL, shaders etc) and you get a "bare bone" FlightGear window. They use the UFO, as it is the least complex aircraft (the default aircraft is the Cessna 172) and a suitable scenery location to rule out aircraft or scenery as a cause for crashes.
=== Preliminary check ===
Make sure you've got the newest driver for your video card installed and running without errors. A suggestion for a thorough updating procedure can be found at [[Graphics drivers configuration]]. '''Before proceeding any further, please make sure that you are actually able to run other OpenGL-based software/simulators/games, unrelated to FlightGear!'''


Once you have launched FlightGear with these minimal settings, you can enable all graphics options one by one. If an option incurs a huge performance hit, take into consideration whether it is necessary for your flying experience or not.  
=== The standard procedure ===
Given the above, it is important to start up FlightGear with a '''minimal startup profile''', with ''all'' eye candy and features disabled (so that FG won't crash due to lack of support for them) and start re-enabling them step by step to see what's supported by your hardware. You may want to [[Commonly used debugging tools#Debug level setting|raise the FlightGear log level]], so that you get to see where the program stops starting/working, and what it was doing before it got terminated. This procedure will also help in [[troubleshooting performance issues]].


The performance impact can be evaluated by using:
These settings use the simple UFO and a suitable scenery location to rule out aircraft or scenery as a cause for crashes. Don't even touch the aircraft and location settings (airport, azimuth, offset) until you have really fixed everything else already. Working step by step is the only real way, so don't bite off more than you can chew.
* The frame rate counter (frames created per second)
* The frame spacing indicator (latencies between rendered frames)
* The [[Howto:Use the system monitor|built-in system monitor]] (can be inspected via telnet/http using [[Telnet usage|--telnet=5900]])
* The OSG on-screen stats
* The [[Built-in Profiler]]


It makes sense to keep using the ufo until you have optimized all settings accordingly and found out what's working and what isn't, i.e. don't even touch the aircraft settings or the location settings (airport, azimuth, offset) until you have really optimized everything else already.
If you find it looks like the Windows "compatibility mode" that you boot into after a crash, then you understand perfectly what we're doing here. However it's not as straightforward to do, so if in trouble {{forum link|f=17|text=ask for support at the forum}}!


=== Performance targets ===
=== Minimal startup profile ===
Generally, try to optimize FlightGear for at least 30+ fps before changing to a regular aircraft and/or location! Realistically, the ufo is not very representative - so that it would be better to aim for something in between 50-100 fps prior to switching to a more complex aircraft like the c172p for example. For instance, aircraft like the A380, IAR80, Concorde or 777 are known to require fairly powerful hardware.
{{Note|If you find any other settings causing a massive improvement of performance/compatibility, please feel free to add those below, so that others can benefit from your findings.}}


Thus, if you are not getting frame rates '''much''' higher than 100 fps with the minimal startup profile (without using frame throttling, i.e. locking the refresh rate at a certain frequency, obviously), then using complex aircraft (777,787) or scenery (KSFO, KLSV etc) will be next to impossible with your system.
<gallery mode="packed">
Tweaking-nvidia-settings1.png|Tweaking nvidia settings to obtain higher frame rates by disabling vsync to blank
Tweaking-nvidia-settings2.png|Tweaking Nvidia Powermizer to obtain higher frame rates
</gallery>


=== Minimal Startup Profile ===
As mentioned, the point of the following settings is to disable '''everything''' that could have an effect on performance, compatibility and stability, to ensure that we get a working FlightGear window up and running with no eye candy at all, and maximum frame rate. Once that is working, features can be re-enabled step by step.
Depending on the hardware specs of the target platform, adjusting the threading mode of FG/OSG may also help: [[Howto:Activate multi core and multi GPU support]]. (If you don't know what we're talking about, you can safely leave it as it is.)


Depending on the hardware specs of the target platform, adjusting the threading mode of FG/OSG may also help: [[Howto:Activate multi core and multi GPU support]].
You can directly use a custom [[Fgfsrc]] file for the following sections or parse the following lines to the console (separated by one empty character, e.g. a space) after "FGFS" or set the respective options in [[Fgrun]] or in [[FlightGear configuration via XML|one of FlightGear's XML configuration files]].  
 
You can directly use a custom [[Fgfsrc]] file for the following sections or parse the following lines to the console (separated by one empty character) after "FGFS" or set the respective options in [[Fgrun]] or in [[FlightGear configuration via XML|one of FlightGear's XML configuration files]].  


Please make sure to rename/delete your autosave and fgfsrc files prior to trying these settings, or your local settings would possibly override these settings.
Please make sure to rename/delete your autosave and fgfsrc files prior to trying these settings, or your local settings would possibly override these settings.
{{Startup Profile|name=minimal|version=3.2}}


<pre>
{{Tip|Beginning with FlightGear 3.1+, you can also toggle individual scenegraph traversal masks on/off (these can be changed at runtime using the [[Property browser]]:
# --ignore-autosave # uncomment this for FlightGear versions >= 2.99
* <nowiki>--prop:/sim/sceneryloaded-override=1</nowiki>
--airport=ksfo
--offset-distance=4000
--offset-azimuth=90
--altitude=500
--heading=0
--model-hz=60
--disable-random-objects
--prop:/sim/rendering/texture-compression=off
--prop:/sim/rendering/quality-level=0
--prop:/sim/rendering/shaders/quality-level=0
--disable-ai-traffic
--prop:/sim/ai/enabled=0
--aircraft=ufo
--disable-sound
--prop:/sim/rendering/random-vegetation=0
--prop:/sim/rendering/random-buildings=0
--disable-specular-highlight
--disable-ai-models
--disable-clouds
--disable-clouds3d
# --disable-textures
--fog-fastest
--visibility=5000
--disable-distance-attenuation
--disable-enhanced-lighting
--disable-real-weather-fetch
--prop:/sim/rendering/particles=0
</pre>
 
{{Note|Beginning with FlightGear 3.1+, you can also toggle individual scenegraph traversal masks on/off (these can be changed at runtime using the [[Property browser]]:
* <nowiki>--prop:browser=/sim/rendering/draw-mask</nowiki>
* <nowiki>--prop:/sim/rendering/draw-mask/terrain=0</nowiki>
* <nowiki>--prop:/sim/rendering/draw-mask/terrain=0</nowiki>
* <nowiki>--prop:/sim/rendering/draw-mask/aircraft=0</nowiki>
* <nowiki>--prop:/sim/rendering/draw-mask/aircraft=0</nowiki>
Line 96: Line 67:
* <nowiki>--prop:/sim/rendering/draw-mask/clouds=0</nowiki>
* <nowiki>--prop:/sim/rendering/draw-mask/clouds=0</nowiki>
}}
}}
If you're running on Windows, you can use the following batch file to start the simulator using the minimal profile. Open Notepad, copy and paste the following lines and save the file on the Desktop (name it ''fgminimal.bat''); change ''C:\Program Files\FlightGear'' to the directory you installed FlightGear in and change ''Win64'' to ''Win32'' if you are using the 32-bit version.
<pre>
@echo off
"C:\Program Files\FlightGear\bin\Win64\fgfs.exe" --fg-root="C:\Program Files\FlightGear\data" --ignore-autosave --airport=ksfo --offset-distance=4000 --offset-azimuth=90 --altitude=500 --heading=0 --model-hz=60 --disable-random-objects --prop:/sim/rendering/texture-compression=off --prop:/sim/rendering/quality-level=0 --prop:/sim/rendering/shaders/quality-level=0 --disable-ai-traffic --prop:/sim/ai/enabled=0 --aircraft=ufo --disable-sound --prop:/sim/rendering/random-vegetation=0 --prop:/sim/rendering/random-buildings=0 --disable-specular-highlight --disable-ai-models --disable-clouds --disable-clouds3d --fog-fastest --visibility=5000 --disable-distance-attenuation --disable-enhanced-lighting --disable-real-weather-fetch --prop:/sim/rendering/particles=0 --prop:/sim/rendering/multi-sample-buffers=1 --prop:/sim/rendering/multi-samples=2
</pre>
Double-click the batch file to start FlightGear.


Using these settings, frame rates between 400-500 fps and a steady frame spacing (latency) of 3-11 ms, can be achieved on a notebook from 2007 (1024x768). Disabling wireframe mode lowers the framerate by about 50-80 fps. The window should be up and running (fully initialized) within 3-5 seconds, even on moderately old hardware (absent a navcache rebuild).
Using these settings, frame rates between 400-500 fps and a steady frame spacing (latency) of 3-11 ms, can be achieved on a notebook from 2007 (1024x768). Disabling wireframe mode lowers the framerate by about 50-80 fps. The window should be up and running (fully initialized) within 3-5 seconds, even on moderately old hardware (absent a navcache rebuild).
Line 105: Line 83:
As can be seen, additional resources can be freed by flying in areas without scenery (ocean tiles), by using simple aircraft like the ufo, and by using a HUD instead of a 2D/3D cockpit panel.
As can be seen, additional resources can be freed by flying in areas without scenery (ocean tiles), by using simple aircraft like the ufo, and by using a HUD instead of a 2D/3D cockpit panel.


<!--
=== Minimal Rembrandt Startup Profile ===
=== Minimal Rembrandt Startup Profile ===
<pre>
<syntaxhighlight lang="text">
--enable-rembrandt
--enable-rembrandt
--prop:/sim/rendering/rembrandt/ambient-occlusion-buffers=false
--prop:/sim/rendering/rembrandt/ambient-occlusion-buffers=false
Line 138: Line 117:
--disable-real-weather-fetch  
--disable-real-weather-fetch  
--prop:/sim/rendering/particles=0
--prop:/sim/rendering/particles=0
</syntaxhighlight>
(Again, the point not being to look good, but just to start up without crashing or extreme lag)
-->
=== Reporting incompatible default settings ===
Once you have located and identified settings that seem to cause reproducible issues with your hardware, it would be a good idea to report these settings using the issue tracker, so that these can hopefully be made optional in the upcoming release: http://flightgear-bugs.googlecode.com/
=== Tracking RAM utilization ===
[[File:Fg-3.2-minimal-settings.png|400px|Screen shot showing FlightGear 3.2 using the minimal startup profile with frame spacing and frame counter displayed.]]
{{WIP}}


</pre>
{{FGCquote
  |My measurements (below) suggest that terrain mesh (as loaded at "bare" <br/>
LOD) is the dominant memory user, with only small contributions from the <br/>
aircraft, terrain textures, and random objects/trees/buildings.
  |{{cite web |url=http://sourceforge.net/p/flightgear/mailman/message/32445574/
    |title=<nowiki>Re: [Flightgear-devel] Memory consumption reduction</nowiki>
    |author=<nowiki>Rebecca N. Palmer</nowiki>
    |date=<nowiki>2014-06-11</nowiki>
  }}
}}
 
{{FGCquote
  |In Ubuntu 64-bit, measured using System Monitor after it stabilizes (~1min)<br/>
Except as stated, c172p stationary on KSFO 28R, night, Terrasync <br/>
scenery, default settings (regional textures, LOD range 1500/9000/30000)<br/>
baseline: 2.3-2.4GB<br/>
Changing shader effects level or enabling/disabling random <br/>
objects/trees/buildings: no significant change<br/>
Global texture set: 2.2GB<br/>
DDS texture set: 2.2GB<br/>
Global texture set, no Textures.high: 2.1GB (and some missing effects <br/>
textures, as they don't have the "try Textures.high first, then <br/>
Textures" fallback mechanism)<br/>
LOD range 1500/9000/10000: 1.4GB<br/>
LOD range 1500/2000/30000: 2.3GB<br/>
LOD range 500/9000/30000: 2.3GB<br/>
1.0 scenery (from flightgear-data 3.0 due to above issue): 1.1GB<br/>
Ocean location: 0.75GB<br/>
F-14: 2.1GB<br/>
777-200: 2.6GB<br/>


(Again, the point not being to look good, but just to start up without crashing or extreme lag)
  |{{cite web |url=http://sourceforge.net/p/flightgear/mailman/message/32445574/
    |title=<nowiki>Re: [Flightgear-devel] Memory consumption reduction</nowiki>
    |author=<nowiki>Rebecca N. Palmer</nowiki>
    |date=<nowiki>2014-06-11</nowiki>
  }}
}}


=== Further assistance ===
[[File:Fg-3.2-RAM-utilization-in-minimal-mode.png|right|500px|htop showing FlightGear 3.2 RAM utilization when in minimal startup mode]]


== Further assistance ==
If you need help with the whole process, please get in touch via the FG forums.
If you need help with the whole process, please get in touch via the FG forums.


Line 155: Line 180:
* use the about dialog to provide other important information
* use the about dialog to provide other important information


=== Reporting incompatible default settings ===
== Troubleshooting Performance Issues ==
Once you have located and identified settings that seem to cause reproducible issues with your hardware, it would be a good idea to report these settings using the issue tracker, so that these can hopefully be made optional in the upcoming release: http://flightgear-bugs.googlecode.com/
 
If you're able to build from source and don't mind using development looks like the built-in profiler, it would be interesting to see comparison of both builds - ideally using a simple test case with everything else disabled - you could use a pre-recorded flight and/or a flight recorder type to come up with a "test flight". From our standpoint, it would help tremendously if all features that don't seem to have an effect could be completely disabled, including complex aircraft and scenery/locations. In other words, if you can reproduce the problem using "bare" minimum settings, the resulting log file will be much easier to process.
 
There's a built-in profiler which you can use to create these profiles: http://wiki.flightgear.org/Built-in_Profiler
You would then want to use two different build directories, where SG/FG build settings would be identical, but using an older version of OSG: http://wiki.flightgear.org/Building_using_CMake#Multiple_build_directories
 
For the sake of simplicity there's a so called "minimal startup profile" that you can use: http://wiki.flightgear.org/Howto:Debugging_FlightGear_Crashes#Minimal_startup_profile
While unlikely, it would be great if the issue could be reproduced that way - but more likely than not, you'll have to re-add a few features and change a few settings, e.g. by using a different location.
 
Like I said, you could then use the replay system to create an test flight than can be easily reproduced - to get going more quickly, you can also use the built-in route manager to create a simple flight plan and fly the whole thing on autopilot: http://wiki.flightgear.org/Instant_Replay
 
You can use time warp mode to speed up simulation time and finish more quickly.
 
Admittedly, all this may seem a bit intimidating and tedious, but once you have such a setup working, you can reuse it over and over again for different startup profiles and provide profiling logs for each.
We do have a number of people interested in adding features to support benchmarking/profiling workflows natively:
 
http://wiki.flightgear.org/FlightGear_Benchmark
http://wiki.flightgear.org/Testing
 
If that's something you'd like to pursue, feel free to get in touch - it is definitely a worthwhile thing, even regardless of any OSG specific issues, as it will also help with unrelated performance issues.
Obviously, there's a bit of a steep learning curve involved, which is why it's normally only core developers that build multiple versions against different OSG versions and run the profiler.
But once you are able to build from source, you have already completed the most difficult step - everything else is fairly straightforward in comparison - as as you can probably tell, creating test flights using the route manager and/or flight recorder also isn't exactly rocket science.
 
However, before you do anything, I'd suggest to check first if you've possibly changed your driver, or if you can install a newer version


=== Debugging Segfaults & Obtaining Backtraces ===
== Debugging Segfaults & Obtaining Backtraces ==
{{Note|This section is currently Linux/gdb specific}}
{{Note|This section is currently Linux/gdb specific. Any help in updating it to become less platform/OS/debugger-specific would be greatly appreciated!}}


A '''segfault''' is another word for "crash", it's a so called "segmentation fault", where the program is doing things that it wasn't designed to do, such as accessing invalid memory for example, so that the operating system will terminate the process because its behavior is no longer valid/correct. These '''segfaults''' will typically happen due to coding errors, either in FlightGear or one of its dependencies (libraries like SimGear, plib, OpenSceneGraph or OpenAL).  
A '''segfault''' is another word for "crash", it's a so called "segmentation fault", where the program is doing things that it wasn't designed to do, such as accessing invalid memory for example, so that the operating system will terminate the process because its behavior is no longer valid/correct. These '''segfaults''' will typically happen due to coding errors, either in FlightGear or one of its dependencies (libraries like SimGear, plib, OpenSceneGraph or OpenAL).  
A typical segfault may look like this:
<syntaxhighlight lang="text">
Initializing splash screen
Splash screen progress init
Tungsten Graphics, Inc.
Mesa DRI R200 (RV250 4C66) x86/MMX/SSE2 TCL DRI2
1.3 Mesa 9.1.1
Program received signal SIGSEGV, Segmentation fault.
0xb71467e6 in __strlen_sse2_bsf () from /usr/lib/libc.so.6
</syntaxhighlight>
The last two lines are basically showing the actual crash, anything shown earlier is normally just an indicator how far the code could proceed shortly before the crash actually occurred. 


In order to see what was going on when the crash occurred, you'll need to build a binary with '''debugging symbols''' enabled/included, usually this includes FlightGear and SimGear - and possibly even OSG (OpenSceneGraph, normally only if your crash happens inside rendering related code).
In order to see what was going on when the crash occurred, you'll need to build a binary with '''debugging symbols''' enabled/included, usually this includes FlightGear and SimGear - and possibly even OSG (OpenSceneGraph, normally only if your crash happens inside rendering related code).
Line 177: Line 239:
A good way to to provide a reproducible test case is using the [[Replay]]/flight recorder system to create a flight recorder tape that triggers the crash, alternatively, consider providing a "flight plan" that triggers the crash - ideally, without it being aircraft specific, i.e. reproducible using aircraft like the ufo
A good way to to provide a reproducible test case is using the [[Replay]]/flight recorder system to create a flight recorder tape that triggers the crash, alternatively, consider providing a "flight plan" that triggers the crash - ideally, without it being aircraft specific, i.e. reproducible using aircraft like the ufo


Thus, you'll want to reconfigure your SimGear/FlightGearGear ($SG_SRC & $FG_SRC) source trees using the '''-CMAKE_BUILD_TYPE=Debug''' switch, for details please refer to: [[Building using CMake#Debug_Builds]]. It is a good idea not to touch your existing build trees for this, but instead create an additional directory hierarchy for your debugging binary, please see [[Building using CMake#Multiple build directories]] for details.  
Thus, you'll want to reconfigure your SimGear/FlightGear ($SG_SRC & $FG_SRC) source trees using the '''-CMAKE_BUILD_TYPE=RelWithDebInfo''' switch, for details please refer to: [[Building using CMake#Debug_Builds]]. It is a good idea not to touch your existing build trees for this, but instead create an additional directory hierarchy for your debugging binary, please see [[Building using CMake#Multiple build directories]] for details.  


Once you have rebuilt and relinked SimGear and FlightGear, you'll want to use a debugger like '''gdb''' to run your new binary. It does help to have a good way to reproduce a crash, such as using certain startup/runtime settings. For the sake of simplicity it is usually a good idea to disable all unrelated features and subsystems/settings, this includes complex aircraft and complex scenery locations (airports) if possible. For details, refer to the minimal startup profile detailed in this article.
Once you have rebuilt and relinked SimGear and FlightGear, you'll want to use a debugger like '''gdb''' to run your new binary. It does help to have a good way to reproduce a crash, such as using certain startup/runtime settings. For the sake of simplicity it is usually a good idea to disable all unrelated features and subsystems/settings, this includes complex aircraft and complex scenery locations (airports) if possible. For details, refer to the minimal startup profile detailed in this article.
Line 192: Line 254:
# only if that isn't conclusive, use  '''thread apply all bt full''' instead to get a full backtrace for all threads (background tasks)
# only if that isn't conclusive, use  '''thread apply all bt full''' instead to get a full backtrace for all threads (background tasks)
# please use the issue tracker to post your backtrace
# please use the issue tracker to post your backtrace
This is what a typical backtrace may look like:
<syntaxhighlight lang="gdb">(gdb) thread apply all bt
Thread 1 (Thread 0xb66c4740 (LWP 10908)):
#0  0xb71467e6 in __strlen_sse2_bsf () from /usr/lib/libc.so.6
#1  0x0886c080 in copy_string (s=s@entry=0x0)
    at /home/cesar/compilation/pkg/simgear/2.10.0/src/simgear-2.10.0/simgear/props/props.cxx:161
#2  0x08870550 in set_string (val=0x0, this=0x8dd55b8)
    at /home/cesar/compilation/pkg/simgear/2.10.0/src/simgear-2.10.0/simgear/props/props.cxx:524
#3  SGPropertyNode::setStringValue (this=0x8dd55b8, value=0x0)
    at /home/cesar/compilation/pkg/simgear/2.10.0/src/simgear-2.10.0/simgear/props/props.cxx:1587
#4  0x08326ae1 in (anonymous namespace)::GeneralInitOperation::run (this=0x8bdea48, gc=0x8be4310)
    at /home/cesar/compilation/pkg/flightgear/2.10.0/src/flightgear-2.10.0/src/GUI/gui.cxx:123
#5  0x085c2b7b in flightgear::GraphicsContextOperation::operator() (this=0x8bdea48, gc=0x8be4310)
    at /home/cesar/compilation/pkg/flightgear/2.10.0/src/flightgear-2.10.0/src/Viewer/WindowSystemAdapter.cxx:41
#6  0xb78ca246 in osg::GraphicsOperation::operator()(osg::Object*) () from /usr/lib/libosg.so.80
#7  0xb78c7784 in osg::GraphicsContext::runOperations() () from /usr/lib/libosg.so.80
#8  0xb7b29fe2 in osgViewer::ViewerBase::renderingTraversals() () from /usr/lib/libosgViewer.so.80
#9  0xb7b2707c in osgViewer::ViewerBase::frame(double) () from /usr/lib/libosgViewer.so.80
#10 0x085c4841 in fgOSMainLoop ()
    at /home/cesar/compilation/pkg/flightgear/2.10.0/src/flightgear-2.10.0/src/Viewer/fg_os_osgviewer.cxx:286
#11 0x082221fa in fgMainInit (argc=32, argv=0xbffff494)
    at /home/cesar/compilation/pkg/flightgear/2.10.0/src/flightgear-2.10.0/src/Main/main.cxx:339
#12 0x081e71ee in main (argc=32, argv=0xbffff494)
    at /home/cesar/compilation/pkg/flightgear/2.10.0/src/flightgear-2.10.0/src/Main/bootstrap.cxx:251
</syntaxhighlight>
As can be seen, the first few lines basically show exactly what happened, and where it happened, while all the subsequent lines are basically showing the callgraph, i.e. how the piece of code got triggered. For example, FlightGear started in bootstrap.cxx line 251, then called into main.cxx (line 339), at which point the osgviewer setup took place. Next, there are a few lines that  do not contain a lot of interesting info, because those are all library calls into OSG - things are starting to be interesting again around line #5, where FlightGear specific code is getting called again. As can be seen in this particular example, the actual segfault was triggered in a SimGear operation here, which was due to a NULL pointer.
At first, this may look a bit confusing and even intimidating, but it does make sense to people familiar with C++ and FlightGear/SimGear - so even if you cannot understand a single thing, just post such output on the issue tracker, so that others can have a look. We'll ask for additional information if necessary.
However, at times it may be nearly impossible to see what actually goes in in FG these days simply because there's too many disparate parts involved here. For example, in the rare instance that our [[Nasal]] scripting interpreter could lead to a segfault, we could potentially see where the C code was using a debugger like gdb – but that wouldn't tell us where we were in the Nasal code.
And that similarly applies to many things, especially things that "jump" boundaries here. Another prime example is property tree listeners - Nasal listeners don't have a backtrace beyond the C code, even if it is triggered directly from a Nasal setprop()/.setValue(), because it passes through C and C++ stack frames and then back into Nasal.
Thus, valgrind/gdb/cachegrind/perftools may be too low-level to be meaningful in most cases, i.e. just look at what's going on in the scenery/aircraft department, here just identifying "hot spots" in terms of code/functions would be a red herring, because the real culprit is typically not at all in the run-time code, but the underlying scenegraph data, such as highly complex textures, 3D models or terrain/scenery.
This can now be easily verified by using Zakalawe's new draw-masks to switch off aircraft/scenery/clouds or AI model rendering.
Having acess to gdb/valgrind etc is one thing, being able to actually run and interpret the results is a completely different thing.
Thus, it is more promising -in the long term- to optionally expose certain info at run-time, so that users can access it to see how their aircraft/scenery/feature/configuration behaves in comparison to some other aircraft/scenery/feature/configuration.
Also, when it comes to using tools like gdb -and especially valgrind- the main challenge is that FlightGear is primarily a GUI application, and it doesn't lend itself it being debugged/profiled or leak-tested in an isolated fashion at all.
For example, for valgrind runs to be useful and straightforward, we would need to be able to minimize what's done inside fg_init.cxx so that we can disable things not needed, but also even run FlightGear in a [[FlightGear Headless|"headless" mode]] without a visible GUI window, so that such tests could be run directly on the build server as part of a regression testing suite. Otherwise, even just running fgfs in valgrind for 10 minutes sim time may turn out to take hours due to the nature of valgrind, i.e. it being an emulator.
valgrind et all are great if you know how to use them, but for end users, these tools are too fine-grained, we would ideally support some per-subsystem or per-feature granularity for end users.
== Using AddressSanitizer ==
{{FGCquote
  |AddressSanitizer does a similar job to Valgrind with less overhead <br/>
(~2.5x memory use, and nearly full speed), but requires recompiling; I <br/>
haven't seen the crash under it yet (or at all today), if anyone does <br/>
please post your log.
  |{{cite web |url=http://sourceforge.net/p/flightgear/mailman/message/32749837/
    |title=<nowiki>Re: [Flightgear-devel] crash in SGPropertyNode::fireValueChanged</nowiki>
    |author=<nowiki>Rebecca Palmer</nowiki>
    |date=<nowiki>2014-08-22</nowiki>
  }}
}}
<syntaxhighlight lang="bash">
#this is for Linux with llvm-3.4, libsqlite3-dev, flite1-dev, libhtsengine-dev; it will probably work on Mac (with possibly different  ASAN_SYMBOLIZER_PATH) but not Windows
#simgear
cmake ../../git/simgear -DCMAKE_BUILD_TYPE=RelWithDebInfo
-DCMAKE_C_FLAGS="-fsanitize=address -fno-omit-frame-pointer -O2 -g"
-DCMAKE_CXX_FLAGS="-fsanitize=address -fno-omit-frame-pointer -O2 -g"
-DCMAKE_SHARED_LINKER_FLAGS="-fsanitize=address -fno-omit-frame-pointer
-O2 -g" -DCMAKE_VERBOSE_MAKEFILE=1 -DSIMGEAR_SHARED=ON
make
sudo make install
#flightgear
cmake ../../git/flightgear -DCMAKE_BUILD_TYPE=RelWithDebInfo
-DCMAKE_C_FLAGS="-fsanitize=address -fno-omit-frame-pointer -O2 -g"
-DCMAKE_CXX_FLAGS="-fsanitize=address -fno-omit-frame-pointer -O2 -g"
-DCMAKE_VERBOSE_MAKEFILE=1 -DSIMGEAR_SHARED=ON -DENABLE_IAX=OFF
-DENABLE_FGCOM=OFF -DSYSTEM_SQLITE=ON
make
sudo make install
#run
ASAN_SYMBOLIZER_PATH=/usr/lib/llvm-3.4/bin/llvm-symbolizer
ASAN_OPTIONS="symbolize=1 alloc_dealloc_mismatch=0" fgfs [options] 2>
asan_log.txt
#post asan_log.txt to developers mailing list and/or the issue tracker
</syntaxhighlight>
== The original idea and future developments ==
Our hope is that we can come up with a fairly safe subset of FlightGear settings that should work for 99% of our users, even on  moderately old hardware. This would enable us to change the FlightGear defaults accordingly, and use these settings as a safe fallback alternative - so that FlightGear should no longer just crash for people without certain hardware/features.
There's a trend to expose more and more information via the property tree, to make the simulator increasingly runtime-configurable, and to allow subsystems to be re-initialized at runtime, so being able to start up FlightGear in some form of "safe" mode is generally a good thing and useful for anybody helping with troubleshooting on the forums, because it helps us exclude many potential problems. In the long run, such a profile could also be used as the foundation for a [[FlightGear Benchmark]] or some scripted regression suite to run FlightGear in a [[FlightGear Headless|headless]] mode to help with release preparations, and [[FGCanvas]].


== Related ==
== Related ==
* http://flightgear.org/forums/viewtopic.php?f=37&t=17199&p=165503#p163684
* {{forum url|p=163684}}
* [[System Crashes]]
* [[System Crashes]]
* [[Hardware Recommendations]]
* [[Hardware Recommendations]]
* [[Settings for slower graphics cards]]
* [[Troubleshooting performance issues]]
* [[Howto:Improve framerates]]
* [[Problematic Video Cards]]
* [[Problematic Video Cards]]


[[Category:Howto|Debugging FlightGear Crashes]]
[[Category:Howto|Debugging FlightGear Crashes]]
[[Category:Troubleshooting]]

Revision as of 20:37, 9 June 2019

NoteThis article is valid for FlightGear versions: 2.6, 2.8, 2.10, 2.12, 3.0+. Help us in keeping this article up to date! If you're facing some crash not listed here and manage to solve it, please add it to a proper section, so that other can benefit from your experience!

Crashes are never easy to troubleshoot. It's very likely that you'll need to experiment a bit to find out the cause, or at least to provide useful feedback. In fact issues depend a lot on things like hardware components, driver support and so on. Developers can't work on your machine, so you'll need to provide them with some specific information. This article will guide you in that.

If not only FlightGear, but the whole system crashes, or freezes, please refer to System Crashes.

If in the end you like bug hunting, please consider helping in testing future releases.

Known problems

Before diving into the real troubleshooting, first try to see if one of the following is your case. This could be a huge time saver!

If your problem is not listed here, please consider adding it after you have solved it! Others like you will be grateful!

Optimus/Hybrid graphics technologies for laptops

If FlightGear crashes with

Unknown exception in the main loop. Aborting... 
Possible cause: Not enough space

and you happen to have a laptop with Optimus Technology (NVidia) or Hybrid Graphics technology (ATI) onboard, you'll find some help at Troubleshooting performance issues#Operating system and driver setup. (Note that the problem is treated there because it's mostly cause of low performance.)

World Scenery 2.0 and 32-bit machines

World Scenery 2.0 (the one with CORINE and OpenStreetMap data) highly raised memory requirements with respect to the previous version. Basically, with 4GB of RAM once you could safely use a visibility of a hundred kilometers; with Scenery 2.0, 4GB is barely enough for 10 km. A 32-bit machine can't use more than that memory, and simply kills FlightGear when it asks for more. To verify this, reduce your visibility (in-sim with View > Level of Detail adjustment and set Bare to 5000 m) and see if it stops crashing. You can also check memory consumption with the tools provided by your OS.

Sadly, the solution is keeping a low visibility, or moving to a 64-bit machine with at least 8GB of RAM, which can provide even 45km of visibility. If you're feeling "hackish", you can try the solution reported at Unexpected Crashes with Scenery ver.2.

Old graphics cards and FlightGear 2.10

If you're using FlightGear 2.10, you're seeing crashes during simulator startup (without seeing the actual aircraft/scenery), and you're using an old enough graphics card, it is possible that your crash is due to a bug in that FG version and to your card lacking GLSL support. This bug would prevent using FG even with shaders off! Ironically, this bug comes from a feature introduced to help troubleshooting bugs.

So, if you're on 2.10 and think your card could be old enough, try the minimal startup configuration below: if it keeps crashing, that's definitely it. The bug was solved immediately later (ver. 2.11)

Troubleshooting your crash

FlightGear crashes are due to a number of reasons, not necessarily to bugs in its code. After all, it works "out of the box" for most users, so if something's broken it's often a system-specific issue, and yet again, it could be a system configuration that the developers couldn't test - you can help with that!)

It's just as well possible that the FlightGear default settings are not supported with your hardware/drivers. Since version 2.6, FlightGear will assume support for certain features that hardware older than 4-5 years might not give (especially graphics cards), so the default settings need to be customized to disable all unsupported features. See Troubleshooting performance issues, Problematic Video Cards and FlightGear Hardware Recommendations for more details.

Preliminary check

Make sure you've got the newest driver for your video card installed and running without errors. A suggestion for a thorough updating procedure can be found at Graphics drivers configuration. Before proceeding any further, please make sure that you are actually able to run other OpenGL-based software/simulators/games, unrelated to FlightGear!

The standard procedure

Given the above, it is important to start up FlightGear with a minimal startup profile, with all eye candy and features disabled (so that FG won't crash due to lack of support for them) and start re-enabling them step by step to see what's supported by your hardware. You may want to raise the FlightGear log level, so that you get to see where the program stops starting/working, and what it was doing before it got terminated. This procedure will also help in troubleshooting performance issues.

These settings use the simple UFO and a suitable scenery location to rule out aircraft or scenery as a cause for crashes. Don't even touch the aircraft and location settings (airport, azimuth, offset) until you have really fixed everything else already. Working step by step is the only real way, so don't bite off more than you can chew.

If you find it looks like the Windows "compatibility mode" that you boot into after a crash, then you understand perfectly what we're doing here. However it's not as straightforward to do, so if in trouble ask for support at the forum This is a link to the FlightGear forum.!

Minimal startup profile

Note  If you find any other settings causing a massive improvement of performance/compatibility, please feel free to add those below, so that others can benefit from your findings.


Depending on the hardware specs of the target platform, adjusting the threading mode of FG/OSG may also help: Howto:Activate multi core and multi GPU support. (If you don't know what we're talking about, you can safely leave it as it is.)

You can directly use a custom Fgfsrc file for the following sections or parse the following lines to the console (separated by one empty character, e.g. a space) after "FGFS" or set the respective options in Fgrun or in one of FlightGear's XML configuration files.

Please make sure to rename/delete your autosave and fgfsrc files prior to trying these settings, or your local settings would possibly override these settings.

Note  The following FlightGear startup profile assumes that you have a $FG_ROOT environment variable set up, or that you are explicitly setting fg-root using the --fg-root command line argument, this startup profile is intended to be put into your Fgfsrc file or to be used when starting FlightGear from the command line.

For FlightGear 2020.3 LTS and later versions: you can simply copy and paste the profile in Flightgear Qt launcher > Settings tab > Additional Settings .

This is a minimum startup profile. After using it, you can turn up settings one by one as performance allows - and observe your performance bottlenecks. See Hardware recommendations for discussion of performance bottlenecks.

The profile listed below is

  • name: minimal
  • version: 3.2
  • description: n/a
# --ignore-autosave # uncomment this for FlightGear versions >= 2.99
--disable-terrasync
--disable-splash-screen
--airport=ksfo
--offset-distance=4000
--offset-azimuth=90
--altitude=500
--heading=0
--model-hz=60
--disable-random-objects
--prop:/sim/rendering/texture-compression=off
--prop:/sim/rendering/quality-level=0
--prop:/sim/rendering/shaders/quality-level=0
--disable-ai-traffic
--prop:/sim/ai/enabled=0
--aircraft=ufo
--disable-sound
--prop:/sim/rendering/random-vegetation=0
--prop:/sim/rendering/random-buildings=0
--disable-specular-highlight
--disable-ai-models
--disable-clouds
--disable-clouds3d
# --disable-textures
--fog-fastest
--visibility=5000
--disable-distance-attenuation
--disable-real-weather-fetch
--prop:/sim/rendering/particles=0
--prop:/sim/rendering/multi-sample-buffers=1
--prop:/sim/rendering/multi-samples=2
--prop:/sim/rendering/draw-mask/clouds=false
--prop:/sim/rendering/draw-mask/aircraft=false
--prop:/sim/rendering/draw-mask/models=false
--prop:/sim/rendering/draw-mask/terrain=false
--disable-random-vegetation
--disable-random-buildings
--disable-horizon-effect



Tip  Beginning with FlightGear 3.1+, you can also toggle individual scenegraph traversal masks on/off (these can be changed at runtime using the Property browser:
  • --prop:/sim/sceneryloaded-override=1
  • --prop:/sim/rendering/draw-mask/terrain=0
  • --prop:/sim/rendering/draw-mask/aircraft=0
  • --prop:/sim/rendering/draw-mask/models=0
  • --prop:/sim/rendering/draw-mask/clouds=0

If you're running on Windows, you can use the following batch file to start the simulator using the minimal profile. Open Notepad, copy and paste the following lines and save the file on the Desktop (name it fgminimal.bat); change C:\Program Files\FlightGear to the directory you installed FlightGear in and change Win64 to Win32 if you are using the 32-bit version.

@echo off
"C:\Program Files\FlightGear\bin\Win64\fgfs.exe" --fg-root="C:\Program Files\FlightGear\data" --ignore-autosave --airport=ksfo --offset-distance=4000 --offset-azimuth=90 --altitude=500 --heading=0 --model-hz=60 --disable-random-objects --prop:/sim/rendering/texture-compression=off --prop:/sim/rendering/quality-level=0 --prop:/sim/rendering/shaders/quality-level=0 --disable-ai-traffic --prop:/sim/ai/enabled=0 --aircraft=ufo --disable-sound --prop:/sim/rendering/random-vegetation=0 --prop:/sim/rendering/random-buildings=0 --disable-specular-highlight --disable-ai-models --disable-clouds --disable-clouds3d --fog-fastest --visibility=5000 --disable-distance-attenuation --disable-enhanced-lighting --disable-real-weather-fetch --prop:/sim/rendering/particles=0 --prop:/sim/rendering/multi-sample-buffers=1 --prop:/sim/rendering/multi-samples=2

Double-click the batch file to start FlightGear.

Using these settings, frame rates between 400-500 fps and a steady frame spacing (latency) of 3-11 ms, can be achieved on a notebook from 2007 (1024x768). Disabling wireframe mode lowers the framerate by about 50-80 fps. The window should be up and running (fully initialized) within 3-5 seconds, even on moderately old hardware (absent a navcache rebuild).

FlightGear 2.10 bare bones on 2006-era hardware (NVIDIA 7600GO)
FG 2.12 zero eyecandy mode on 2009-era hardware (NVIDIA 260M)
FG 2.12 minimal eye candy mode on 2009-era hardware (NVIDIA 260M)

As can be seen, additional resources can be freed by flying in areas without scenery (ocean tiles), by using simple aircraft like the ufo, and by using a HUD instead of a 2D/3D cockpit panel.


Reporting incompatible default settings

Once you have located and identified settings that seem to cause reproducible issues with your hardware, it would be a good idea to report these settings using the issue tracker, so that these can hopefully be made optional in the upcoming release: http://flightgear-bugs.googlecode.com/

Tracking RAM utilization

Screen shot showing FlightGear 3.2 using the minimal startup profile with frame spacing and frame counter displayed.

WIP.png Work in progress
This article or section will be worked on in the upcoming hours or days.
See history for the latest developments.
Cquote1.png My measurements (below) suggest that terrain mesh (as loaded at "bare"

LOD) is the dominant memory user, with only small contributions from the
aircraft, terrain textures, and random objects/trees/buildings.


— Rebecca N. Palmer (2014-06-11). Re: [Flightgear-devel] Memory consumption reduction.
(powered by Instant-Cquotes)
Cquote2.png
Cquote1.png In Ubuntu 64-bit, measured using System Monitor after it stabilizes (~1min)

Except as stated, c172p stationary on KSFO 28R, night, Terrasync
scenery, default settings (regional textures, LOD range 1500/9000/30000)
baseline: 2.3-2.4GB
Changing shader effects level or enabling/disabling random
objects/trees/buildings: no significant change
Global texture set: 2.2GB
DDS texture set: 2.2GB
Global texture set, no Textures.high: 2.1GB (and some missing effects
textures, as they don't have the "try Textures.high first, then
Textures" fallback mechanism)
LOD range 1500/9000/10000: 1.4GB
LOD range 1500/2000/30000: 2.3GB
LOD range 500/9000/30000: 2.3GB
1.0 scenery (from flightgear-data 3.0 due to above issue): 1.1GB
Ocean location: 0.75GB
F-14: 2.1GB
777-200: 2.6GB


— Rebecca N. Palmer (2014-06-11). Re: [Flightgear-devel] Memory consumption reduction.
(powered by Instant-Cquotes)
Cquote2.png
htop showing FlightGear 3.2 RAM utilization when in minimal startup mode

Further assistance

If you need help with the whole process, please get in touch via the FG forums.

When doing that, please fully document the whole process, i.e. including:

  • startup settings (and fgfsrc if used)
  • error messages and warnings (console output)
  • screen shots
  • screen shots showing frame rate, frame spacing, performance monitor, osg on-screen stats (see Howto:Use the system monitor )
  • use the about dialog to provide other important information

Troubleshooting Performance Issues

If you're able to build from source and don't mind using development looks like the built-in profiler, it would be interesting to see comparison of both builds - ideally using a simple test case with everything else disabled - you could use a pre-recorded flight and/or a flight recorder type to come up with a "test flight". From our standpoint, it would help tremendously if all features that don't seem to have an effect could be completely disabled, including complex aircraft and scenery/locations. In other words, if you can reproduce the problem using "bare" minimum settings, the resulting log file will be much easier to process.

There's a built-in profiler which you can use to create these profiles: http://wiki.flightgear.org/Built-in_Profiler You would then want to use two different build directories, where SG/FG build settings would be identical, but using an older version of OSG: http://wiki.flightgear.org/Building_using_CMake#Multiple_build_directories

For the sake of simplicity there's a so called "minimal startup profile" that you can use: http://wiki.flightgear.org/Howto:Debugging_FlightGear_Crashes#Minimal_startup_profile While unlikely, it would be great if the issue could be reproduced that way - but more likely than not, you'll have to re-add a few features and change a few settings, e.g. by using a different location.

Like I said, you could then use the replay system to create an test flight than can be easily reproduced - to get going more quickly, you can also use the built-in route manager to create a simple flight plan and fly the whole thing on autopilot: http://wiki.flightgear.org/Instant_Replay

You can use time warp mode to speed up simulation time and finish more quickly.

Admittedly, all this may seem a bit intimidating and tedious, but once you have such a setup working, you can reuse it over and over again for different startup profiles and provide profiling logs for each. We do have a number of people interested in adding features to support benchmarking/profiling workflows natively:

http://wiki.flightgear.org/FlightGear_Benchmark http://wiki.flightgear.org/Testing

If that's something you'd like to pursue, feel free to get in touch - it is definitely a worthwhile thing, even regardless of any OSG specific issues, as it will also help with unrelated performance issues. Obviously, there's a bit of a steep learning curve involved, which is why it's normally only core developers that build multiple versions against different OSG versions and run the profiler. But once you are able to build from source, you have already completed the most difficult step - everything else is fairly straightforward in comparison - as as you can probably tell, creating test flights using the route manager and/or flight recorder also isn't exactly rocket science.

However, before you do anything, I'd suggest to check first if you've possibly changed your driver, or if you can install a newer version

Debugging Segfaults & Obtaining Backtraces

Note  This section is currently Linux/gdb specific. Any help in updating it to become less platform/OS/debugger-specific would be greatly appreciated!

A segfault is another word for "crash", it's a so called "segmentation fault", where the program is doing things that it wasn't designed to do, such as accessing invalid memory for example, so that the operating system will terminate the process because its behavior is no longer valid/correct. These segfaults will typically happen due to coding errors, either in FlightGear or one of its dependencies (libraries like SimGear, plib, OpenSceneGraph or OpenAL).

A typical segfault may look like this:

Initializing splash screen
Splash screen progress init
Tungsten Graphics, Inc.
Mesa DRI R200 (RV250 4C66) x86/MMX/SSE2 TCL DRI2
1.3 Mesa 9.1.1

Program received signal SIGSEGV, Segmentation fault.
0xb71467e6 in __strlen_sse2_bsf () from /usr/lib/libc.so.6

The last two lines are basically showing the actual crash, anything shown earlier is normally just an indicator how far the code could proceed shortly before the crash actually occurred.

In order to see what was going on when the crash occurred, you'll need to build a binary with debugging symbols enabled/included, usually this includes FlightGear and SimGear - and possibly even OSG (OpenSceneGraph, normally only if your crash happens inside rendering related code).

A so called backtrace is basically pretty much like a flight data recorder tape/log that is used to conduct a post-crash analysis.

A binary with debugging symbols included will contain additional information that helps developers track down what was causing a certain crash, such as for example the file name, function, line number, variable that was accessed before the crash happened. This is analogous to how a FDR/CVR on an airplane records certain parameters like position, orientation, altitude, velocities, configuration, cockpit conversations etc. This can also be used by developers to see who is responsible for a certain segfault/bug, i.e. which commit introduced the bug.

Debugging FlightGear crashes is significantly simplified by having access to such runtime data, because we can basically go back in time and see what the FlightGear program was doing shortly before the crash.

Unfortunately, logging all this data by default has a certain run-time overhead and requires additional tools and knowledge, so it is usually too costly and not done, FlightGear will by default be heavily optimized by your compiler and not include code logging important debugging/post-crash parameters, for the sake of better performance. . However, as developers, we do appreciate all efforts to provide meaningful bug reports, and a reproducible bug including a good crash analysis, i.e. a so called backtrace is as good as it gets, and the likelihood of a certain bug being examined and fixed is significantly higher when the bug report includes both, a backtrace, and a set of steps to reproduce the error reliably.

A good way to to provide a reproducible test case is using the Replay/flight recorder system to create a flight recorder tape that triggers the crash, alternatively, consider providing a "flight plan" that triggers the crash - ideally, without it being aircraft specific, i.e. reproducible using aircraft like the ufo

Thus, you'll want to reconfigure your SimGear/FlightGear ($SG_SRC & $FG_SRC) source trees using the -CMAKE_BUILD_TYPE=RelWithDebInfo switch, for details please refer to: Building using CMake#Debug_Builds. It is a good idea not to touch your existing build trees for this, but instead create an additional directory hierarchy for your debugging binary, please see Building using CMake#Multiple build directories for details.

Once you have rebuilt and relinked SimGear and FlightGear, you'll want to use a debugger like gdb to run your new binary. It does help to have a good way to reproduce a crash, such as using certain startup/runtime settings. For the sake of simplicity it is usually a good idea to disable all unrelated features and subsystems/settings, this includes complex aircraft and complex scenery locations (airports) if possible. For details, refer to the minimal startup profile detailed in this article.

  1. For gdb to be available, you'll normally have to use your package manager (apt, yum, yast etc) and install the "gdb" package first.
  2. next, you'd navigate to your build directory where your fgfs debug binary is located
  3. you'd then, run gdb
  4. via the gdb shell, you can specify the file to be used via file src/Main/fgfs
  5. this will preload the binary and its dependencies
  6. to actually run the file, you would use the run command
  7. you'll normally want to pass arguments right behind run, especially the mandatory ones (i.e. to specify $FG_ROOT)
  8. creating a simple .Fgfsrc file helps for the sake of simplicity
  9. once the segfault/crash occurs, you'll want to run backtrace (with bt being the shorter equivalent)
  10. only if that isn't conclusive, use thread apply all bt full instead to get a full backtrace for all threads (background tasks)
  11. please use the issue tracker to post your backtrace

This is what a typical backtrace may look like:

(gdb) thread apply all bt

Thread 1 (Thread 0xb66c4740 (LWP 10908)):
#0  0xb71467e6 in __strlen_sse2_bsf () from /usr/lib/libc.so.6
#1  0x0886c080 in copy_string (s=s@entry=0x0)
    at /home/cesar/compilation/pkg/simgear/2.10.0/src/simgear-2.10.0/simgear/props/props.cxx:161
#2  0x08870550 in set_string (val=0x0, this=0x8dd55b8)
    at /home/cesar/compilation/pkg/simgear/2.10.0/src/simgear-2.10.0/simgear/props/props.cxx:524
#3  SGPropertyNode::setStringValue (this=0x8dd55b8, value=0x0)
    at /home/cesar/compilation/pkg/simgear/2.10.0/src/simgear-2.10.0/simgear/props/props.cxx:1587
#4  0x08326ae1 in (anonymous namespace)::GeneralInitOperation::run (this=0x8bdea48, gc=0x8be4310)
    at /home/cesar/compilation/pkg/flightgear/2.10.0/src/flightgear-2.10.0/src/GUI/gui.cxx:123
#5  0x085c2b7b in flightgear::GraphicsContextOperation::operator() (this=0x8bdea48, gc=0x8be4310)
    at /home/cesar/compilation/pkg/flightgear/2.10.0/src/flightgear-2.10.0/src/Viewer/WindowSystemAdapter.cxx:41
#6  0xb78ca246 in osg::GraphicsOperation::operator()(osg::Object*) () from /usr/lib/libosg.so.80
#7  0xb78c7784 in osg::GraphicsContext::runOperations() () from /usr/lib/libosg.so.80
#8  0xb7b29fe2 in osgViewer::ViewerBase::renderingTraversals() () from /usr/lib/libosgViewer.so.80
#9  0xb7b2707c in osgViewer::ViewerBase::frame(double) () from /usr/lib/libosgViewer.so.80
#10 0x085c4841 in fgOSMainLoop ()
    at /home/cesar/compilation/pkg/flightgear/2.10.0/src/flightgear-2.10.0/src/Viewer/fg_os_osgviewer.cxx:286
#11 0x082221fa in fgMainInit (argc=32, argv=0xbffff494)
    at /home/cesar/compilation/pkg/flightgear/2.10.0/src/flightgear-2.10.0/src/Main/main.cxx:339
#12 0x081e71ee in main (argc=32, argv=0xbffff494)
    at /home/cesar/compilation/pkg/flightgear/2.10.0/src/flightgear-2.10.0/src/Main/bootstrap.cxx:251

As can be seen, the first few lines basically show exactly what happened, and where it happened, while all the subsequent lines are basically showing the callgraph, i.e. how the piece of code got triggered. For example, FlightGear started in bootstrap.cxx line 251, then called into main.cxx (line 339), at which point the osgviewer setup took place. Next, there are a few lines that do not contain a lot of interesting info, because those are all library calls into OSG - things are starting to be interesting again around line #5, where FlightGear specific code is getting called again. As can be seen in this particular example, the actual segfault was triggered in a SimGear operation here, which was due to a NULL pointer.

At first, this may look a bit confusing and even intimidating, but it does make sense to people familiar with C++ and FlightGear/SimGear - so even if you cannot understand a single thing, just post such output on the issue tracker, so that others can have a look. We'll ask for additional information if necessary.

However, at times it may be nearly impossible to see what actually goes in in FG these days simply because there's too many disparate parts involved here. For example, in the rare instance that our Nasal scripting interpreter could lead to a segfault, we could potentially see where the C code was using a debugger like gdb – but that wouldn't tell us where we were in the Nasal code.

And that similarly applies to many things, especially things that "jump" boundaries here. Another prime example is property tree listeners - Nasal listeners don't have a backtrace beyond the C code, even if it is triggered directly from a Nasal setprop()/.setValue(), because it passes through C and C++ stack frames and then back into Nasal.

Thus, valgrind/gdb/cachegrind/perftools may be too low-level to be meaningful in most cases, i.e. just look at what's going on in the scenery/aircraft department, here just identifying "hot spots" in terms of code/functions would be a red herring, because the real culprit is typically not at all in the run-time code, but the underlying scenegraph data, such as highly complex textures, 3D models or terrain/scenery.

This can now be easily verified by using Zakalawe's new draw-masks to switch off aircraft/scenery/clouds or AI model rendering.

Having acess to gdb/valgrind etc is one thing, being able to actually run and interpret the results is a completely different thing.

Thus, it is more promising -in the long term- to optionally expose certain info at run-time, so that users can access it to see how their aircraft/scenery/feature/configuration behaves in comparison to some other aircraft/scenery/feature/configuration.

Also, when it comes to using tools like gdb -and especially valgrind- the main challenge is that FlightGear is primarily a GUI application, and it doesn't lend itself it being debugged/profiled or leak-tested in an isolated fashion at all.

For example, for valgrind runs to be useful and straightforward, we would need to be able to minimize what's done inside fg_init.cxx so that we can disable things not needed, but also even run FlightGear in a "headless" mode without a visible GUI window, so that such tests could be run directly on the build server as part of a regression testing suite. Otherwise, even just running fgfs in valgrind for 10 minutes sim time may turn out to take hours due to the nature of valgrind, i.e. it being an emulator.

valgrind et all are great if you know how to use them, but for end users, these tools are too fine-grained, we would ideally support some per-subsystem or per-feature granularity for end users.

Using AddressSanitizer

Cquote1.png AddressSanitizer does a similar job to Valgrind with less overhead

(~2.5x memory use, and nearly full speed), but requires recompiling; I
haven't seen the crash under it yet (or at all today), if anyone does
please post your log.


Cquote2.png
#this is for Linux with llvm-3.4, libsqlite3-dev, flite1-dev, libhtsengine-dev; it will probably work on Mac (with possibly different  ASAN_SYMBOLIZER_PATH) but not Windows

#simgear
cmake ../../git/simgear -DCMAKE_BUILD_TYPE=RelWithDebInfo 
-DCMAKE_C_FLAGS="-fsanitize=address -fno-omit-frame-pointer -O2 -g" 
-DCMAKE_CXX_FLAGS="-fsanitize=address -fno-omit-frame-pointer -O2 -g" 
-DCMAKE_SHARED_LINKER_FLAGS="-fsanitize=address -fno-omit-frame-pointer 
-O2 -g" -DCMAKE_VERBOSE_MAKEFILE=1 -DSIMGEAR_SHARED=ON
make
sudo make install

#flightgear
cmake ../../git/flightgear -DCMAKE_BUILD_TYPE=RelWithDebInfo
-DCMAKE_C_FLAGS="-fsanitize=address -fno-omit-frame-pointer -O2 -g" 
-DCMAKE_CXX_FLAGS="-fsanitize=address -fno-omit-frame-pointer -O2 -g" 
-DCMAKE_VERBOSE_MAKEFILE=1 -DSIMGEAR_SHARED=ON -DENABLE_IAX=OFF 
-DENABLE_FGCOM=OFF -DSYSTEM_SQLITE=ON
make
sudo make install

#run
ASAN_SYMBOLIZER_PATH=/usr/lib/llvm-3.4/bin/llvm-symbolizer 
ASAN_OPTIONS="symbolize=1 alloc_dealloc_mismatch=0" fgfs [options] 2> 
asan_log.txt

#post asan_log.txt to developers mailing list and/or the issue tracker

The original idea and future developments

Our hope is that we can come up with a fairly safe subset of FlightGear settings that should work for 99% of our users, even on moderately old hardware. This would enable us to change the FlightGear defaults accordingly, and use these settings as a safe fallback alternative - so that FlightGear should no longer just crash for people without certain hardware/features.

There's a trend to expose more and more information via the property tree, to make the simulator increasingly runtime-configurable, and to allow subsystems to be re-initialized at runtime, so being able to start up FlightGear in some form of "safe" mode is generally a good thing and useful for anybody helping with troubleshooting on the forums, because it helps us exclude many potential problems. In the long run, such a profile could also be used as the foundation for a FlightGear Benchmark or some scripted regression suite to run FlightGear in a headless mode to help with release preparations, and FGCanvas.


Related