Troubleshooting crashes

From FlightGear wiki
Jump to navigation Jump to search

Targeted FlightGear versions: 2.6, 2.8, 2.10, 2.12+

Motivation

This article should be suitable for pretty much any OSG-based version beyond 2.6/2.8 - the 2.10 reference/disclaimer below is just about a specific GLSL issue that got introduced (and now fixed in 2.11+) in FG 2.10 in particular, the point of the whole write-up is to provide a dedicated place for people to go with these sorts of problems, and to reduce the workload on our part obviously - and provide some guidance to end-users facing such issues, allowing them to experiment a little and provide more specific feedback. The idea is to keep it up-to-date for upcoming FG versions, too - including 2.12 and beyond. In other words, any and all help is appreciated, so please feel free to get involved and contribute to the article by adding your own findings, suggestions and techniques. Your help is really appreciated!

If you should find yourself having to disable additional settings in order to further maximize FlightGear performance, please do add your findings to this article, so that others can benefit.

Our hope is that we can come up with a fairly safe subset of FlightGear settings that should work for 99% of our users, even on moderately old hardware. This would enable us to change the FlightGear defaults accordingly, and use these settings as a safe fallback alternative - so that FlightGear should no longer just crash for people without certain hardware/features.

There's a trend to expose more and more information via the property tree, to make the simulator increasinly runtime-configurable, and to allow subsystems to be re-initialized at runtime, so being able to start up FlightGear in some form of "safe" mode is generally a good thing and useful for anybody helping with troubleshooting on the forums, because it helps us exclude many potential problems. In the long run, such a profile could also be used as the foundation for a FlightGear Benchmark or some scripted regression suite to run FlightGear in a headless mode to help with release preparations.

The Issue with 2.10

Note: FlightGear 2.10 users seeing crashes during simulator startup (without seeing the actual aircraft/scenery) with very old graphics cards that lack GLSL (shader) support should know that there used to be a bug in 2.10 that broke support for graphics cards without any shader support. In other words, if you are able to run other OpenGL software, but cannot even display the about dialog in FG 2.10 using the minimal startup profile detailed below, it is almost certainly your graphics card's lack of shader support causing the issue, because FlightGear 2.10 tries to copy graphics card information to the property tree for debugging purposes, to allow end-users to provide better troubleshooting reports - ironically, this very feature introduced crashes for all 2.10 users without any GLSL support. This issue has been fixed in FlightGear 2.11+

Crashes in general

If FlightGear crashes (or looks/performs very badly/slowly), it's possible that this is due a number of reasons, such as limited hardware resources (CPU, GPU, RAM etc) or insufficient driver support - so crashes are not necessarily due to a faulty program. FlightGear does work for hundreds of forum users, and usually without any form of sophisticated prior setup or configuration. So if something doesn't work "out of the box" it's usually some local or system-specific issue. In general, you are unlikely to encounter any major bugs when using a release - if you don't mind having to deal with bugs and technical issues, we'd like to invite you to help us test prereleases (so called "release candidates"/RCs) and nightly builds - or directly build from git.

Possible causes

It's just as well possible that the FlightGear default settings are not supported with your hardware/drivers, so that the process will be killed by the OS because it's trying to do something not supported by your computer. If you are experiencing full system crashes, you should see System Crashes instead.

By default, FlightGear 2.6+ will assume a certain runtime environment and certain GPU features which may not be suppported by most older cards (e.g. shader support), such as GeForce 6/7 generation hardware and most older hardware (older than 4-5 years), so that it can be expected that FG will probably not work on such computers "out of the box" and may get terminated by your operating system eventually, i.e. the default startup settings will need to be customized accordingly, to disable all unsupported default features. See Settings for slower graphics cards, Problematic Video Cards and FlightGear Hardware Recommendations for more details.

Also, make sure you've got the newest driver for your video card installed and running without errors. A suggestion for a thorough updating procedure can be found here.

Before proceeding any further, please make sure that you are actually able to run other OpenGL-based software/simulators/games, unrelated to FlightGear!

The startup profile detailed below disables tons of features that are known to have a massive impact on frame rate and overall performance by making FG look plain bad and crude, as such, it gives you a rough idea of the maximum performance fgfs is able to achieve on your system - which in turn, allows you then to re-enable individual settings one-by-one, and see their impact on framerate/performance - so that you can determine which settings are particularly problematic - so that you can decide to come up with a custom profile to make the 60-100 fps goal for example - as you can see, on a 2009-era laptop, you could hope for ~790 fps and 3ms of frame spacing when in the "ugly" mode - so by hand-picking my settings and tweaking them as needed, you'll be getting beyond 50 fps, even with 75-85% of eye candy enabled, with fairly moderate GPU hardware

Running FlightGear with minimal settings

In such cases, it is important to start up FlightGear with minimum settings, and start re-enabling features step by step to see what settings are supported by your hardware. You may want to raise the FlightGear log level, so that you get to see where the program stops starting/working, and what it was doing before it got terminated.

The following settings are really just intended to provide a minimal startup profile, with ALL eye candy disabled by default - so that FG doesn't crash due to lack of certain features (OpenGL, shaders etc) and you get a "bare bone" FlightGear window. They use the UFO, as it is the least complex aircraft (the default aircraft is the Cessna 172) and a suitable scenery location to rule out aircraft or scenery as a cause for crashes.

Once you have launched FlightGear with these minimal settings, you can enable all graphics options one by one. If an option incurs a huge performance hit, take into consideration whether it is necessary for your flying experience or not.

The performance impact can be evaluated by using:

It makes sense to keep using the ufo until you have optimized all settings accordingly and found out what's working and what isn't, i.e. don't even touch the aircraft settings or the location settings (airport, azimuth, offset) until you have really optimized everything else already.

Performance targets

Generally, try to optimize FlightGear for at least 30+ fps before changing to a regular aircraft and/or location! Realistically, the ufo is not very representative - so that it would be better to aim for something in between 50-100 fps prior to switching to a more complex aircraft like the c172p for example. For instance, aircraft like the A380, IAR80, Concorde or 777 are known to require fairly powerful hardware.

Thus, if you are not getting frame rates much higher than 100 fps with the minimal startup profile (without using frame throttling, i.e. locking the refresh rate at a certain frequency, obviously), then using complex aircraft (777,787) or scenery (KSFO, KLSV etc) will be next to impossible with your system.

Minimal Startup Profile

As mentioned, the point of the following settings is to disable everything that could have an effect on performance, to ensure that we get a working FlightGear window up and running with no eye candy at all, and maximum frame rate. Once that is working, features can be re-enabled step by step.

Depending on the hardware specs of the target platform, adjusting the threading mode of FG/OSG may also help: Howto:Activate multi core and multi GPU support.

You can directly use a custom Fgfsrc file for the following sections or parse the following lines to the console (separated by one empty character) after "FGFS" or set the respective options in Fgrun or in one of FlightGear's XML configuration files.

Please make sure to rename/delete your autosave and fgfsrc files prior to trying these settings, or your local settings would possibly override these settings.

# --ignore-autosave # uncomment this for FlightGear versions >= 2.99
--airport=ksfo
--offset-distance=4000
--offset-azimuth=90
--altitude=500
--heading=0
--model-hz=60
--disable-random-objects
--prop:/sim/rendering/texture-compression=off
--prop:/sim/rendering/quality-level=0
--prop:/sim/rendering/shaders/quality-level=0
--disable-ai-traffic
--prop:/sim/ai/enabled=0
--aircraft=ufo
--disable-sound
--prop:/sim/rendering/random-vegetation=0
--prop:/sim/rendering/random-buildings=0
--disable-specular-highlight
--disable-ai-models
--disable-clouds
--disable-clouds3d
# --disable-textures
--fog-fastest
--visibility=5000
--disable-distance-attenuation
--disable-enhanced-lighting
--disable-real-weather-fetch
--prop:/sim/rendering/particles=0
Note  Beginning with FlightGear 3.1+, you can also toggle individual scenegraph traversal masks on/off (these can be changed at runtime using the Property browser:
  • --prop:browser=/sim/rendering/draw-mask
  • --prop:/sim/rendering/draw-mask/terrain=0
  • --prop:/sim/rendering/draw-mask/aircraft=0
  • --prop:/sim/rendering/draw-mask/models=0
  • --prop:/sim/rendering/draw-mask/clouds=0

Using these settings, frame rates between 400-500 fps and a steady frame spacing (latency) of 3-11 ms, can be achieved on a notebook from 2007 (1024x768). Disabling wireframe mode lowers the framerate by about 50-80 fps. The window should be up and running (fully initialized) within 3-5 seconds, even on moderately old hardware (absent a navcache rebuild).

FlightGear 2.10 bare bones on 2006-era hardware (NVIDIA 7600GO)
FG 2.12 zero eyecandy mode on 2009-era hardware (NVIDIA 260M)
FG 2.12 minimal eye candy mode on 2009-era hardware (NVIDIA 260M)

As can be seen, additional resources can be freed by flying in areas without scenery (ocean tiles), by using simple aircraft like the ufo, and by using a HUD instead of a 2D/3D cockpit panel.

Minimal Rembrandt Startup Profile

--enable-rembrandt
--prop:/sim/rendering/rembrandt/ambient-occlusion-buffers=false
--prop:/sim/rendering/shadows/map-size=1024
--prop:/sim/rendering/shadows/num-cascades=1
--airport=ksfo
--offset-distance=4000
--offset-azimuth=90
--altitude=500
--heading=0
--model-hz=60
--disable-random-objects
--prop:/sim/rendering/texture-compression=off
--prop:/sim/rendering/quality-level=1
--prop:/sim/rendering/shaders/quality-level=1
--disable-ai-traffic
--prop:/sim/ai/enabled=0
--aircraft=ufo
--disable-sound
--prop:/sim/rendering/random-vegetation=0
--prop:/sim/rendering/random-buildings=0
--disable-specular-highlight
--disable-ai-models 
--disable-clouds
--disable-clouds3d
--disable-textures
--fog-fastest
--visibility=5000
--disable-distance-attenuation
--disable-enhanced-lighting
--disable-real-weather-fetch 
--prop:/sim/rendering/particles=0

(Again, the point not being to look good, but just to start up without crashing or extreme lag)

Further assistance

If you need help with the whole process, please get in touch via the FG forums.

When doing that, please fully document the whole process, i.e. including:

  • startup settings (and fgfsrc if used)
  • error messages and warnings (console output)
  • screen shots
  • screen shots showing frame rate, frame spacing, performance monitor, osg on-screen stats (see Howto:Use the system monitor )
  • use the about dialog to provide other important information

Reporting incompatible default settings

Once you have located and identified settings that seem to cause reproducible issues with your hardware, it would be a good idea to report these settings using the issue tracker, so that these can hopefully be made optional in the upcoming release: http://flightgear-bugs.googlecode.com/

Debugging Segfaults & Obtaining Backtraces

A segfault is another word for "crash", it's a so called "segmentation fault". In order to see what was going on when the crash occurred, you'll need to build a binary with debugging symbols enabled, usually this includes FlightGear and SimGear - and possibly even OSG (OpenSceneGraph, only if your crash happens inside rendernig related code).

A binary with debugging symbols included will contain additional information that helps developer track down what is causing a certain crash, such as for example the file name, function, line number, variable that was accessed before the crash happened.

You'll want to reconfigure your SimGear/FlightGearGear ($SG_SRC & $FG_SRC) source trees using the -CMAKE_BUILD_TYPE=Debug switch, for details please refer to: Building using CMake#Debug_Builds. It is a good idea not to touch your existing build trees for this, but instead create an additional directory hierarchy for your debugging binary, please see Building using CMake#Multiple build directories for details.

Once you have rebuilt and relinked SimGear and FlightGear, you'll want to use a debugger like gdb to run your new binary. It does help to have a good way to reproduce a crash, such as using certain startup/runtime settings. For the sake of simplicity it is usually a good idea to disable all unrelated features and subsystems/settings, this includes complex aircraft and complex scenery locations (airports) if possible. For details, refer to the minimal startup profile detailed in this article.

  1. For gdb to be available, you'll normally have to use your package manager (apt, yum, yast etc) and install the "gdb" package first.
  2. next, you'd navigate to your build directory where your fgfs debug binary is located
  3. you'd then, run gdb
  4. via the gdb shell, you can specify the file to be used via file src/Main/fgfs
  5. this will preload the binary and its dependencies
  6. to actually run the file, you would use the run command
  7. you'll normally want to pass arguments right behind run, especially the mandatory ones (i.e. to specify $FG_ROOT)
  8. creating a simple .Fgfsrc file helps for the sake of simplicity
  9. once the segfault/crash occurs, you'll want to run backtrace (with bt being the shorter equivalent)
  10. only if that isn't conclusive, use thread apply all bt full instead to get a full backtrace for all threads (background tasks)
  11. please use the issue tracker to post your backtrace

Related