Troubleshooting crashes
| Troubleshooting |
|---|
| Note This article is valid for FlightGear versions: 2.6, 2.8, 2.10, 2.12, 3.0+. Help us in keeping this article up to date! If you're facing some crash not listed here and manage to solve it, please add it to a proper section, so that other can benefit from your experience! |
Crashes are never easy to troubleshoot. It's very likely that you'll need to experiment a bit to find out the cause, or at least to provide useful feedback. In fact issues depend a lot on things like hardware components, driver support and so on. Developers can't work on your machine, so you'll need to provide them with some specific information. This article will guide you in that.
If not only FlightGear, but the whole system crashes, or freezes, please refer to System Crashes.
If in the end you like bug hunting, please consider helping in testing future releases.
Known problems
Before diving into the real troubleshooting, first try to see if one of the following is your case. This could be a huge time saver!
If your problem is not listed here, please consider adding it after you have solved it! Others like you will be grateful!
Optimus/Hybrid graphics technologies for laptops
If FlightGear crashes with
Unknown exception in the main loop. Aborting... Possible cause: Not enough space
and you happen to have a laptop with Optimus Technology (NVidia) or Hybrid Graphics technology (ATI) onboard, you'll find some help at Troubleshooting performance issues#Operating system and driver setup. (Note that the problem is treated there because it's mostly cause of low performance.)
World Scenery 2.0 and 32-bit machines
World Scenery 2.0 (the one with CORINE and OpenStreetMap data) highly raised memory requirements with respect to the previous version. Basically, with 4GB of RAM once you could safely use a visibility of a hundred kilometers; with Scenery 2.0, 4GB is barely enough for 10 km. A 32-bit machine can't use more than that memory, and simply kills FlightGear when it asks for more. To verify this, reduce your visibility (in-sim with View > Level of Detail adjustment and set Bare to 5000 m) and see if it stops crashing. You can also check memory consumption with the tools provided by your OS.
Sadly, the solution is keeping a low visibility, or moving to a 64-bit machine with at least 8GB of RAM, which can provide even 45km of visibility. If you're feeling "hackish", you can try the solution reported at Unexpected Crashes with Scenery ver.2.
Old graphics cards and FlightGear 2.10
If you're using FlightGear 2.10, you're seeing crashes during simulator startup (without seeing the actual aircraft/scenery), and you're using an old enough graphics card, it is possible that your crash is due to a bug in that FG version and to your card lacking GLSL support. This bug would prevent using FG even with shaders off! Ironically, this bug comes from a feature introduced to help troubleshooting bugs.
So, if you're on 2.10 and think your card could be old enough, try the minimal startup configuration below: if it keeps crashing, that's definitely it. The bug was solved immediately later (ver. 2.11)
Troubleshooting your crash
FlightGear crashes are due to a number of reasons, not necessarily to bugs in its code. After all, it works "out of the box" for most users, so if something's broken it's often a system-specific issue, and yet again, it could be a system configuration that the developers couldn't test - you can help with that!)
It's just as well possible that the FlightGear default settings are not supported with your hardware/drivers. Since version 2.6, FlightGear will assume support for certain features that hardware older than 4-5 years might not give (especially graphics cards), so the default settings need to be customized to disable all unsupported features. See Troubleshooting performance issues, Problematic Video Cards and FlightGear Hardware Recommendations for more details.
Preliminary check
Make sure you've got the newest driver for your video card installed and running without errors. A suggestion for a thorough updating procedure can be found at Graphics drivers configuration. Before proceeding any further, please make sure that you are actually able to run other OpenGL-based software/simulators/games, unrelated to FlightGear!
The standard procedure
Given the above, it is important to start up FlightGear with a minimal startup profile, with all eye candy and features disabled (so that FG won't crash due to lack of support for them) and start re-enabling them step by step to see what's supported by your hardware. You may want to raise the FlightGear log level, so that you get to see where the program stops starting/working, and what it was doing before it got terminated. This procedure will also help in troubleshooting performance issues.
These settings use the simple UFO and a suitable scenery location to rule out aircraft or scenery as a cause for crashes. Don't even touch the aircraft and location settings (airport, azimuth, offset) until you have really fixed everything else already. Working step by step is the only real way, so don't bite off more than you can chew.
If you find it looks like the Windows "compatibility mode" that you boot into after a crash, then you understand perfectly what we're doing here. However it's not as straightforward to do, so if in trouble ask for support at the forum 
!
Minimal startup profile
| Note If you find any other settings causing a massive improvement of performance/compatibility, please feel free to add those below, so that others can benefit from your findings. |
Tweaking nvidia settings to obtain higher frame rates by disabling vsync to blank
Tweaking Nvidia Powermizer to obtain higher frame rates
Depending on the hardware specs of the target platform, adjusting the threading mode of FG/OSG may also help: Howto:Activate multi core and multi GPU support. (If you don't know what we're talking about, you can safely leave it as it is.)
You can directly use a custom Fgfsrc file for the following sections or parse the following lines to the console (separated by one empty character, e.g. a space) after "FGFS" or set the respective options in Fgrun or in one of FlightGear's XML configuration files.
Please make sure to rename/delete your autosave and fgfsrc files prior to trying these settings, or your local settings would possibly override these settings.
| Note The following FlightGear startup profile assumes that you have a $FG_ROOT environment variable set up, or that you are explicitly setting fg-root using the --fg-root command line argument, this startup profile is intended to be put into your Fgfsrc file or to be used when starting FlightGear from the command line.
For FlightGear 2020.3 LTS and later versions: you can simply copy and paste the profile in Flightgear Qt launcher > Settings tab > Additional Settings . This is a minimum startup profile. After using it, you can turn up settings one by one as performance allows - and observe your performance bottlenecks. See Hardware recommendations for discussion of performance bottlenecks. The profile listed below is
|
# --ignore-autosave # uncomment this for FlightGear versions >= 2.99 --disable-terrasync --disable-splash-screen --airport=ksfo --offset-distance=4000 --offset-azimuth=90 --altitude=500 --heading=0 --model-hz=60 --disable-random-objects --prop:/sim/rendering/texture-compression=off --prop:/sim/rendering/quality-level=0 --prop:/sim/rendering/shaders/quality-level=0 --disable-ai-traffic --prop:/sim/ai/enabled=0 --aircraft=ufo --disable-sound --prop:/sim/rendering/random-vegetation=0 --prop:/sim/rendering/random-buildings=0 --disable-specular-highlight --disable-ai-models --disable-clouds --disable-clouds3d # --disable-textures --fog-fastest --visibility=5000 --disable-distance-attenuation --disable-real-weather-fetch --prop:/sim/rendering/particles=0 --prop:/sim/rendering/multi-sample-buffers=1 --prop:/sim/rendering/multi-samples=2 --prop:/sim/rendering/draw-mask/clouds=false --prop:/sim/rendering/draw-mask/aircraft=false --prop:/sim/rendering/draw-mask/models=false --prop:/sim/rendering/draw-mask/terrain=false --disable-random-vegetation --disable-random-buildings --disable-horizon-effect
Tip Beginning with FlightGear 3.1+, you can also toggle individual scenegraph traversal masks on/off (these can be changed at runtime using the Property browser:
|
If you're running on Windows, you can use the following batch file to start the simulator using the minimal profile. Open Notepad, copy and paste the following lines and save the file on the Desktop (name it fgminimal.bat); change C:\Program Files\FlightGear to the directory you installed FlightGear in and change Win64 to Win32 if you are using the 32-bit version.
@echo off "C:\Program Files\FlightGear\bin\Win64\fgfs.exe" --fg-root="C:\Program Files\FlightGear\data" --ignore-autosave --airport=ksfo --offset-distance=4000 --offset-azimuth=90 --altitude=500 --heading=0 --model-hz=60 --disable-random-objects --prop:/sim/rendering/texture-compression=off --prop:/sim/rendering/quality-level=0 --prop:/sim/rendering/shaders/quality-level=0 --disable-ai-traffic --prop:/sim/ai/enabled=0 --aircraft=ufo --disable-sound --prop:/sim/rendering/random-vegetation=0 --prop:/sim/rendering/random-buildings=0 --disable-specular-highlight --disable-ai-models --disable-clouds --disable-clouds3d --fog-fastest --visibility=5000 --disable-distance-attenuation --disable-enhanced-lighting --disable-real-weather-fetch --prop:/sim/rendering/particles=0 --prop:/sim/rendering/multi-sample-buffers=1 --prop:/sim/rendering/multi-samples=2
Double-click the batch file to start FlightGear.
Using these settings, frame rates between 400-500 fps and a steady frame spacing (latency) of 3-11 ms, can be achieved on a notebook from 2007 (1024x768). Disabling wireframe mode lowers the framerate by about 50-80 fps. The window should be up and running (fully initialized) within 3-5 seconds, even on moderately old hardware (absent a navcache rebuild).
As can be seen, additional resources can be freed by flying in areas without scenery (ocean tiles), by using simple aircraft like the ufo, and by using a HUD instead of a 2D/3D cockpit panel.
Reporting incompatible default settings
Once you have located and identified settings that seem to cause reproducible issues with your hardware, it would be a good idea to report these settings using the issue tracker, so that these can hopefully be made optional in the upcoming release: http://flightgear-bugs.googlecode.com/
Tracking RAM utilization
 |
Work in progress This article or section will be worked on in the upcoming hours or days. See history for the latest developments. |
|
My measurements (below) suggest that terrain mesh (as loaded at "bare" LOD) is the dominant memory user, with only small contributions from the — Rebecca N. Palmer (2014-06-11). Re: [Flightgear-devel] Memory consumption reduction.
(powered by Instant-Cquotes) |
|
|
|
In Ubuntu 64-bit, measured using System Monitor after it stabilizes (~1min) Except as stated, c172p stationary on KSFO 28R, night, Terrasync — Rebecca N. Palmer (2014-06-11). Re: [Flightgear-devel] Memory consumption reduction.
(powered by Instant-Cquotes) |
|
Further assistance
If you need help with the whole process, please get in touch via the FG forums.
When doing that, please fully document the whole process, i.e. including:
- startup settings (and fgfsrc if used)
- error messages and warnings (console output)
- screen shots
- screen shots showing frame rate, frame spacing, performance monitor, osg on-screen stats (see Howto:Use the system monitor )
- use the about dialog to provide other important information
Troubleshooting Performance Issues
If you're able to build from source and don't mind using development looks like the built-in profiler, it would be interesting to see comparison of both builds - ideally using a simple test case with everything else disabled - you could use a pre-recorded flight and/or a flight recorder type to come up with a "test flight". From our standpoint, it would help tremendously if all features that don't seem to have an effect could be completely disabled, including complex aircraft and scenery/locations. In other words, if you can reproduce the problem using "bare" minimum settings, the resulting log file will be much easier to process.
There's a built-in profiler which you can use to create these profiles: Built-in Profiler You would then want to use two different build directories, where SG/FG build settings would be identical, but using an older version of OSG: Building using CMake#Multiple build directories
For the sake of simplicity there's a so called "minimal startup profile" that you can use: Howto:Debugging FlightGear Crashes#Minimal startup profile While unlikely, it would be great if the issue could be reproduced that way - but more likely than not, you'll have to re-add a few features and change a few settings, e.g. by using a different location.
Like I said, you could then use the replay system to create an test flight than can be easily reproduced - to get going more quickly, you can also use the built-in route manager to create a simple flight plan and fly the whole thing on autopilot: Instant Replay
You can use time warp mode to speed up simulation time and finish more quickly.
Admittedly, all this may seem a bit intimidating and tedious, but once you have such a setup working, you can reuse it over and over again for different startup profiles and provide profiling logs for each. We do have a number of people interested in adding features to support benchmarking/profiling workflows natively:
If that's something you'd like to pursue, feel free to get in touch - it is definitely a worthwhile thing, even regardless of any OSG specific issues, as it will also help with unrelated performance issues. Obviously, there's a bit of a steep learning curve involved, which is why it's normally only core developers that build multiple versions against different OSG versions and run the profiler. But once you are able to build from source, you have already completed the most difficult step - everything else is fairly straightforward in comparison - as as you can probably tell, creating test flights using the route manager and/or flight recorder also isn't exactly rocket science.
However, before you do anything, I'd suggest to check first if you've possibly changed your driver, or if you can install a newer version
Debugging Segfaults & Obtaining Backtraces
| Note This section is currently Linux/gdb specific. Any help in updating it to become less platform/OS/debugger-specific would be greatly appreciated! |
A segfault is another word for "crash", it's a so called "segmentation fault", where the program is doing things that it wasn't designed to do, such as accessing invalid memory for example, so that the operating system will terminate the process because its behavior is no longer valid/correct. These segfaults will typically happen due to coding errors, either in FlightGear or one of its dependencies (libraries like SimGear, plib, OpenSceneGraph or OpenAL).
A typical segfault may look like this:
Initializing splash screen
Splash screen progress init
Tungsten Graphics, Inc.
Mesa DRI R200 (RV250 4C66) x86/MMX/SSE2 TCL DRI2
1.3 Mesa 9.1.1
Program received signal SIGSEGV, Segmentation fault.
0xb71467e6 in __strlen_sse2_bsf () from /usr/lib/libc.so.6
The last two lines are basically showing the actual crash, anything shown earlier is normally just an indicator how far the code could proceed shortly before the crash actually occurred.
In order to see what was going on when the crash occurred, you'll need to build a binary with debugging symbols enabled/included, usually this includes FlightGear and SimGear - and possibly even OSG (OpenSceneGraph, normally only if your crash happens inside rendering related code).
A so called backtrace is basically pretty much like a flight data recorder tape/log that is used to conduct a post-crash analysis.
A binary with debugging symbols included will contain additional information that helps developers track down what was causing a certain crash, such as for example the file name, function, line number, variable that was accessed before the crash happened. This is analogous to how a FDR/CVR on an airplane records certain parameters like position, orientation, altitude, velocities, configuration, cockpit conversations etc. This can also be used by developers to see who is responsible for a certain segfault/bug, i.e. which commit introduced the bug.
Debugging FlightGear crashes is significantly simplified by having access to such runtime data, because we can basically go back in time and see what the FlightGear program was doing shortly before the crash.
Unfortunately, logging all this data by default has a certain run-time overhead and requires additional tools and knowledge, so it is usually too costly and not done, FlightGear will by default be heavily optimized by your compiler and not include code logging important debugging/post-crash parameters, for the sake of better performance. . However, as developers, we do appreciate all efforts to provide meaningful bug reports, and a reproducible bug including a good crash analysis, i.e. a so called backtrace is as good as it gets, and the likelihood of a certain bug being examined and fixed is significantly higher when the bug report includes both, a backtrace, and a set of steps to reproduce the error reliably.
A good way to to provide a reproducible test case is using the Replay/flight recorder system to create a flight recorder tape that triggers the crash, alternatively, consider providing a "flight plan" that triggers the crash - ideally, without it being aircraft specific, i.e. reproducible using aircraft like the ufo
Thus, you'll want to reconfigure your SimGear/FlightGear ($SG_SRC & $FG_SRC) source trees using the -CMAKE_BUILD_TYPE=RelWithDebInfo switch, for details please refer to: Building using CMake#Debug_Builds. It is a good idea not to touch your existing build trees for this, but instead create an additional directory hierarchy for your debugging binary, please see Building using CMake#Multiple build directories for details.
Once you have rebuilt and relinked SimGear and FlightGear, you'll want to use a debugger like gdb to run your new binary. It does help to have a good way to reproduce a crash, such as using certain startup/runtime settings. For the sake of simplicity it is usually a good idea to disable all unrelated features and subsystems/settings, this includes complex aircraft and complex scenery locations (airports) if possible. For details, refer to the minimal startup profile detailed in this article.
- For gdb to be available, you'll normally have to use your package manager (apt, yum, yast etc) and install the "gdb" package first.
- next, you'd navigate to your build directory where your fgfs debug binary is located
- you'd then, run gdb
- via the gdb shell, you can specify the file to be used via file src/Main/fgfs
- this will preload the binary and its dependencies
- to actually run the file, you would use the run command
- you'll normally want to pass arguments right behind run, especially the mandatory ones (i.e. to specify $FG_ROOT)
- creating a simple .Fgfsrc file helps for the sake of simplicity
- once the segfault/crash occurs, you'll want to run backtrace (with bt being the shorter equivalent)
- only if that isn't conclusive, use thread apply all bt full instead to get a full backtrace for all threads (background tasks)
- please use the issue tracker to post your backtrace
This is what a typical backtrace may look like:
(gdb) thread apply all bt
Thread 1 (Thread 0xb66c4740 (LWP 10908)):
#0 0xb71467e6 in __strlen_sse2_bsf () from /usr/lib/libc.so.6
#1 0x0886c080 in copy_string (s=s@entry=0x0)
at /home/cesar/compilation/pkg/simgear/2.10.0/src/simgear-2.10.0/simgear/props/props.cxx:161
#2 0x08870550 in set_string (val=0x0, this=0x8dd55b8)
at /home/cesar/compilation/pkg/simgear/2.10.0/src/simgear-2.10.0/simgear/props/props.cxx:524
#3 SGPropertyNode::setStringValue (this=0x8dd55b8, value=0x0)
at /home/cesar/compilation/pkg/simgear/2.10.0/src/simgear-2.10.0/simgear/props/props.cxx:1587
#4 0x08326ae1 in (anonymous namespace)::GeneralInitOperation::run (this=0x8bdea48, gc=0x8be4310)
at /home/cesar/compilation/pkg/flightgear/2.10.0/src/flightgear-2.10.0/src/GUI/gui.cxx:123
#5 0x085c2b7b in flightgear::GraphicsContextOperation::operator() (this=0x8bdea48, gc=0x8be4310)
at /home/cesar/compilation/pkg/flightgear/2.10.0/src/flightgear-2.10.0/src/Viewer/WindowSystemAdapter.cxx:41
#6 0xb78ca246 in osg::GraphicsOperation::operator()(osg::Object*) () from /usr/lib/libosg.so.80
#7 0xb78c7784 in osg::GraphicsContext::runOperations() () from /usr/lib/libosg.so.80
#8 0xb7b29fe2 in osgViewer::ViewerBase::renderingTraversals() () from /usr/lib/libosgViewer.so.80
#9 0xb7b2707c in osgViewer::ViewerBase::frame(double) () from /usr/lib/libosgViewer.so.80
#10 0x085c4841 in fgOSMainLoop ()
at /home/cesar/compilation/pkg/flightgear/2.10.0/src/flightgear-2.10.0/src/Viewer/fg_os_osgviewer.cxx:286
#11 0x082221fa in fgMainInit (argc=32, argv=0xbffff494)
at /home/cesar/compilation/pkg/flightgear/2.10.0/src/flightgear-2.10.0/src/Main/main.cxx:339
#12 0x081e71ee in main (argc=32, argv=0xbffff494)
at /home/cesar/compilation/pkg/flightgear/2.10.0/src/flightgear-2.10.0/src/Main/bootstrap.cxx:251As can be seen, the first few lines basically show exactly what happened, and where it happened, while all the subsequent lines are basically showing the callgraph, i.e. how the piece of code got triggered. For example, FlightGear started in bootstrap.cxx line 251, then called into main.cxx (line 339), at which point the osgviewer setup took place. Next, there are a few lines that do not contain a lot of interesting info, because those are all library calls into OSG - things are starting to be interesting again around line #5, where FlightGear specific code is getting called again. As can be seen in this particular example, the actual segfault was triggered in a SimGear operation here, which was due to a NULL pointer.
At first, this may look a bit confusing and even intimidating, but it does make sense to people familiar with C++ and FlightGear/SimGear - so even if you cannot understand a single thing, just post such output on the issue tracker, so that others can have a look. We'll ask for additional information if necessary.
However, at times it may be nearly impossible to see what actually goes in in FG these days simply because there's too many disparate parts involved here. For example, in the rare instance that our Nasal scripting interpreter could lead to a segfault, we could potentially see where the C code was using a debugger like gdb – but that wouldn't tell us where we were in the Nasal code.
And that similarly applies to many things, especially things that "jump" boundaries here. Another prime example is property tree listeners - Nasal listeners don't have a backtrace beyond the C code, even if it is triggered directly from a Nasal setprop()/.setValue(), because it passes through C and C++ stack frames and then back into Nasal.
Thus, valgrind/gdb/cachegrind/perftools may be too low-level to be meaningful in most cases, i.e. just look at what's going on in the scenery/aircraft department, here just identifying "hot spots" in terms of code/functions would be a red herring, because the real culprit is typically not at all in the run-time code, but the underlying scenegraph data, such as highly complex textures, 3D models or terrain/scenery.
This can now be easily verified by using Zakalawe's new draw-masks to switch off aircraft/scenery/clouds or AI model rendering.
Having acess to gdb/valgrind etc is one thing, being able to actually run and interpret the results is a completely different thing.
Thus, it is more promising -in the long term- to optionally expose certain info at run-time, so that users can access it to see how their aircraft/scenery/feature/configuration behaves in comparison to some other aircraft/scenery/feature/configuration.
Also, when it comes to using tools like gdb -and especially valgrind- the main challenge is that FlightGear is primarily a GUI application, and it doesn't lend itself it being debugged/profiled or leak-tested in an isolated fashion at all.
For example, for valgrind runs to be useful and straightforward, we would need to be able to minimize what's done inside fg_init.cxx so that we can disable things not needed, but also even run FlightGear in a "headless" mode without a visible GUI window, so that such tests could be run directly on the build server as part of a regression testing suite. Otherwise, even just running fgfs in valgrind for 10 minutes sim time may turn out to take hours due to the nature of valgrind, i.e. it being an emulator.
valgrind et all are great if you know how to use them, but for end users, these tools are too fine-grained, we would ideally support some per-subsystem or per-feature granularity for end users.
Using AddressSanitizer
|
|
AddressSanitizer does a similar job to Valgrind with less overhead (~2.5x memory use, and nearly full speed), but requires recompiling; I — Rebecca Palmer (2014-08-22). Re: [Flightgear-devel] crash in SGPropertyNode::fireValueChanged.
(powered by Instant-Cquotes) |
|
#this is for Linux with llvm-3.4, libsqlite3-dev, flite1-dev, libhtsengine-dev; it will probably work on Mac (with possibly different ASAN_SYMBOLIZER_PATH) but not Windows
#simgear
cmake ../../git/simgear -DCMAKE_BUILD_TYPE=RelWithDebInfo
-DCMAKE_C_FLAGS="-fsanitize=address -fno-omit-frame-pointer -O2 -g"
-DCMAKE_CXX_FLAGS="-fsanitize=address -fno-omit-frame-pointer -O2 -g"
-DCMAKE_SHARED_LINKER_FLAGS="-fsanitize=address -fno-omit-frame-pointer
-O2 -g" -DCMAKE_VERBOSE_MAKEFILE=1 -DSIMGEAR_SHARED=ON
make
sudo make install
#flightgear
cmake ../../git/flightgear -DCMAKE_BUILD_TYPE=RelWithDebInfo
-DCMAKE_C_FLAGS="-fsanitize=address -fno-omit-frame-pointer -O2 -g"
-DCMAKE_CXX_FLAGS="-fsanitize=address -fno-omit-frame-pointer -O2 -g"
-DCMAKE_VERBOSE_MAKEFILE=1 -DSIMGEAR_SHARED=ON -DENABLE_IAX=OFF
-DENABLE_FGCOM=OFF -DSYSTEM_SQLITE=ON
make
sudo make install
#run
ASAN_SYMBOLIZER_PATH=/usr/lib/llvm-3.4/bin/llvm-symbolizer
ASAN_OPTIONS="symbolize=1 alloc_dealloc_mismatch=0" fgfs [options] 2>
asan_log.txt
#post asan_log.txt to developers mailing list and/or the issue tracker
The original idea and future developments
Our hope is that we can come up with a fairly safe subset of FlightGear settings that should work for 99% of our users, even on moderately old hardware. This would enable us to change the FlightGear defaults accordingly, and use these settings as a safe fallback alternative - so that FlightGear should no longer just crash for people without certain hardware/features.
There's a trend to expose more and more information via the property tree, to make the simulator increasingly runtime-configurable, and to allow subsystems to be re-initialized at runtime, so being able to start up FlightGear in some form of "safe" mode is generally a good thing and useful for anybody helping with troubleshooting on the forums, because it helps us exclude many potential problems. In the long run, such a profile could also be used as the foundation for a FlightGear Benchmark or some scripted regression suite to run FlightGear in a headless mode to help with release preparations, and FGCanvas.