CrashRpt

This article is a stub. You can help the wiki by expanding it.

See Towards better troubleshooting for the main article about this subject.

Note  James' crash report system, is still in a preliminary state. Anyway, until this system is improved, the Issue tracker tickets system is the way to communicate bugs to the developers. the " crash report system", this the window that pops up and ask you if you want to send the report, is basically useless in as far as you getting help with your issue. It will be used to analye trends and larger scoped problems. To help analyze and solve your issue you need to do a couple things. One is to file a detailed report to the "bug tracker". There is also a link in the above masthead.

Status

As of 12/2016, James has re-enabled (after recompiling for MSVC2015) the CrashRpt tool for our Windows nightlies (and in the next 2017.1 release). We have a recurring issue that James never properly setup the PDB archiving and server-side infrastructure to generated symbolicated crash reports automatically. CrashRpt includes a tool to do this, and James has a web-server + web space, but running on Linux of course, and the symbolification process requires Windows. If someone wanted to contribute, figuring out how to turn the reports that arrive on my web-server into useful crash-traces would be an invaluable tool for seeing how the software performs in the real world.

the Windows builds (and the 3.0 release, assuming no issues are encountered) now includes a crash-reporting framework, to help getting higher quality crash reports }}

If you experience a crash, please send a report using the dialog which appears - you can provide some additional info if you wish. Your log file, some system info (processor, time of day, Windows version, path to fgfs.exe) and a crash dump are zipped and sent to a server. Using some additional steps a real back-trace can be produced. (We need to write a privacy policy document for this - currently the URL is a 404 - basically saying we promise not to do anything evil with the report info, although I’m not sure how much evil is possible since it’s the same info we ask for in a bug-report - I have switched off the options to include registry contents, and FGFS doesn’t touch the registry anyway) (For the curious, I’m not using Google Breakpad because it’s huge and seems somewhat inactive, instead I’m using CrashRpt which is very compact, completely open-source and new-BSD-licensed: https://code.google.com/p/crashrpt/ - although we do need to include a copy of their license.txt file somewhere in the installer)

CrashRpt was something James wanted to get on the release branch without being sure whether to disable it for non-release branches entirely, or support it in the Windows nightly builds, where it would potentially be useful. (In which case there are further change to be done, since the nightlies use a different installer script)

Originally James only enabled CrashRpt for release builds, not nightlies, but it has been enabled since then. Also James explicitly *disabled* the video capture support, because he felt the overhead of it would be too invasive when running FG, and the amount of video you’d need to record to make any difference to most crashes would be large.

Help wanted

There is one other Windows area James could really use some help with - he added CrashRpt integration several releases ago and has a web server full of crash data.

To make this useful someone with Windows skills needs to write some tooling to automatically match a crash report to the archived PDB file and hence produce a usable back-trace.

We could then start analyzing the most common sources of crashes. If that’s something you have any interest in, James can explain in more detail.

someone who knows Windows (and has some time available) needs to actually glue everything together using those tools and scripts. Ideally in a way we can run on the Jenkins slave or similar as an hourly job. That’s the current piece of the flow I never found time to work on myself, yet.

the build-ID need to be matched to the debug symbols (.PDB) which the Jenkins Windows build archives for releases. What’s missing as I said in the emails a long time ago, is the scripting to turn mini-dump + PDB files into a a back trace with function names, if not line numbers. That would be a huge benefit. I’m happy to give anyone read access to my server to try this, and adjust where / how Jenkins archives the PDBs - but not being on Windows, my energy and ability to move this forward myself is very low.

James is imagining it could be done as a batch job:

• connect to his web-server and download all new reports
• for each one, find / cache the corresponding PDB for fgfs.exe
• symbolicate
• upload symbolised crashes to ‘somewhere’ (could be James' web host again)

Usually it is possible to do some clustering on such crashes, to work out which ones are common. The crash reports also include the log up to that point, so we should have a fair bit of info I hope. So, volunteers much appreciated for this!

Crash Report Analysis

there are the following solutions:

1. get them from James' server and process them manually (e.g. via the script I mentioned in my previous e-mail, altered as necessary): pros: no changes in existing libraries/infrastructure, dumps remain private and accessible only to trusted developers; cons: Windows-only, some manual steps and script editing are needed;
2. use an automated solution like Dr Dump [1], which can be configured to accept CrashRpt reports; pros: only a small code change is needed, the service automatically analyzes crash reports and groups them according to their stack traces (it is only needed to upload .pdb symbols); cons: Windows-only, possible privacy concerns (dumps might contain portions of memory storing personal data, and some people might not be comfortable with storing them on a third-party service);
3. change the crash reporter to Google Breakpad [2]/Crashpad [3] and use an already developed frontend, like Socorro [4] or the Mini Breakpad Server [5], or some custom scripting; pros: cross-platform bug reports; cons: integrating Breakpad is not trivial (as well as huge [6]); Socorro is heavyweight (at least in my opinion).

• [1] https://drdump.com/
• [2] https://chromium.googlesource.com/breakpad/breakpad/
• [3] https://crashpad.chromium.org/index.html
• [4] https://wiki.mozilla.org/Socorro
• [5] https://github.com/electron/mini-breakpad-server
• [6] http://sourceforge.net/p/flightgear/mailman/message/31877484/

MD5 errors are normal - the address I gave you is the one where the error reports are uploaded by CrashRpt itself (I should have specified that), and the crashrpt.php script requires several parameters to be passed, including an MD5 hash of the minidump (I think), hence the message. There is currently no public interface to get crash reports.

Right now, they are (unfortunately) archived there (see the end of [1]). You can help by writing a tool to match crash reports to PDB files; I've written a small script [2] which could help you get started.

[1] http://article.gmane.org/gmane.games.flightgear.devel/77237 [2] http://article.gmane.org/gmane.games.flightgear.devel/77508

Might sth like a bug-report (or maybe crash-report?) feature be of help? We got a lot of information in the property-tree, like GPU-drivers etc. For instance, do we have OS and stuff like that too? Might be fairly easy to put this important info into a tmp-file, and in the case we crash, add it to some kind of http-request to flightgear.org. An experience-report might be of help, what frequently asked questions do you pose all the time? + the red box... — chris (Nov 19th, 2015). Re: [Flightgear-devel] Some thoughts about the release process. (powered by Instant-Cquotes)

people have been working on features to provide back traces to developers without end-users having to be aware of the whole process, as in the whole crashrpt thing that Zakalawe implemented - but all this is still in its infancy unfortunately. Just keep in mind that FlightGear users are not dealing with a finished or "polished" product - things are very much in flux, and people can be part of the evolutionary process here. — Hooray (Apr 22nd, 2014). Re: Forum communication. (powered by Instant-Cquotes)

Related

• forum search for crashrpt FlightGear Forum

References

References