Hi fellow wiki editors!

To help newly registered users get more familiar with the wiki (and maybe older users too) there is now a {{Welcome to the wiki}} template. Have a look at it and feel free to add it to new users discussion pages (and perhaps your own).

I have tried to keep the template short, but meaningful. /Johan G

Navdata cache

From FlightGear wiki
Revision as of 05:42, 31 October 2016 by Mhab (Talk | contribs) (Add another possibility to solve the cache startup problem, if everything else fails)

Jump to: navigation, search
This article is a stub. You can help the wiki by expanding it.
Caution  This feature is crucial to the FlightGear startup sequence, but is unfortunately infamous for rendering FlightGear non-startable under certain situations. Particularly, after experiencing a FlightGear crash or when installing/running multiple FlightGear instances at the same time[1]. It is assumed that this is due to lack of synchronization and missing atexit() handlers[2][3], so that the SQLite database may get corrupted under certain circumstances.

The way to resolve this issue is by renaming/deleting the navdata cache entirely (navdata_x_y.cache), renaming the folder containing $FG_HOME so that the cache gets rebuilt during the next start, or to use the powerful yet dangerous --restore-defaults command line option to reset all user settings to their defaults and to rebuild the cache.

On Windows another option is to go to yourusername/AppData/Roaming/flightgear.org, and delete the navdata file inside that folder. If it still crashes, rename the flightgear.org folder to flightgearBAK.orgBAK, which will fix the errors, but you lose your user settings, such as options you have set in sim (ALS, don't show fps / ms, etc), joystick settings, etc ... In case FG starts up well, you can restore these settings by copiing back the files from the backup directory.

Unfortunately, this bug is hard to reproduce. So if you are encountering this bug, please be sure to create a ticket, including instructions for reproducing the bug, so that this can be hopefully fixed.

Screen shot showing structure of $FG_HOME
Screen shot showing the SQLite3 schema of the nav cache using a GUI SQLite database editor

The navdata cache, navdb, (sometimes also navcache) is automatically built by FlightGear during startup by parsing/processing the gzipped nav.dat file and building a spatial SQLite-based database in $FG_HOME so that more efficient queries can be run at run-time, but, in particular when starting up/re-initializing FlightGear. Memory consumption is also lower, since we don't keep airports / fixes / taxiways / runways in memory until they're needed.

This feature have been available since FlightGear 2.10 released in February 2013.

Background

Screen shot showing several navcache (SQLite) files in $FG_HOME for various fgfs versions

We need a lot of data available at startup for position init. Parsing it from apt.dat and nav.dat is 'slow', but has been optimised over the years. (Parsing 20MB of text data each launch is still kind of crazy, though) Collecting it from many XML files would also be slow, but there's a general point that we should be decoupling availability from loading.

Cache initialization

Usually, the navdb will only be compiled once during the first start of the simulator - however, some changes may require rebuilding the cache on disk (see below for details).

Touching the navdata files will cause a rebuild, so rebuilds need to take a 'reasonable' amount of time. Shipping the binary data improves first-launch perceptions, but many applications do additional work on their first launch after an install, so it's not a top concern for the time being.

When the scenery paths change, we have to rebuild, because Airports XML data (from the <scenery>/Airports/ tree) is overlaid into the cache, replacing the data from apt.dat. This means when the search paths or order changes, we have to completely rebuild, because there's no way to restore the unmodified (prior to XML overlay) version.

Cquote1.png we should absolutely stop telling anyone to edit preferences.xml in FG_ROOT; any documentation or advice which says to should be changes ASAP.
Cquote2.png

Zakalawe considers the time and fragility to implement such a system much greater than the impact of a rebuild when switching scenery paths, which most people don't do very often.

The rebuild is purely a CPU / disk-bound operation, and for most people, it takes around 60 sec for debug and 30 sec for release times for a complete rebuild. (That time will be dramatically reduced if we get the taxiway data out of apt.dat in the future, since taxiways account for >80% of the lines in the file).

Usually the navdb should be under 200 MB and normally even a rebuild should not take much longer than a few minutes. However, on some platforms/OS the navdb rebuild is known to take exceptionally long, which seems to affect mainly Windows users - some of whom have reported the navcache rebuild taking more than 30 minutes.


Technical details

sqliteman (Qt based) showing navcache view

The nav-cache is versioned, it will be wiped when the cache schema version changes, but right now it’s not wiped when the flightgear version does.

The DB schema version is tracked internally in the file, and if the schema changes during a development version (3.3 or whatever) it will force a drop and rebuild.

The naming scheme is why the first run of a new stable release of FG always does a cache rebuild - which means you can run stable and dev versions side by side without continual cache rebuilds of course.

we /do/ also run a Sqlite verification check on the DB at launch time - this should catch many kinds of file-/index- level corruption (bad commit, etc), and in any case where we detect an issue, the solution is again to drop the cache and rebuild.

The internal format of the cache (for example the fact it's a SQL DB) is deliberately opaque. There are no guarantees made about the format of the file, the SQL scheme or anything. There is a version field of course, and we're not planning or expecting any changes, but we really don't want to be tied to the current scheme if we discover problems or want to add something else in the future.

The cache is stored in $FG_HOME/navdata.cache, and is rebuilt if the timestamps on any of the data files change (apt.dat, nav.dat, fix.dat and so on). When the cache needs to be rebuilt, startup will take a bit longer than before, but when the cache is valid, startup is much faster, especially for debug builds - because all the usual parsing/processing will be skipped, and the corresponding data will be read from the binary cache instead.

Known issues

Note  See also these bug reports:


The whole navdb build thing is proving to be the single most problematic feature added in the last couple of years affecting potentially all users, because unlike most additions, it's not an optional thing, i.e. it must be executed, and is needed for FG to work - given that, it would really make sense exploring to ship a pre-built navdb with future FlightGear releases, or at least factor out the corresponding code so that it can be provided as a standalone executable that is shipped as part of fgfs, so that it can be executed by the installer, without fgfs being affected by it inevitably - that way, troubleshooting related issues should also become much easier, in addition, this separation would facilitate having sync'ed navdbs across multiple instances (think virtual ATC, multiplayer, combat simulation, or even just a fsweekend/linuxTag-like setup).

This work would also greatly facilitate turning the navdb into a HLA/IPC-enabled service one day.[4]

all startup/repositioning logic needs navdb access - otherwise, it would be possible to also make the whole system optional during startup/runtime, i.e. via some kind of property, so that people could opt-out until certain issues are addressed. Obviously, that would break any conventional radio-navigation equipped aircraft, but also interactive mapping/charting avionics/dialogs, so may not be desirable ... Anyway, moving the navdb build into the release process should be possible, and would help address this, until the underlying C++ code has been ironed out[5]


Unfortunately the cache must be regenerated each time you change scenery paths. So you cannot pre-build it. Recent changes to remove dynamic airports from it should help significantly, but it needs to be made more fail-safe.

[6]


the issue has been around for so long, it would be better to recognie that the system needs to be separated/split up, so that a pre-built database can be provided, or so that a central navdb "service" (server) can be used - that way, it could still be a separate executable that doesn't cripple the fgfs experience for >= ~25% of our newcomers The whole thing started out as a navdb after all, there is no reason why we should be adding so many inter-dependencies to it until some of the key problems have been solved. Besides, this isn't the first time that fgfs had a cached navdb layer (anybody here remember the original metakit implementation from the early 2000s?) In its current form, this whole feature contributes to the majority of bug reports, and actual crashes, and is causing tons of support threads on the forum. Thus, it would definitely make sense to review what's there currently, and see if some things can be changed/re-implemented to make this a less critical feature[7]


Windows Issues

As as we know, it's a Windows-specific issue related to SQLite and/or the way $FG_HOME works on Windows: http://wiki.flightgear.org/$FG_HOME Most core developers, as well as most people who regularly build from source, are using non-Windows OS, i.e. are UNIX-based (MacOS/Linux etc), where the problem usually doesn't show up at all, which makes it hard to reproduce for people able to troubleshoot these things. And unfortunately, we don't really have any Windows-based core developers remaining - the main guy who would handle such compatibility issues on Windows used to be FredB who hasn't been seen in quite a while - and the main guy behind the navcache/navdb is also a Mac user, so not in a good position to reproduce these things, absent some complex VM setup.

Thus, if you are in any position to provide additional feedback/info using development tools (e.g. a debugger) that would indeed be very helpful. For now, it is even clear if this is related to the location of $FG_HOME on Windows (which has recently changed AFAIK), or if it's related to SQLite itself. Personally, I've been able to reproduce a handful of SQLite related issues - but I found those difficult to troubleshoot due to the way SQLite is used by FlightGear, i.e. without any logging support/options being exposed currently.

In general, the SQLite-navdb is a great idea - but in practice, it's been crippling FlightGear quite severely since the day it's been added for a number of users, especially those on Windows - simply because we're getting very little action-able feedback/community support, and core contributors who regularly build from source are usually not able to reproduce these issues, while also typically not being very familiar with potential Windows-level issues.

Again, it would really help to file an issue for this - or at least add to any existing (open) ones to raise some awareness of this, and hopefully help improve the situation over time.

Cquote1.png I'd sugest to review the atexit() handlers in $FG_SRC/Main to ensure that the SQLite/navcache properly cleans up all resources there, i.e. by closing files etc - as far as I remember, the navcache is not an actual/proper "SGSubsystem" - so maybe the corresponding ctors aren't getting called during segfaults/crashes - IIRC, it's implemented as a singleton: https://sourceforge.net/p/flightgear/flightgear/ci/49ce7cd318d00be44c43cffea028a6556104fe18/tree/src/Main/fg_init.cxx#l570
— Hooray (Dec 31st, 2014). Re: FATAL ERROR.
(powered by Instant-Cquotes)
Cquote2.png

Read only database

The SQLite-based navcache seems to have a tendency to get corrupted causing FlightGear to not start and give for example the following error messages:

  • Sqlite error : attempt to write a readonly database received from DELETE FROM groundnet_edge
  • Sqlite error : attempt to write a readonly database

The easiest way around these errors is to delete the navcache and let FlightGear automatically rebuild it the next startup. Depending on the amount of scenery available to FlightGear on your computer it may take some minutes.

Corruption due to crashes

The navcache also seems to get corrupted during segfaults/crashes, especially on Windows, but more recently also on OSX:

Cquote1.png Sqlite error : attempt to write a readonly database received from DELETE FROM groundnet_edge
— daniel36 (Jul 9th, 2015). Sqlite error.
(powered by Instant-Cquotes)
Cquote2.png
Cquote1.png at least on my 10.7 build, I’m seeing an issue with the nav-cache not initialising. This is actually good, because it gives me a way, finally, to trace down the ‘nav-cache never initialises’ bug that some folks have reported. But it will take a little time. Since the same binary works perfectly on my 10.10 box, I assume it’s something /very/ subtle, probably uninitialised memory or some tiny change in zlib’s gzread functions (since we use the system Zlib library which hence could be different between 10.7 and 10.10).
— James Turner (2015-02-18). Re: [Flightgear-devel] Release 3.4.0 is coming.
(powered by Instant-Cquotes)
Cquote2.png
Cquote1.png I'd sugest to review the atexit() handlers in $FG_SRC/Main to ensure that the SQLite/navcache properly cleans up all resources there, i.e. by closing files etc - as far as I remember, the navcache is not an actual/proper "SGSubsystem" - so maybe the corresponding ctors aren't getting called during segfaults/crashes - IIRC, it's implemented as a singleton:

https://sourceforge.net/p/flightgear/flightgear/ci/49ce7cd318d00be44c43cffea028a6556104fe18/tree/src/Main/fg_init.cxx#l570


— Hooray (Wed Dec 31). Re: FATAL ERROR.
(powered by Instant-Cquotes)
Cquote2.png

High Level Architecture

1rightarrow.png See High-Level Architecture for the main article about this subject.

Cquote1.png we'll need the capability to to have two clients access the NavDB concurrently - I've hit lots of errors trying to do this
— Stuart Buchanan (Dec 29th, 2015). [Flightgear-devel] HLA Update.
(powered by Instant-Cquotes)
Cquote2.png
Cquote1.png I'm looking forward to some feedback and Python code from Jean-Paul to start working on a NavDataCache interface, to create a 'nav_data_cache' or 'navdb' Python C/C++ module. I also had the idea of running this subsystem in a thread. This avoids adding more strain to the main loop, making FGPythonSys even more inconspicuous.
Cquote2.png
Cquote1.png it would be possibly to deprecate virtually all the custom Nasal APIs in favour of commands, but much harder to do that solely using properties. (Think about searching the NavDB or appending a waypoint to a flightplan). If we started moving in that direction, we don’t need to create separate APIs for multiple languages, they simply work using the properties + commands system, and the command system already has the exact set of property modification operations you proposed.
Cquote2.png
Cquote1.png it would conceptually be the right thing to do to turn the navdb/positioned stuff into a "client/server" subsystem where the navdb acts as a server that can handle client-requests asynchronously. The funny thing is that SQLite itself is obviously based on the whole "server-less" notion. From a design standpoint, the NavDB is basically an "environmental" subsystem, i.e. heavily reliant on the underlying terrain/scenery because it provides navaids - thus, in a distributed/MP setup it would actually make sense for multi-instance setups to query the same DB to replicate/sync state.
— Hooray (Jun 11th, 2014). Performance: .
(powered by Instant-Cquotes)
Cquote2.png
Cquote1.png There are so many similar programs that need to access this very data, that it is simply inefficient to copy/paste code and adapt that, and in a MP/VA or ATC scenario, you would also want to ensure that all "players" (clients) are using the same underlying database, which is why I believe that it would be better to have a true server/client model, where fgfs would only connect to a locally running database - which in the fg case, could even be running in a backgound thread (e.g. via HLA).

It is already unfortunate that the navcache is not a true subsystem and that its internal data dependencies are not formalied, so the whole thing could just as well be moved to simgear, so that OpenRadar/ATC-Pie can more easily reuse it (either as a binary library or via standalone program/networking (IPC)). Note that this is not a new discussion/idea, for some background info refer to the atlas project on sourceforge: http://sourceforge.net/p/atlas/feature-requests/10/

http://sourceforge.net/p/atlas/feature-requests/16/
— Hooray (Nov 16th, 2015). Re: Improving stability: No-scenery-emergency mode.
(powered by Instant-Cquotes)
Cquote2.png

Upcoming developments

we are looking at making the loading of this data asynchronous so startup is not delayed, but we also lack Windows developers who can investigate the issue, since Linux, Mac and some Windows boxes are absolutely fine.

There's future work to move even more data into the cache — for example parking positions — which will further help performance for FMS / map systems since we won't need to parse lots of XML data repeatedly.

Cquote1.png This makes me think I should do some validation on FG_HOME, i.e that it exists, and is writeable. That would have caught the UTF-8 / windows-local-8-bit encoding screw-up I had for 2.10, too. I think we have all the pieces to do this, will come up with something in the next few days.
Cquote2.png
Cquote1.png I need people to track down *why* the cache rebuild is being triggered. The possibilities are: a scenery path reconfiguration (added / removed / re-ordered scenery paths) or that one of the .dat files we build the cache from has been modified. All the .dat files (Navaids/nav.dat.gz, Airports/apt.dat.gz, and similar .dat.gz files) should have a modification timestamp that corresponds to when the program was installed. If people could confirm that's the case, it would be one useful step.


The next step after that is for someone with a debug build (compiled from source) to step through the cache startup code and see which conditional is tripping the rebuild.


— zakalawe (Thu Oct 17). Re: Loading forever: "loading navigation data".
(powered by Instant-Cquotes)
Cquote2.png
Cquote1.png Having the option would be awesome, because it would greatly simplify troubleshooting, as you say - so having additional startup options that explicitly override/ignore autosave.xml and ~/.fgfsrc would be awesome as it would be much more straightforward to exclude a plethora of potential problem sources, we could save tons of time that way - but I wouldn't make it part of the same option. Ideally, being able to ignore autosave/fgfsrc would be different command-line arguments.Preferably, we would have a "soft ignore" mode (retaining data) and a "delete" mode, which resets any local settings that may interfere with FG.
Cquote2.png
Cquote1.png I'll add a --rebuild-caches option and it can drop the aircraft and terrasync caches too.
Cquote2.png
Cquote1.png I mean the update cache which stops us spamming the server with update requests every launch - instead we only check the server once per 24-hour period, which greatly cuts down on nuisance update polls to the server.

The only thing I can't decide if this option should drop is the auto-save file. Then the options becomes a bit more of a 'reset to defaults', which might be more useful since people often end up with old values in their auto-save and again we don't have a UI to drop it.


Cquote2.png
Cquote1.png So I think only the following is missing:
  • make --restore-defaults kill the caches (nav-cache, aircraft cache and terrasync update cache)
  • add a --ignore-defaults which stops both read *and* writing of autosave.xml
  • log any XML config file we process

Cquote2.png

Discussion

Note  Also see A NavDB Web Service.
Cquote1.png We do not currently have a dedicated interface for exchanging internal data structures like the navdb, terrain data or images/video streams. However, something like that would be generally useful for a whole number of purposes and projects, so please feel free to leave a feature request at the tracker:

http://flightgear-bugs.googlecode.com/


Cquote2.png
Cquote1.png the navigation databases


Currently these are gzipped compressed text - they’re not small and currently we change them infrequently but the upstream sources does receive updates relatively often (monthly?). Again there is some code dependency here, since new versions could break things.


Cquote2.png
Cquote1.png My proposed initial solution, but this is where the discussion starts:

move the category [1] files, and potentially unzipped [2] files, into the main flightgear repository, under, say, ‘flightgear/data’. I don’t think a submodule gains us anything, all the files are tiny so it won’t bloat the repository. (And it avoids the complexity of forgetting to update the submodule - but if that issue can be solved, a submodule also works for me)

(We need to switch to unzipping the category [2] files anyway, for another improvement to startup performance, and once they are text rather than binary, Git will be able to diff them efficiently, hence my thought they can maybe be treated the same way. But I’m not sure. Also there is the option to start getting upstream updates more often, e.g. between FG releases)


Cquote2.png

FGPositioned

Cquote1.png it will become a base for the following:
  • FGAirport
  • FGRunway
  • FGFix
  • FGNavRecord
  • ATCData

— James Turner (2008-08-16). [Flightgear-devel] FGPositioned refactoring.
(powered by Instant-Cquotes)
Cquote2.png
Cquote1.png I've been carrying on with my experiments with my FGPositioned idea, and I now have several of my original steps done:
  • Fix, NavRecord, Airport and Runway all inherit the base class
  • they all live on the heap, where previously Runway and Fix were stack based, and hence rather heavy to work with
  • since the liftetime is now (generally) long, I can use a persistent spatial index (currently not Matthias', but it's an easy, internal change)
  • I can query all of the above in a unified fashion, and add more types easily (Jon Stockhill has obstacle data I can add in)
  • I've written a bunch of test cases, all of which pass identically on the mainline and with my changes applied

— James Turner (2008-09-06). [Flightgear-devel] FGPositioned update.
(powered by Instant-Cquotes)
Cquote2.png
Cquote1.png it's also the starting point for working on improving the airways code and creating a standard FGFlightPlan class - an airway or flightplan is essentially built out of FGPositioned objects, tagged with some extra data.
— James Turner (2008-08-16). [Flightgear-devel] FGPositioned refactoring.
(powered by Instant-Cquotes)
Cquote2.png

Access

You can use any SQLite editor/browser to access the navcache file and visualize it There are basically two ways to access this via Nasal 1) the "proper" way is to use/extend the NasalPositioned(_cpbind).cxx file in $FG_SRC/Scripting - with 2), being direct SQLite access using the sqlite module from the Nasal github repo.

Either way, you would need to modify/patch and rebuild fgfs. In general, I would recommend to file a feature request and/or raise this on the devel list.

Cquote1.png my suggestion would be to use an SQLite browser/editor and go to $FG_HOME to open the navcache there (see the wiki link above), you can then inspect the whole navcache - what you find in it is basically a mix up different sources (including Xplane data) - and most of it is available at run-time via FGPositioned APIs and their corresponding Nasal equivalents.

So given that we are talking about Python and Java here, you could either turn the FGPositioned C++ stuff into a reusable library for use by OpenRadar/ATC-Pie, or simply read the navcache directly. With the latter having the disadvantage that people need to have fgfs installed locally.

Otherwise, you could also extract the relevant data from the navcache and ship that with both ATC clients.

However, please see the wiki: the main guy behind the navcache is Zakalawe, and he would prefer if people didn't access/process the navcache directly but only use the FG-specific APIs.
— Hooray (Nov 16th, 2015). Re: Improving stability: No-scenery-emergency mode.
(powered by Instant-Cquotes)
Cquote2.png
Cquote1.png this opens up the question if the navcache should really be an integrated component, or if it might not be better to turn it into a standalone library/program that could be used by multiple processes, regardless of fgfs having to be installed/running, i.e. some form of true database.

Overall that would seem like the best solution to me, because the navcache is a feature that has severely crippled the fg experience for many new users, so that it would make sense to move this to a dedicated/standalone component and use it like a conventional database. There are so many similar programs that need to access this very data, that it is simply inefficient to copy/paste code and adapt that, and in a MP/VA or ATC scenario, you would also want to ensure that all "players" (clients) are using the same underlying database, which is why I believe that it would be better to have a true server/client model, where fgfs would only connect to a locally running database - which in the fg case, could even be running in a backgound thread (e.g. via HLA). It is already unfortunate that the navcache is not a true subsystem and that its internal data dependencies are not formalied, so the whole thing could just as well be moved to simgear, so that OpenRadar/ATC-Pie can more easily reuse it (either as a binary library or via standalone program/networking (IPC)). Note that this is not a new discussion/idea, for some background info refer to the atlas project on sourceforge: A NavDB Web Service

If in doubt, get in touch with Zakalawe on the devel list, the navcache is his baby after all ...
— Hooray (Nov 16th, 2015). Re: Improving stability: No-scenery-emergency mode.
(powered by Instant-Cquotes)
Cquote2.png

Accessing via Nasal

Ambox move.png
It has been suggested that this article be moved to Nasal library. Discuss this move.

Several Nasal functions exist to query the navdata cache of 'positioned' objects, i.e items with a defined location in the simulation world. (These objects all inherit from FGPositioned internally). The functions return either one, or a vector, of wrapped Nasal objects. These are efficient, but unlike a hash they cannot simply be debug.dump() to view all their members and methods, many of which are computed in a lazy fashion (i.e. on demand).

When the query functions take a position, the default value is the current aircraft location. An alternative location can be supplied by passing two number (lat, lon), a Geo.Coord object, or any positioned object or waypoint retrieved from another query.

findAirportsWithinRange()
Find all airports within a specified range (in NM) of the current position or explicit position
findAirportsByICAO()
Find airports matching a complete or partial ICAO code. In particular this can search for all airports starting with a two or three letter prefix.
navinfo()
Return a list of navaids near a location by type and ident. Type should be 'fix', 'vor', 'ndb', 'dme'
findNavaidsWithinRange()
Search for navaids within a particular range (in NM), optionally limited by type. This provides a 'find the closest ILS' function
findNavaidByFrequency()
find the closest navaid (of a particular type) matching an exact frequency
findNavaidsByFrequency()
find all the navaids matching a particular frequency and optional type, sorted by distance from the search location.

All positioned objects returned by the above methods have the following members:

id
Identifier - ICAO code for airports, published ident for navaids and fixes
lon
degrees longitude
lat
degrees latitude

Depending on type, the following members are also available:

name
the full name of the airport or navaid if one is defined
elevation
the ASL elevation of the object in feet

For navaids, the following members are available:

frequency
the navaid frequency in kHz
type
the navaid type as a string: vor, dme, loc, ils, ndb
course
the degrees course associated with the navaid, for localiser and ILS stations
References
  1. Thorsten Renk (Nov 12th, 2014). [Flightgear-devel] Random oddities.
  2. Hooray (Wed Dec 31, 2014 9:24 am). FATAL ERROR.
  3. Hooray (Wed Oct 21, 2015 1:55 pm). http://forum.flightgear.org/viewtopic.php?f=35&t=27364&p=261267&hilit=#p261267.
  4. Hooray  (Aug 25th, 2016).  Re: takes a long time to open .
  5. Hooray  (Aug 26th, 2016).  Re: takes a long time to open .
  6. bugman  (Aug 27th, 2016).  Re: takes a long time to open .
  7. Hooray  (Aug 27th, 2016).  Re: takes a long time to open .