Navdata cache

This article or section contains out-of-date information Please help improve this article by updating it. There may be additional information on the talk page.

See also Fgfs.pid.

Note  On Windows, versions of Flightgear up to 2017.1.1 can have problems restarting if the simulator did not close normally the last time it ran. Typically the error message suggests that there is a problem with a read-only navigation data cache. In fact, the underlying problem is the failure of a previous run to delete a status file. Flightgear then assumes that another instance of the program is running and it makes the $FG_HOME directory read-only. The solution is to delete the fgfs.pid file from $FG_HOME and restart Flightgear. It should not be necessary to delete any other files or directories.

The navdata cache, navdb, (sometimes also navcache) is automatically built by FlightGear during startup by parsing/processing the gzipped nav.dat file and building a spatial SQLite-based database in $FG_HOME so that more efficient queries can be run at run-time, but, in particular when starting up/re-initializing FlightGear. Memory consumption is also lower, since we don't keep airports / fixes / taxiways / runways in memory until they're needed.

This feature have been available since FlightGear 2.10 released in February 2013.

Background

Screen shot showing several navcache (SQLite) files in $FG_HOME for various fgfs versions

We need a lot of data available at startup for position init. Parsing it from apt.dat and nav.dat is 'slow', but has been optimised over the years. (Parsing 20MB of text data each launch is still kind of crazy, though) Collecting it from many XML files would also be slow, but there's a general point that we should be decoupling availability from loading.

Cache initialization

Usually, the navdb will only be compiled once during the first start of the simulator - however, some changes may require rebuilding the cache on disk (see below for details).

Touching the navdata files will cause a rebuild, so rebuilds need to take a 'reasonable' amount of time. Shipping the binary data improves first-launch perceptions, but many applications do additional work on their first launch after an install, so it's not a top concern for the time being.

When the scenery paths change, we have to rebuild, because Airports XML data (from the <scenery>/Airports/ tree) is overlaid into the cache, replacing the data from apt.dat. This means when the search paths or order changes, we have to completely rebuild, because there's no way to restore the unmodified (prior to XML overlay) version.

we should absolutely stop telling anyone to edit preferences.xml in FG_ROOT; any documentation or advice which says to should be changes ASAP. — zakalawe (Sat Oct 26). Re: NavCache:init failed:Sqlite error:Sqlite API abuse. (powered by Instant-Cquotes)

Zakalawe considers the time and fragility to implement such a system much greater than the impact of a rebuild when switching scenery paths, which most people don't do very often.

The rebuild is purely a CPU / disk-bound operation, and for most people, it takes around 60 sec for debug and 30 sec for release times for a complete rebuild. (That time will be dramatically reduced if we get the taxiway data out of apt.dat in the future, since taxiways account for >80% of the lines in the file).

Usually the navdb should be under 200 MB and normally even a rebuild should not take much longer than a few minutes. However, on some platforms/OS the navdb rebuild is known to take exceptionally long, which seems to affect mainly Windows users - some of whom have reported the navcache rebuild taking more than 30 minutes.

Technical details

The nav-cache is versioned, it will be wiped when the cache schema version changes, but right now it’s not wiped when the flightgear version does.

The DB schema version is tracked internally in the file, and if the schema changes during a development version (3.3 or whatever) it will force a drop and rebuild.

The naming scheme is why the first run of a new stable release of FG always does a cache rebuild - which means you can run stable and dev versions side by side without continual cache rebuilds of course.

we /do/ also run a Sqlite verification check on the DB at launch time - this should catch many kinds of file-/index- level corruption (bad commit, etc), and in any case where we detect an issue, the solution is again to drop the cache and rebuild.

The internal format of the cache (for example the fact it's a SQL DB) is deliberately opaque. There are no guarantees made about the format of the file, the SQL scheme or anything. There is a version field of course, and we're not planning or expecting any changes, but we really don't want to be tied to the current scheme if we discover problems or want to add something else in the future.

The cache is stored in $FG_HOME/navdata.cache, and is rebuilt if the timestamps on any of the data files change (apt.dat, nav.dat, fix.dat and so on). When the cache needs to be rebuilt, startup will take a bit longer than before, but when the cache is valid, startup is much faster, especially for debug builds - because all the usual parsing/processing will be skipped, and the corresponding data will be read from the binary cache instead.

Troubleshooting =

Note  A copy of your navdata.cache would be appreciated (the broken one), since of the two copies James was sent so far, neither were corrupted - they both passed the SQLite integrity check. This has led James to think the issue is permissions, not corruption of the DB. Of course, there could be multiple issues. If you can find the suspect navcache, please upload it to: https://swanson.kdab.com/owncloud/index.php/s/P0DMwvV6TdQR5CW and let James know.

High Level Architecture

See High-Level Architecture for the main article about this subject.

we'll need the capability to to have two clients access the NavDB concurrently - I've hit lots of errors trying to do this — Stuart Buchanan (Dec 29th, 2015). [Flightgear-devel] HLA Update. (powered by Instant-Cquotes)

I'm looking forward to some feedback and Python code from Jean-Paul to start working on a NavDataCache interface, to create a 'nav_data_cache' or 'navdb' Python C/C++ module. I also had the idea of running this subsystem in a thread. This avoids adding more strain to the main loop, making FGPythonSys even more inconspicuous. — Edward d'Auvergne (Jan 26th, 2016). Re: [Flightgear-devel] A FGPythonSys implementation: The embedded Python interpreter as a FlightGear subsystem.. (powered by Instant-Cquotes)

it would be possibly to deprecate virtually all the custom Nasal APIs in favour of commands, but much harder to do that solely using properties. (Think about searching the NavDB or appending a waypoint to a flightplan). If we started moving in that direction, we don’t need to create separate APIs for multiple languages, they simply work using the properties + commands system, and the command system already has the exact set of property modification operations you proposed. — James Turner (Jan 26th, 2016). Re: [Flightgear-devel] Designing a thread-safe property tree API (was Re: A FGPythonSys implementation: ...). (powered by Instant-Cquotes)

it would conceptually be the right thing to do to turn the navdb/positioned stuff into a "client/server" subsystem where the navdb acts as a server that can handle client-requests asynchronously. The funny thing is that SQLite itself is obviously based on the whole "server-less" notion. From a design standpoint, the NavDB is basically an "environmental" subsystem, i.e. heavily reliant on the underlying terrain/scenery because it provides navaids - thus, in a distributed/MP setup it would actually make sense for multi-instance setups to query the same DB to replicate/sync state. — Hooray (Jun 11th, 2014). Performance: . (powered by Instant-Cquotes)

There are so many similar programs that need to access this very data, that it is simply inefficient to copy/paste code and adapt that, and in a MP/VA or ATC scenario, you would also want to ensure that all "players" (clients) are using the same underlying database, which is why I believe that it would be better to have a true server/client model, where fgfs would only connect to a locally running database - which in the fg case, could even be running in a backgound thread (e.g. via HLA). It is already unfortunate that the navcache is not a true subsystem and that its internal data dependencies are not formalied, so the whole thing could just as well be moved to simgear, so that OpenRadar/ATC-Pie can more easily reuse it (either as a binary library or via standalone program/networking (IPC)). Note that this is not a new discussion/idea, for some background info refer to the atlas project on sourceforge: http://sourceforge.net/p/atlas/feature-requests/10/ http://sourceforge.net/p/atlas/feature-requests/16/ — Hooray (Nov 16th, 2015). Re: Improving stability: No-scenery-emergency mode. (powered by Instant-Cquotes)

Upcoming developments

we are looking at making the loading of this data asynchronous so startup is not delayed, but we also lack Windows developers who can investigate the issue, since Linux, Mac and some Windows boxes are absolutely fine.

There's future work to move even more data into the cache — for example parking positions — which will further help performance for FMS / map systems since we won't need to parse lots of XML data repeatedly.

This makes me think I should do some validation on FG_HOME, i.e that it exists, and is writeable. That would have caught the UTF-8 / windows-local-8-bit encoding screw-up I had for 2.10, too. I think we have all the pieces to do this, will come up with something in the next few days. — zakalawe (Wed Oct 30). Re: NavCache:init failed:Sqlite error:Sqlite API abuse. (powered by Instant-Cquotes)

I need people to track down *why* the cache rebuild is being triggered. The possibilities are: a scenery path reconfiguration (added / removed / re-ordered scenery paths) or that one of the .dat files we build the cache from has been modified. All the .dat files (Navaids/nav.dat.gz, Airports/apt.dat.gz, and similar .dat.gz files) should have a modification timestamp that corresponds to when the program was installed. If people could confirm that's the case, it would be one useful step. The next step after that is for someone with a debug build (compiled from source) to step through the cache startup code and see which conditional is tripping the rebuild. — zakalawe (Thu Oct 17). Re: Loading forever: "loading navigation data". (powered by Instant-Cquotes)

Having the option would be awesome, because it would greatly simplify troubleshooting, as you say - so having additional startup options that explicitly override/ignore autosave.xml and ~/.fgfsrc would be awesome as it would be much more straightforward to exclude a plethora of potential problem sources, we could save tons of time that way - but I wouldn't make it part of the same option. Ideally, being able to ignore autosave/fgfsrc would be different command-line arguments.Preferably, we would have a "soft ignore" mode (retaining data) and a "delete" mode, which resets any local settings that may interfere with FG. — Hooray (Sat Oct 26). Re: NavCache:init failed:Sqlite error:Sqlite API abuse. (powered by Instant-Cquotes)

I'll add a --rebuild-caches option and it can drop the aircraft and terrasync caches too. — zakalawe (Thu Oct 24). Re: NavCache:init failed:Sqlite error:Sqlite API abuse. (powered by Instant-Cquotes)

I mean the update cache which stops us spamming the server with update requests every launch - instead we only check the server once per 24-hour period, which greatly cuts down on nuisance update polls to the server. The only thing I can't decide if this option should drop is the auto-save file. Then the options becomes a bit more of a 'reset to defaults', which might be more useful since people often end up with old values in their auto-save and again we don't have a UI to drop it. — zakalawe (Sat Oct 26). Re: NavCache:init failed:Sqlite error:Sqlite API abuse. (powered by Instant-Cquotes)

So I think only the following is missing: make --restore-defaults kill the caches (nav-cache, aircraft cache and terrasync update cache) add a --ignore-defaults which stops both read *and* writing of autosave.xml log any XML config file we process — zakalawe (Sat Oct 26). Re: NavCache:init failed:Sqlite error:Sqlite API abuse. (powered by Instant-Cquotes)

Discussion

Note  Also see A NavDB Web Service.

We do not currently have a dedicated interface for exchanging internal data structures like the navdb, terrain data or images/video streams. However, something like that would be generally useful for a whole number of purposes and projects, so please feel free to leave a feature request at the tracker: http://flightgear-bugs.googlecode.com/ — Hooray (Mon Jul 15). Re: How to access navigation data in FlightGear ?. (powered by Instant-Cquotes)

the navigation databases Currently these are gzipped compressed text - they’re not small and currently we change them infrequently but the upstream sources does receive updates relatively often (monthly?). Again there is some code dependency here, since new versions could break things. — James Turner (2015-02-25). [Flightgear-devel] FGData splitting / assembling the base package. (powered by Instant-Cquotes)

My proposed initial solution, but this is where the discussion starts: move the category [1] files, and potentially unzipped [2] files, into the main flightgear repository, under, say, ‘flightgear/data’. I don’t think a submodule gains us anything, all the files are tiny so it won’t bloat the repository. (And it avoids the complexity of forgetting to update the submodule - but if that issue can be solved, a submodule also works for me) (We need to switch to unzipping the category [2] files anyway, for another improvement to startup performance, and once they are text rather than binary, Git will be able to diff them efficiently, hence my thought they can maybe be treated the same way. But I’m not sure. Also there is the option to start getting upstream updates more often, e.g. between FG releases) — James Turner (2015-02-25). [Flightgear-devel] FGData splitting / assembling the base package. (powered by Instant-Cquotes)

FGPositioned

it will become a base for the following: FGAirport FGRunway FGFix FGNavRecord ATCData — James Turner (2008-08-16). [Flightgear-devel] FGPositioned refactoring. (powered by Instant-Cquotes)

I've been carrying on with my experiments with my FGPositioned idea, and I now have several of my original steps done: Fix, NavRecord, Airport and Runway all inherit the base class they all live on the heap, where previously Runway and Fix were stack based, and hence rather heavy to work with since the liftetime is now (generally) long, I can use a persistent spatial index (currently not Matthias', but it's an easy, internal change) I can query all of the above in a unified fashion, and add more types easily (Jon Stockhill has obstacle data I can add in) I've written a bunch of test cases, all of which pass identically on the mainline and with my changes applied — James Turner (2008-09-06). [Flightgear-devel] FGPositioned update. (powered by Instant-Cquotes)

it's also the starting point for working on improving the airways code and creating a standard FGFlightPlan class - an airway or flightplan is essentially built out of FGPositioned objects, tagged with some extra data. — James Turner (2008-08-16). [Flightgear-devel] FGPositioned refactoring. (powered by Instant-Cquotes)

Access

You can use any SQLite editor/browser to access the navcache file and visualize it There are basically two ways to access this via Nasal 1) the "proper" way is to use/extend the NasalPositioned(_cpbind).cxx file in $FG_SRC/Scripting - with 2), being direct SQLite access using the sqlite module from the Nasal github repo.

Either way, you would need to modify/patch and rebuild fgfs. In general, I would recommend to file a feature request and/or raise this on the devel list.

my suggestion would be to use an SQLite browser/editor and go to $FG_HOME to open the navcache there (see the wiki link above), you can then inspect the whole navcache - what you find in it is basically a mix up different sources (including Xplane data) - and most of it is available at run-time via FGPositioned APIs and their corresponding Nasal equivalents. So given that we are talking about Python and Java here, you could either turn the FGPositioned C++ stuff into a reusable library for use by OpenRadar/ATC-Pie, or simply read the navcache directly. With the latter having the disadvantage that people need to have fgfs installed locally. Otherwise, you could also extract the relevant data from the navcache and ship that with both ATC clients. However, please see the wiki: the main guy behind the navcache is Zakalawe, and he would prefer if people didn't access/process the navcache directly but only use the FG-specific APIs. — Hooray (Nov 16th, 2015). Re: Improving stability: No-scenery-emergency mode. (powered by Instant-Cquotes)

this opens up the question if the navcache should really be an integrated component, or if it might not be better to turn it into a standalone library/program that could be used by multiple processes, regardless of fgfs having to be installed/running, i.e. some form of true database. Overall that would seem like the best solution to me, because the navcache is a feature that has severely crippled the fg experience for many new users, so that it would make sense to move this to a dedicated/standalone component and use it like a conventional database. There are so many similar programs that need to access this very data, that it is simply inefficient to copy/paste code and adapt that, and in a MP/VA or ATC scenario, you would also want to ensure that all "players" (clients) are using the same underlying database, which is why I believe that it would be better to have a true server/client model, where fgfs would only connect to a locally running database - which in the fg case, could even be running in a backgound thread (e.g. via HLA). It is already unfortunate that the navcache is not a true subsystem and that its internal data dependencies are not formalied, so the whole thing could just as well be moved to simgear, so that OpenRadar/ATC-Pie can more easily reuse it (either as a binary library or via standalone program/networking (IPC)). Note that this is not a new discussion/idea, for some background info refer to the atlas project on sourceforge: A NavDB Web Service If in doubt, get in touch with Zakalawe on the devel list, the navcache is his baby after all ... — Hooray (Nov 16th, 2015). Re: Improving stability: No-scenery-emergency mode. (powered by Instant-Cquotes)

Accessing via Nasal

See Nasal library for the main article about this subject.

Several Nasal functions exist to query the navdata cache of 'positioned' objects, i.e items with a defined location in the simulation world. (These objects all inherit from FGPositioned internally). The functions return either one, or a vector, of wrapped Nasal objects. These are efficient, but unlike a hash they cannot simply be debug.dump() to view all their members and methods, many of which are computed in a lazy fashion (i.e. on demand).

When the query functions take a position, the default value is the current aircraft location. An alternative location can be supplied by passing two number (lat, lon), a Geo.Coord object, or any positioned object or waypoint retrieved from another query.

References