TerraSync

Not to be confused with TerraGear, a toolset to generate scenery.

To see the terrain below your aircraft, you have to install the respective scenery. This can happen by downloading certain bits of scenery before flying as described in the article installing scenery.

Alternatively, if you have a steady and reasonably fast internet connection, you can use TerraSync. It is a utility that automatically downloads the newest version of the needed FlightGear scenery while the simulator is running. TerraSync runs in the background (optionally as a separate process), monitors your position, and downloads (or updates) the latest scenery from the master scenery server "just in time". For some time now TerraSync has been integrated into the core FlightGear process, so there is no need to deal with TerraSync for the typical user.

The master repository for TerraSync, i.e. the online resource from which TerraSync downloads its files, is synchronized with the FlightGear Scenery Database once a day. So when using TerraSync, you will always have

1. the latest .stg-files, which tell FlightGear where to place an object
2. the latest static models for objects. (Static models define unique objects that exist in one place only, such as famous buildings or landmarks.)
3. the latest shared models for objects. (Generic models used more than once in different places, each can represent many different objects, like generic houses or ships)

Announcements

Also as a result from FSweekend, we are going to test some ideas how to distribute osm2city generated scenery with terrasync. As a first step I have just added a folder called Objects_1 containing the osm2city buildings for e005n46 up to e010n48 [1]. People using the fgfs integrated terrasync should not (yet) notice any difference. When using terrasync.py you should notice download of files in those folders, however they do not (yet) show up in the scenery. We probably throw away some or all files in that directory without prior notice, so please don't rely on them or make backup copies. Torsten

News

Terrasync started out as a standalone rsync-based utility. However, there doesn't appear to be any good/robust multi-platform librsync available for our use. We wanted to embed terrasync right into the core code so we could better coordinate downloading tiles and then loading them immediately into the sim (versus sometimes running into no tiles or partial tiles if an external tool was running rsync.)

as some of you have probably noticed, some coding is currently happening in the terrasync corner, mostly by James and some parts by Torsten. The main intention is to get rid of the SVN protocol to distribute scenery but use easier to implement protocols. The idea is to use plain HTTP as the access method for files required by terrasync. This should make it much easier for hosts to set up a mirror, probably enables us to use a CDN for load balancing and also makes the client code slim.

The intention is to migrate to a pure HTTP solution for TerraSync (instead of SVN) after 2016.1 is released, since this will give us many more mirroring and hosting options. The cache files are tiny and reflect the current revision of each file in the local copy; they’re analogous to the data a real SVN client stores in its ‘.svn’ directory in the root of the checkout. I expect the pure HTTP solution will need some very similar files to record the HTTP cache stamps of files it downloads - either one file at the root of the tree, or per-directory. (There’s advantages and disadvantages to both).

• supporting the various possible backends inside the code is a headache, so the intention is to support /only/ the HTTP method inside FlightGear directly, aka ‘built-in terrasync’
• one of the major benefits of the HTTP system, is that is uses SHA1 hashes instead of SVN revisions, to identify files, and there is no base (server) URL stored inside the downloaded directories. This means you can prep a terrasync directory with files downloaded by any mechanism you like (DVD, BitTorrent, copying from another machine) and providing the SHA1 hashes match, the files will not be download again. Any files which are out of date will be refreshed, but only those files. This also means switching from SVN -> HTTP does /not/ download everything again.

Switching servers, even within the same FG session, is also possible and again does not result in any re-downloads, unlike the SVN setup where the server URL is part of the repository structure.

• the terrasync /network protocol/ will remain, which means you can write (in a wide ranges of languages, including Python) your own external terrasync engine, using rsync or any other mechanism you like. My hope is this provides a good and maintainable extension point for very advanced users, while the built-in HTTP mechanism will be be simple and robust (thanks to be being hash-based) for 95% of users.

We already have three mirrors of the complete scenery online, one at sourceforge and two at privately owned servers. The sourceforge mirror receives daily updates from the main scenery export from http://scenery.flightgear.org/ and all other mirrors pull their data from the sourceforge server. We decided to drop the previous method of telling the client about it's nearest mirror by querying a web service ( http://scenery.flightgear.org/svn-server) in favour of using a more stable service provider of the internet: the domain name service (DNS) system. The fgfs client will query a fancy resource record (RR) type of NAPTR (network authority pointer, https://en.wikipedia.org/wiki/NAPTR_record) for the dns name terrasync.flightgear.org and analye the entries to find the best terrasync mirror. Those entries are already configured and can be examined here: https://toolbox.googleapps.com/apps/dig/#ANY/terrasync.flightgear.org This technique should enable us to provide different sets of scenery, eventually even for only parts of the world in a much easier fashion that it would have been the case with SVN. It also spares many gigabytes on the scenery mirror as they don't have to keep all the 50,000+ revisions we already carry around. I'll start adding the required code over the next couple of days to have it running hopefully for the 2016.2 release. I'll post a message to this list when I flip the final switch and enable the new terrasync. In the mean time, you might see some build failures on Jenkins or other hickups of the system. I promise to fix those as quick as possible. BTW: If you use terrafs, you already receive your data from the new http mirrors, although not use the DNS service resolver.

the http download method will function very much as before where only the changed files (if any) are downloaded. The svn method requires 2x the actual space because of the way svn works under the hood. Git typically requires that you download every version ever created to get the latest copy (there are work arounds if you just want the most recent version.) So the end result for the average user that just wants to download and use the scenery is that the http method will be lighter weight overall. It also simplifies the code inside FlightGear quite a bit and should make the synchroniation more robust. And as mentioned before, it should simplify the task of setting up a scenery mirror for those that wish to do that. As before, power users will have multiple options for updating scenery or mixing and matching from different sources if they choose.

The technique of the http repository is much simpler than that of the svn repository and it will be pretty simple to create a local mirror or to download chunks of data. With a bit of to-be-developed scripting (python e.g.) there is no need to download existing data again. I don't know how terramaster works but I am sure it is easy to adapt to the new http access method.

The goal is to completely remove SVN and only use the HTTP repository. I'd like to reach that goal better sooner than later as we currently have two single-points of failure for the SVN-service. The first is the http://scenery.flightgear.org/svn-service web service (runs on a private web server) and the second is the one-and-only svn-server available for public access, also hosted on a private server. If either on of those fails, the SVN-terrasync is dead. I have learned the hard way that not having a fallback is a bad idea.

Torsten pushed a patch to SimGear and included a bunch of debug messages in the suspicious loop. (They only pop up with --log-level=debug - also add --log-class=terrasync to keep output sane)

we now have C++ and Python reference examples, both hopefully easy to hack on. BTW, the C++ tool is here: https://sourceforge.net/p/flightgear/simgear/ci/next/tree/simgear/io/http_repo_sync.cxx Again, enhancements are welcome.

References

terrasync.py

First of all: you will now find it at its new home here: https://sourceforge.net/p/flightgear/flightgear/ci/next/tree/scripts/python/TerraSync/terrasync.py

The script now is able to provide a complete mirror of the scenery data also known as terrasync scenery or just terrascenery.

Caution  For the impatient, before you try it out: If you run this script for the first time, it will download more than 90GB of data and more than 1,800,000 files in 40,000 folders to your hard disk. It will run for hours, probably days (depending on your internet connection).

If you already have at least a partial set of scenery data from flightgear's built-in terrasync or even a full SVN checkout, existing and up-to-date files will be reused and are not being downloaded a second time.

How does it work

The HTTP terrascenery server holds all the well-know files required for scenery, organied in the also well-know directory structure (/Models, /Objects, /Terrain and /Airports) with all their subdirectories. The content of the /Models folder for example is here: http://flightgear.sourceforge.net/scenery/Models/ Each folder contains a (hidden) file called .dirindex (e.g. http://flightgear.sourceforge.net/scenery/Models/.dirindex) that describes the content of that folder by listing it's files, subdirectories and checksums of the files and their subdirectories' .dirindex files. The terrasync.py script does the following (simplified):

• a) download the file at the root of the scenery-url ( http://flightgear.sourceforge.net/scenery/.dirindex)
• b) walks into any subdirectory listed within the .dirindex
• c) compares the sha1sum of listed files with the local copy and downloads them on mismatch
• d) for listed subdirectories, calls recursively step b)

Even if all scenery files exist on the server, there are initially still approx. 40,000 .dirindex files missing on your drive and downloading will take hours. To speed up the procedure once you have all .dirindex files locally, try running terrasync.py with the --quick option. This will tell terrasync.py to compare the computed sha1sum of every .dirindex file on your disk with that published in the entry in the parents directories .dirindex. If you have an up-to-date mirror, this will be _very_ quick as only the root .dirindex needs to be downloaded. Subsequent updates with the --quick option will also be very fast as only the updated files will be downloaded and also the .dirindex files of the parent folders up to the root directory.

So far, we have just added files to your disk, what about files that have been removed on the server? The script usually does not touch them unless you add --remove-orphan to the command-line. By doing so, you ask terrasync.py to remove all files (not directories) that exist in a subdirectory but are not listed in the corresponding .dirindex file.

If you don't want to download everything into your current working directory, add --target=/some/folder to your commandline. This describes pretty much everything terrasync.py does. Here are some examples how to use it:

# fetch everything from the master repository server into /home/joe/fg/scenery, keep orphan files, (re-)download every .dirindex file 
terrasync.py --target=/home/joe/fg/scenery 

# just pull in changes, quick and lean for subsequent updates 
terrasync.py --target=/home/joe/fg/scenery --quick 

# run a mirror, only keep those files listed on the server 
terrasync.py --target=/home/joe/fg/scenery --quick --remove-orphan 

# use another server to pull the data, not the sourceforge master 
terrasync.py --target=/home/joe/fg/scenery --url= http://someserver.org/otherscenery 

# restrict the area (must be integers, not decimals)
terrasync.py --target=/home/joe/fg/scenery --top 50 --left 18 --bottom 49 --right 19

Enjoy, feedback welcome.

Enabling TerraSync

FlightGear 3.0 and later

As of the FlightGear 3.0 release, TerraSync has changed significantly. By default, when enabled with --enable-terrasync, TerraSync now downloads to download_dir/TerraSync, where download_dir is FlightGear's download directory.

FlightGear's download directory may be explicitely set using FlightGear's --download-dir command line option, otherwise it defaults to:

• $FG_HOME on non-Windows systems;
• something such as C:\Users\username\Documents\FlightGear on Windows.

Optionally, the TerraSync directory may be explicitely set using FlightGear's --terrasync-dir command line option, or through the Advanced > General dialog of FGRun. TerraSync will then download to that directory, regardless of FlightGear's download directory. The directory is automatically given top priority, unless specified otherwise in $FG_SCENERY. No other configuration is needed.

In summary, if you neither pass --terrasync-dir nor --download-dir to FlightGear, then the default TerraSync directory is:

• $FG_HOME/TerraSync on non-Windows systems;
• something such as C:\Users\username\Documents\FlightGear\TerraSync on Windows.

FlightGear 2.4 up to and including 2.12

As of FlightGear 2.4, TerraSync controls are integrated in the usual FlightGear menu, under Environment > Scenery Download. You can also check the "Download scenery on the fly" in the setup GUI. Note that if you set your aircraft to start in a new area, where you have not yet downloaded any scenery, your aircraft may first appear to be in the water until sufficient scenery has downloaded. You can go to Environment > Scenery Download and choose "Manual Refresh" to apply scenery updates.

POSIX compliant command line shell

Start TerraSync:

% nice terrasync -p 5500 -S -d "$HOME/fgfsScenery"

The -S option tells terrasync to use the SVN protocol to fetch data. If you omit it terrasync will use the rsync program instead (which has to be installed on your system).

Start FlightGear:

% fgfs --atlas=socket,out,1,localhost,5500,udp --fg-scenery="$FG_ROOT/Scenery/:$HOME/fgfsScenery"

The full documentation and source for TerraSync is located in the FlightGear source distribution (in utils/TerraSync/).

FGRun in FlightGear 2.2.0

1. After starting FGRun, make sure you are in the first screen where you can set up directories. One time "Back" from the aircraft selection page. You are now at the "Path" page.
2. You can create a list of scenery directories next to "FG_SCENERY". Select the line that TerraSync will be using and press the "TerraSync directory" button on the right. A small "T" will appear on the selected line, indicating that this one is set up as TerraSync direcotry.
   • The directories are being loaded from top to bottom, so make sure TerraSync is on top (unless you want to "surpass" terrasync and display scenery from another directory). When two directories contain scenery for the same region, FlightGear will take the scenery from the directory higher in the list.
3. Finally, go to the last screen. There you have to activate TerraSync as in the following screenshot. Now TerraSync should work.
   

Note: Expect your firewall to block it the first time you run it; just tell the firewall to allow TerraSync to use the port.

FGRun in FlightGear 1.9.1

1. After starting FGRun, make sure you are in the first screen where you can set up directories. One time "Back" from the aircraft selection page. You are now here:
   
2. Select the destination folder for all files downloaded by terrasync. Usually the folder $FG_ROOT\terrasync already exists and you only have to add it to the list (as in the above example). Insure that it is positioned above your standard scenery folder (here that is FlightGear191\scenery) and all other directories over which the terrasync folder is supposed to have priority. When two directories contain information for the same region, FlightGear will take the information from the directory higher in the list. On Linux make sure the directory does not only to have a T, but also is the topmost folder.
3. For TerraSync to know where to deposit the downloaded files, you have to tell the program which folder is the destination folder. In the above example, it is the 3^rd in the list.
4. Finally, go to the last screen. There you have to activate TerraSync as in the following screenshot. Now TerraSync should work.
   

Note: Expect your firewall to block it the first time you run it; just tell the firewall to allow TerraSync to use the port.

Troubleshooting

Corrupted and unreadable files or directories

If you get an error similar to the following in the command line console (black dialog):

Airports/L ... failed:
Can't move 'C:\FlightGear\terrasync\Airports\L\E\.svn\tmp\entries' to 'C:\FlightGear\terrasync\Airports\L\E\.svn\entries': The file or directory is corrupted and unreadable.

and possibly the following popup appears:

Solution

You can probably fix the error by upgrading to Windows 7 Home Premium Service Pack 1.

Locked airport directories

You get an error indicating locked airport directories while TerraSync is running.

Working copy 'D:\Program Files\FlightGear 2.4.0\terrasync\Airports\K' locked

While those directories often actually are updated, the error is annoying.

Solution

Search the TerraSync directory for files named lock and delete them. They are supposed to be removed automatically when a TerraSync update is completed, but sometimes that fails.

Failure to remove file

You get an error indicating that some files can not be removed.

file remove failed: (./.terrasync_cache)  reason: Permission denied

TerraSync cache files can not be deleted. You will see errors in console like above. There is not much to do other then delete the .terrasync_cache files manually and then run FlightGear again.

Resolution

Caution  Not recommended. Understand the consequences

Forum Post w/ PowerShell command is here

Internals

The SHA1 hashes are what’s in the .dirindex files, both on the server and in the local tree. We do not synchronise time-stamps with the server copies because it complicates (makes much less efficient) mirroring on the server side. I am fairly confident hash clashes are not an issue, if the Git folks don’t think they are. We explicitly do not rely on any custom or particular HTTP headers, to maximise the chance of ‘just working’ with different HTTP providers and CDN options in the future. The structure of the .dirindex files is text based and hopefully trivial to anyone who cares to look at them; they are generated on the server side by some scripts Torsten wrote. The code uses basically identical logic to Git to avoid recalculating SHA hashes for the entire repository; if any of various stat() fields, most especially sie or mod-time, change, we re-compute the hash for that file on disk. Whenever a file’s hash on disk differs from the what the .dirindex file says it should have, we download the file. That’s pretty much a complete description of the synchronisation model. Hence, it’s safe to copy files into the tree using any tool, or modify then - this will cause the stat() data to change, which will cause the code to recompute the SHA hash, which will trigger a re-download to the server-side copy if the SHA does not match. Note there is no provision to have modified (dirty) files inside the tree; they will always be over-written when next checked.

Related content

• Howto: Install scenery
• TerraMaster