TerraSync: Difference between revisions

From FlightGear wiki
Jump to navigation Jump to search
Line 18: Line 18:
   |added  = May 2nd, 2016
   |added  = May 2nd, 2016
   |script_version = 0.28
   |script_version = 0.28
  }}
</ref>
The technique of the http repository is much simpler than that of the svn repository and it will be pretty simple to create a local mirror or to download chunks of data. With a bit of to-be-developed scripting (python e.g.) there is no need to download existing data again. I don't know how terramaster works but I am sure it is easy to adapt to the new http access method. <ref>{{cite web
  |url    = https://sourceforge.net/p/flightgear/mailman/message/35062543/
  |title  = <nowiki>Re: [Flightgear-devel] The future of terrasync</nowiki>
  |author = <nowiki>Torsten Dreyer</nowiki>
  |date  = May 4th, 2016
  |added  = May 4th, 2016
  |script_version = 0.31
  }}
</ref>
The goal is to completely remove SVN and only use the HTTP repository. I'd like to reach that goal better sooner than later as we currently have two single-points of failure for the SVN-service. The first is the http://scenery.flightgear.org/svn-service web service (runs on a private web server) and the second is the one-and-only svn-server available for public access, also hosted on a private server. If either on of those fails, the SVN-terrasync is dead. I have learned the hard way that not having a fallback is a bad idea.<ref>{{cite web
  |url    = https://sourceforge.net/p/flightgear/mailman/message/35062543/
  |title  = <nowiki>Re: [Flightgear-devel] The future of terrasync</nowiki>
  |author = <nowiki>Torsten Dreyer</nowiki>
  |date  = May 4th, 2016
  |added  = May 4th, 2016
  |script_version = 0.31
   }}
   }}
</ref>
</ref>

Revision as of 12:40, 4 May 2016

Not to be confused with TerraGear, a toolset to generate scenery.

To see the terrain below your aircraft, you have to install the respective scenery. This can happen by downloading certain bits of scenery before flying as described in the article installing scenery.

Alternatively, if you have a steady and reasonably fast internet connection, you can use TerraSync. It is a utility that automatically downloads the newest version of the needed FlightGear scenery while the simulator is running. TerraSync runs in the background (optionally as a separate process), monitors your position, and downloads (or updates) the latest scenery from the master scenery server "just in time".

The master repository for TerraSync, i.e. the online resource from which TerraSync downloads its files, is synchronized with the FlightGear Scenery Database once a day. So when using TerraSync, you will always have

  1. the latest .stg-files, which tell FlightGear where to place an object
  2. the latest static models for objects. (Static models define unique objects that exist in one place only, such as famous buildings or landmarks.)
  3. the latest shared models for objects. (Generic models used more than once in different places, each can represent many different objects, like generic houses or ships)

News

as some of you have probably noticed, some coding is currently happening in the terrasync corner, mostly by James and some small parts by myself. The main intention is to get rid of the SVN protocol to distribute scenery but use easier to implement protocols. The idea is to use plain HTTP as the access method for files required by terrasync. This should make it much easier for hosts to set up a mirror, probably enables us to use a CDN for load balancing and also makes the client code slim. We already have three mirrors of the complete scenery online, one at sourceforge and two at privately owned servers. The sourceforge mirror receives daily updates from the main scenery export from http://scenery.flightgear.org/ and all other mirrors pull their data from the sourceforge server. We decided to drop the previous method of telling the client about it's nearest mirror by querying a web service ( http://scenery.flightgear.org/svn-server) in favour of using a more stable service provider of the internet: the domain name service (DNS) system. The fgfs client will query a fancy resource record (RR) type of NAPTR (network authority pointer, https://en.wikipedia.org/wiki/NAPTR_record) for the dns name terrasync.flightgear.org and analye the entries to find the best terrasync mirror. Those entries are already configured and can be examined here: https://toolbox.googleapps.com/apps/dig/#ANY/terrasync.flightgear.org This technique should enable us to provide different sets of scenery, eventually even for only parts of the world in a much easier fashion that it would have been the case with SVN. It also spares many gigabytes on the scenery mirror as they don't have to keep all the 50,000+ revisions we already carry around. I'll start adding the required code over the next couple of days to have it running hopefully for the 2016.2 release. I'll post a message to this list when I flip the final switch and enable the new terrasync. In the mean time, you might see some build failures on Jenkins or other hickups of the system. I promise to fix those as quick as possible. BTW: If you use terrafs, you already receive your data from the new http mirrors, although not use the DNS service resolver.[1]

The technique of the http repository is much simpler than that of the svn repository and it will be pretty simple to create a local mirror or to download chunks of data. With a bit of to-be-developed scripting (python e.g.) there is no need to download existing data again. I don't know how terramaster works but I am sure it is easy to adapt to the new http access method. [2]

The goal is to completely remove SVN and only use the HTTP repository. I'd like to reach that goal better sooner than later as we currently have two single-points of failure for the SVN-service. The first is the http://scenery.flightgear.org/svn-service web service (runs on a private web server) and the second is the one-and-only svn-server available for public access, also hosted on a private server. If either on of those fails, the SVN-terrasync is dead. I have learned the hard way that not having a fallback is a bad idea.[3]


  1. Torsten Dreyer (May 2nd, 2016). [Flightgear-devel] The future of terrasync.
  2. Torsten Dreyer (May 4th, 2016). Re: [Flightgear-devel] The future of terrasync.
  3. Torsten Dreyer (May 4th, 2016). Re: [Flightgear-devel] The future of terrasync.

Enabling TerraSync

FlightGear 3.0 and later

As of the FlightGear 3.0 release, TerraSync has changed significantly. By default, when enabled with --enable-terrasync[1], TerraSync now downloads to download_dir/TerraSync, where download_dir is FlightGear's download directory.

FlightGear's download directory may be explicitely set using FlightGear's --download-dir command line option, otherwise it defaults to:

  • $FG_HOME on non-Windows systems;
  • something such as C:\Users\username\Documents\FlightGear on Windows.

Optionally, the TerraSync directory may be explicitely set using FlightGear's --terrasync-dir command line option, or through the Advanced > General dialog of FGRun. TerraSync will then download to that directory, regardless of FlightGear's download directory. The directory is automatically given top priority, unless specified otherwise in $FG_SCENERY. No other configuration is needed.

In summary, if you neither pass --terrasync-dir nor --download-dir to FlightGear, then the default TerraSync directory[2] is:

  • $FG_HOME/TerraSync on non-Windows systems;
  • something such as C:\Users\username\Documents\FlightGear\TerraSync on Windows.

FlightGear 2.4 up to and including 2.12

As of FlightGear 2.4, TerraSync controls are integrated in the usual FlightGear menu, under Environment > Scenery Download. You can also check the "Download scenery on the fly" in the setup GUI. Note that if you set your aircraft to start in a new area, where you have not yet downloaded any scenery, your aircraft may first appear to be in the water until sufficient scenery has downloaded. You can go to Environment > Scenery Download and choose "Manual Refresh" to apply scenery updates.

POSIX compliant command line shell

Start TerraSync:

% nice terrasync -p 5500 -S -d "$HOME/fgfsScenery"

The -S option tells terrasync to use the SVN protocol to fetch data. If you omit it terrasync will use the rsync program instead (which has to be installed on your system).

Start FlightGear:

% fgfs --atlas=socket,out,1,localhost,5500,udp --fg-scenery="$FG_ROOT/Scenery/:$HOME/fgfsScenery"

The full documentation and source for TerraSync is located in the FlightGear source distribution (in utils/TerraSync/).

FGRun in FlightGear 2.2.0

  1. After starting FGRun, make sure you are in the first screen where you can set up directories. One time "Back" from the aircraft selection page. You are now at the "Path" page.
  2. You can create a list of scenery directories next to "FG_SCENERY". Select the line that TerraSync will be using and press the "TerraSync directory" button on the right. A small "T" will appear on the selected line, indicating that this one is set up as TerraSync direcotry.
    • The directories are being loaded from top to bottom, so make sure TerraSync is on top (unless you want to "surpass" terrasync and display scenery from another directory). When two directories contain scenery for the same region, FlightGear will take the scenery from the directory higher in the list.
  3. Finally, go to the last screen. There you have to activate TerraSync as in the following screenshot. Now TerraSync should work.
    TerraSync 2.png

Note: Expect your firewall to block it the first time you run it; just tell the firewall to allow TerraSync to use the port.

FGRun in FlightGear 1.9.1

  1. After starting FGRun, make sure you are in the first screen where you can set up directories. One time "Back" from the aircraft selection page. You are now here:
    TerraSync 1.png
  2. Select the destination folder for all files downloaded by terrasync. Usually the folder $FG_ROOT\terrasync already exists and you only have to add it to the list (as in the above example). Insure that it is positioned above your standard scenery folder (here that is FlightGear191\scenery) and all other directories over which the terrasync folder is supposed to have priority. When two directories contain information for the same region, FlightGear will take the information from the directory higher in the list. On Linux make sure the directory does not only to have a T, but also is the topmost folder.
  3. For TerraSync to know where to deposit the downloaded files, you have to tell the program which folder is the destination folder. In the above example, it is the 3rd in the list.
  4. Finally, go to the last screen. There you have to activate TerraSync as in the following screenshot. Now TerraSync should work.
    TerraSync 2.png

Note: Expect your firewall to block it the first time you run it; just tell the firewall to allow TerraSync to use the port.

Troubleshooting

Corrupted and unreadable files or directories

If you get an error similar to the following in the command line console (black dialog):

Airports/L ... failed:
Can't move 'C:\FlightGear\terrasync\Airports\L\E\.svn\tmp\entries' to 'C:\FlightGear\terrasync\Airports\L\E\.svn\entries': The file or directory is corrupted and unreadable.

and possibly the following popup appears:

TerraSync Taskbar Error.png

Solution

You can probably fix the error by upgrading to Windows 7 Home Premium Service Pack 1.

Locked airport directories

You get an error indicating locked airport directories while TerraSync is running.

Working copy 'D:\Program Files\FlightGear 2.4.0\terrasync\Airports\K' locked

While those directories often actually are updated, the error is annoying.

Solution

Search the TerraSync directory for files named lock and delete them. They are supposed to be removed automatically when a TerraSync update is completed, but sometimes that fails.

Failure to remove file

You get an error indicating that some files can not be removed.

file remove failed: (./.terrasync_cache)  reason: Permission denied

TerraSync cache files can not be deleted. You will see errors in console like above. There is not much to do other then delete the .terrasync_cache files manually and then run FlightGear again.

Resolution

Caution  Not recommended. Understand the consequences

Forum Post w/ PowerShell command is here

Development

Cquote1.png The intention is to migrate to a pure HTTP solution for TerraSync (instead of SVN) after 2016.1 is released, since this will give us many more mirroring and hosting options. The cache files are tiny and reflect the current revision of each file in the local copy; they’re analogous to the data a real SVN client stores in its ‘.svn’ directory in the root of the checkout. I expect the pure HTTP solution will need some very similar files to record the HTTP cache stamps of files it downloads - either one file at the root of the tree, or per-directory. (There’s advantages and disadvantages to both).
— James Turner (Jan 14th, 2016). Re: [Flightgear-devel] Launcher changes for 3.8.
(powered by Instant-Cquotes)
Cquote2.png

Related content

Footnotes

  1. In FGRun, this can be done via the appropriate checkbox on the last page. In FFGo, just uncomment the --enable-terrasync option from the default configuration in the Options Window (the “big window” on the left, below the aircraft list).
  2. As derived from an inspection of the FlightGear 2016.2.0 source code on March 21, 2016.