455
edits
TerraSync: Difference between revisions
Jump to navigation
Jump to search
Update info about terrasync.py, in particular its new location
(→How does it work: SVN client has been removed from FlightGear) |
(Update info about terrasync.py, in particular its new location) |
||
| Line 12: | Line 12: | ||
== Announcements == | == Announcements == | ||
Also as a result from FSweekend, we are going to test some ideas how to distribute osm2city generated scenery with terrasync. As a first step I have just added a folder called Objects_1 containing the osm2city buildings for e005n46 up to e010n48 [http://flightgear.sourceforge.net/scenery/Objects_1/]. People using the fgfs integrated terrasync should not (yet) notice any difference. When using terrasync.py you should notice download of files in those folders, however they do not (yet) show up in the scenery. We probably throw away some or all files in that directory without prior notice, so please don't rely on them or make backup copies. Torsten <ref>{{cite web | Also as a result from FSweekend, we are going to test some ideas how to distribute osm2city generated scenery with terrasync. As a first step I have just added a folder called Objects_1 containing the osm2city buildings for e005n46 up to e010n48 [http://flightgear.sourceforge.net/scenery/Objects_1/]. People using the fgfs integrated terrasync should not (yet) notice any difference. When using terrasync.py, you should notice download of files in those folders, however they do not (yet) show up in the scenery. We probably throw away some or all files in that directory without prior notice, so please don't rely on them or make backup copies. Torsten <ref>{{cite web | ||
|url = https://sourceforge.net/p/flightgear/mailman/message/35476181/ | |url = https://sourceforge.net/p/flightgear/mailman/message/35476181/ | ||
|title = <nowiki> [Flightgear-devel] ATTN: terrasync changes </nowiki> | |title = <nowiki> [Flightgear-devel] ATTN: terrasync changes </nowiki> | ||
| Line 122: | Line 122: | ||
== terrasync.py == | == terrasync.py == | ||
First of all | First of all, the script is now part of the {{fgmeta-python source | text = fgmeta-python}} repository (this was necessary to share .dirindex processing code with other infrastructure tools, in particular for producing the ''base package''). Installation instructions can be found below the list of files, or more directly in {{fgmeta-python source | path = README.md | text = README.md}}. | ||
The script | The script can provide a complete mirror of the scenery data, also known as TerraSync scenery or just TerraScenery. | ||
{{Caution| | {{Caution|If you fully run this script for the first time, it will download more than 90 GB of data and more than 1,800,000 files in 40,000 folders to your hard disk. It will run for hours, probably days (depending on your Internet connection). Since someone has to pay for the bandwidth, please don't let it download large amounts of data unless you have a good reason to do so (like, you maintain a public scenery mirror); otherwise, the service could be discontinued. That being said, if you just want to see the script in action, you can run it for a few seconds and stop it with {{key press|Ctrl|C}}.}} | ||
If you already have at least a partial set of scenery data from | If you already have at least a partial set of scenery data from the FlightGear built-in TerraSync or even a full TerraScenery, existing and up-to-date files will be reused and won't be downloaded a second time. | ||
=== How | === How it works === | ||
The HTTP | |||
* a) download the file at the root of the scenery-url ( http://flightgear.sourceforge.net/scenery/.dirindex) | {{Note|The following is not completely up-to-date. In particular, current terrasync.py has a <code>check</code> mode in addition to the default <code>sync</code> mode. Run <code>terrasync.py --help</code> for up-to-date usage information.}} | ||
The HTTP TerraScenery server holds all the well-know files required for scenery, organized in the also well-known directory structure (/Models, /Objects, /Terrain and /Airports) with all their subdirectories. The content of the /Models folder for example is here: http://flightgear.sourceforge.net/scenery/Models/ Each folder contains a (hidden) file called .dirindex (e.g. http://flightgear.sourceforge.net/scenery/Models/.dirindex) that describes the content of that folder by listing its files, subdirectories and checksums of the files and their subdirectories' .dirindex files. The terrasync.py script does the following (simplified): | |||
* a) download the file at the root of the scenery-url (http://flightgear.sourceforge.net/scenery/.dirindex) | |||
* b) walks into any subdirectory listed within the .dirindex | * b) walks into any subdirectory listed within the .dirindex | ||
* c) compares the sha1sum of listed files with the local copy and downloads them on mismatch | * c) compares the sha1sum of listed files with the local copy and downloads them on mismatch | ||
* d) for listed subdirectories, calls recursively step b) | * d) for listed subdirectories, calls recursively step b) | ||
Even if all scenery files exist on the server, there are initially still approx. 40,000 .dirindex files missing on your drive and downloading will take hours. To speed up the procedure once you have all .dirindex files locally, try running terrasync.py with the --quick option. This will tell terrasync.py to compare the computed sha1sum of every .dirindex file on your disk with that published in the entry in the parent | Even if all scenery files exist on the server, there are initially still approx. 40,000 .dirindex files missing on your drive and downloading will take hours. To speed up the procedure once you have all .dirindex files locally, try running terrasync.py with the <code>--quick</code> option. This will tell terrasync.py to compare the computed sha1sum of every .dirindex file on your disk with that published in the entry in the parent directory's .dirindex. If you have an up-to-date mirror, this will be ''very'' quick as only the root .dirindex needs to be downloaded. Subsequent updates with the <code>--quick</code> option will also be very fast as only the updated files will be downloaded and also the .dirindex files of the parent folders up to the root directory. | ||
So far, we have just added files to your disk, what about files that have been removed on the server? The script usually does not touch them unless you add --remove-orphan to the command-line. By doing so, you ask terrasync.py to remove all files (not directories) that exist in a subdirectory but are not listed in the corresponding .dirindex file. | So far, we have just added files to your disk, what about files that have been removed on the server? The script usually does not touch them unless you add <code>--remove-orphan</code> to the command-line. By doing so, you ask terrasync.py to remove all files (not directories) that exist in a subdirectory but are not listed in the corresponding .dirindex file. | ||
If you don't want to download everything into your current working directory, add --target=/some/folder to your | If you don't want to download everything into your current working directory, add <code>--target=/some/folder</code> to your command line. This describes pretty much everything terrasync.py does. Here are some examples how to use it: | ||
<syntaxhighlight lang="bash"> | <syntaxhighlight lang="bash"> | ||
| Line 154: | Line 157: | ||
# use another server to pull the data, not the sourceforge master | # use another server to pull the data, not the sourceforge master | ||
terrasync.py --target=/home/joe/fg/scenery --url= http://someserver.org/otherscenery | terrasync.py --target=/home/joe/fg/scenery --url=http://someserver.org/otherscenery | ||
# restrict the area (must be integers, not decimals) | # restrict the area (must be integers, not decimals) | ||