MediaWiki has been updated to version 1.35.1. Please report any problems here.

Downloading fgdata using bittorrent

From FlightGear wiki
Revision as of 13:21, 20 March 2014 by Chrisb (talk | contribs) (added --depth option)
Jump to navigation Jump to search

A small project to adapt to the current size of the fgdata repository. It is getting too large to clone. Attempts to clone fail more often than not. An alternative is to use a bittorent client such as transmission, qbittorrent, deluge, rasterbar etc. This will be developed first as a set of instructions, then as a script, then as a part of download_and_compile.sh

What may be needed to support this effort is a process to create new bundles, rather than a continuing set of updates.

We need to know more about how the bundles are created, and how and where they are stored.

This page is related to

Development Plan:

  • read FlightGear Git: data developers, extract and assemble commands to complete the bittorrent and git process to create a current usable .git for fgdata
  • write script to ensure that a bittorrent command line interface and daemon is available
  • write script to ensure the bittorrent daemon is available
  • write script to ensure the bittorrent daemon provides uploads to other bittorent users in a way that is acceptable to the user of the script.
  • Integrate the bittorrent and git commands to use the bundle into download_and_compile.sh
  • Integrate installation and configuration of a bittorrent cli client and daemon if one is not already present. This would be separate from the installation of library pre-requisites and would explicitly inform the user of the need for doing this.

Raw info from FlightGear Git: data developers


This is where things go wrong:

Single clone

  1. Create a directory on your computer where you'll be storing the data.
  2. Change into that folder.
  3. Clone the repository with the following command. This will create a /fgdata subfolder and put all the contents of the repository in there.
    git clone git://gitorious.org/fg/fgdata.git

And this is the solution:

Bundle

For the FlightGear-data there is a git-bundle (snapshot) torrent (tracker; md5|sha1|sha512) available. This way you can resume interrupted downloads. After unpacking it only a comparatively small amount of data has to be transferred from the git server to synchronize your repository. Also download the fgdata-update-*.bundle updates linked in the tracker. See also the Develop sections in FlightGear Git on Windows.

  1. Create a directory on your computer where you'll be storing the data.
  2. Change into that folder.
  3. Do the following steps to extract the bundle and bring the repository up to date:
    git clone fgdata.bundle fgdata
    cd fgdata
    git checkout -b master-tmp
    for N in 1 2 3 4 5 6 7 8; do git pull ../fgdata-update-00$N.bundle master; done
    git remote rm origin
    git remote add origin git://mapserver.flightgear.org/fgdata
    git fetch origin
    git branch --track master origin/master
    git checkout master
    git branch -D master-tmp

You should be suspicious if based on the printed progress you estimate the data download during the fetch will exceed 1GB (assuming the bundle is not terribly outdated).

Another solution

No History

Another option especially for people that just need a base is to not include any history

Windows

Unknown

Linux

Use the --depth=1 option to only get the current objects As of 20/03/2014 this decreased the number of objects from 226k to 63k

git clone git@gitorious.org:fg/fgdata.git --depth==1