Downloading fgdata using bittorrent

A small project to adapt to the current size of the fgdata repository. It is getting too large to clone. Attempts to clone fail more often than not. An alternative is to use a bittorent client such as transmission, qbittorrent, deluge, rasterbar etc. This will be developed first as a set of instructions, then as a script, then as a part of download_and_compile.sh

What may be needed to support this effort is a process to create new bundles, rather than a continuing set of updates.

We need to know more about how the bundles are created, and how and where they are stored.

This page is related to

• FlightGear Git: data developers - The process is part of the steps an Aircraft developer would use as part of that page.

• Scripted_Compilation_on_Linux_Debian/Ubuntu - The process replaces the part of the script that uses git-clone to obtain a copy of fgdata's git.

Development Plan:

• read FlightGear Git: data developers, extract and assemble commands to complete the bittorrent and git process to create a current usable .git for fgdata
• write script to ensure that a bittorrent command line interface and daemon is available
• write script to ensure the bittorrent daemon is available
• write script to ensure the bittorrent daemon provides uploads to other bittorent users in a way that is acceptable to the user of the script.
• Integrate the bittorrent and git commands to use the bundle into download_and_compile.sh
• Integrate installation and configuration of a bittorrent cli client and daemon if one is not already present. This would be separate from the installation of library pre-requisites and would explicitly inform the user of the need for doing this.

Raw info from FlightGear Git: data developers

This is where things go wrong:

Single clone

1. Create a directory on your computer where you'll be storing the data.
2. Change into that folder.
3. Clone the repository with the following command. This will create a /fgdata subfolder and put all the contents of the repository in there.

   git clone git://gitorious.org/fg/fgdata.git

And this is the solution:

Bundle

For the FlightGear-data there is a git-bundle (snapshot) torrent (tracker; md5|sha1|sha512) available. This way you can resume interrupted downloads. After unpacking it only a comparatively small amount of data has to be transferred from the git server to synchronize your repository. Also download the fgdata-update-*.bundle updates linked in the tracker. See also the Develop sections in FlightGear Git on Windows.

1. Create a directory on your computer where you'll be storing the data.
2. Change into that folder.
3. Do the following steps to extract the bundle and bring the repository up to date:

   git clone fgdata.bundle fgdata

   cd fgdata

   git checkout -b master-tmp

   for N in 1 2 3 4 5 6 7 8; do git pull ../fgdata-update-00$N.bundle master; done

   git remote rm origin

   git remote add origin git://mapserver.flightgear.org/fgdata

   git fetch origin

   git branch --track master origin/master

   git checkout master

   git branch -D master-tmp

You should be suspicious if based on the printed progress you estimate the data download during the fetch will exceed 1GB (assuming the bundle is not terribly outdated).

Another solution

No History

Another option especially for people that just need a base is to not include any history

Windows

Unknown

Linux

Use the --depth=1 option to only get the current objects As of 20/03/2014 this decreased the number of objects from 226k to 63k

git clone git@gitorious.org:fg/fgdata.git --depth==1