Downloading fgdata using bittorrent: Difference between revisions

Jump to navigation Jump to search

Downloading fgdata using bittorrent (view source)

Revision as of 05:53, 29 April 2016

2,539 bytes removed ,  29 April 2016
no edit summary
(→‎Linux: Switched to {{fgdata clone}}.)
No edit summary
Line 1: Line 1:
A small project to adapt to the current size of the fgdata repository. It is getting too large to clone. Attempts to clone fail more often than not. An alternative is to use a bittorent client such as transmission, qbittorrent, deluge, rasterbar etc. This will be developed first as a set of instructions, then as a script, then as a part of download_and_compile.sh
Separately downloading fgdata from git is only of interest to those who wish to create a local flightgear development or build environment.


What may be needed to support this effort is a process to create new bundles, rather than a continuing set of updates.
There are other means of obtaining flightgear that install only the executables and necessary data.  These use far less disk space.


We need to know more about how the bundles are created, and how and where they are stored.
When fgdata was much larger, attempts to clone fgdata from git on gitorious (where it was hosted at the time) frequently failed because the archive was so large.


This page is related to  
Downloading with bittorrent was proposed as an alternative and a torrent  was created at this url: http://mxchange.org:23456/.  This torrent appears to be non-operational.
* [[FlightGear Git: data developers]]  - The process is part of the steps an Aircraft developer would use as part of that page.


* [[Scripted_Compilation_on_Linux_Debian/Ubuntu]] -  The process replaces the part of the script that uses git-clone to obtain a copy of fgdata's git.
Recently flightgear reduced the size of the fgdata .git archive substantially by removing Aircraft to a separate repository.  That brought fgdata down to 2.1 GB.


Development Plan:
I do not know of a current process for creating torrents of fgdata.  Personally, I would only create one if the developers approve the effort and stable network resources for hosting the bundle or bundles is available.


* read [[FlightGear Git: data developers]], extract and assemble commands to complete the bittorrent and git process to create a current usable .git for fgdata
If the developers of flightgear see it as necessary or advantageous to provide bittorrents for initial downloads of fgdata, a new bundle would need to be created and hosted. The bundle could be periodically updated, possibly once for each release.  Syncing with sourceforge's git would keep the user's local data up to date, but this would require a user to install and use git commands.
* write script to ensure that a bittorrent command line interface and daemon is available
* write script to ensure the bittorrent daemon is available
* write script to ensure the bittorrent daemon provides uploads to other bittorent users in a way that is acceptable to the user of the script.
* Integrate the bittorrent and git commands to use the bundle into download_and_compile.sh
* Integrate installation and configuration of a bittorrent cli client and daemon if one is not already present.  This would be separate from the installation of library pre-requisites and would explicitly inform the user of the need for doing this.


Raw info from [[FlightGear Git: data developers]] 


The current best alternative that minimizes download time is:


:<code>{{fgdata clone|opt=--depth=1}}</code>


This is where things go wrong:
This command gets only the most recent version of the data.


==== Single clone ====
[[Category:Development]]
# Create a directory on your computer where you'll be storing the data.
# Change into that folder.
# Clone the repository with the following command. This will create a <tt>/fgdata</tt> subfolder and put all the contents of the repository in there.
#:<code>{{fgdata clone}}</code>
 
And this is the solution:


==== Bundle ====
[[User:Callahanp|Callahanp]] ([[User talk:Callahanp|talk]]) 01:53, 29 April 2016 (EDT)
For the FlightGear-data there is a [http://www.kernel.org/pub/software/scm/git/docs/git-bundle.html git-bundle] (snapshot) [http://mxchange.org:23456/file?info_hash=%BF%FF%AB%0C%16%BF%8Eg%B8%A0%CFw%01%0A%5D%8F%3F%81%96y torrent] ([http://mxchange.org:23456/ tracker]; [http://flightgear.mxchange.org/pub/fgfs/fgdata.bundle.md5 md5]|[http://flightgear.mxchange.org/pub/fgfs/fgdata.bundle.sha1 sha1]|[http://flightgear.mxchange.org/pub/fgfs/fgdata.bundle.sha512 sha512]) available. This way you can resume interrupted downloads. After unpacking it only a comparatively small amount of data has to be transferred from the git server to synchronize your repository. Also download the fgdata-update-*.bundle updates linked in the [http://mxchange.org:23456/ tracker]. See also the Develop sections in [[FlightGear Git on Windows]].
 
# Create a directory on your computer where you'll be storing the data.
# Change into that folder.
# Do the following steps to extract the bundle and bring the repository up to date:
#:<code>git clone fgdata.bundle fgdata</code>
#:<code>cd fgdata</code>
#:<code>git checkout -b master-tmp</code>
#:<code>for N in 1 2 3 4 5 6 7 8; do git pull ../fgdata-update-00$N.bundle master; done</code>
#:<code>git remote rm origin</code>
#:<code><nowiki>git remote add origin git://mapserver.flightgear.org/fgdata</nowiki></code>
#:<code>git fetch origin</code>
#:<code>git branch --track master origin/master</code>
#:<code>git checkout master</code>
#:<code>git branch -D master-tmp</code>
 
You should be suspicious if based on the printed progress you estimate the data download during the fetch will exceed 1GB (assuming the bundle is not terribly outdated).
 
 
Another solution
 
==== No History ====
Another option especially for people that just need the base files is to not include any history
 
===== Windows =====
Unknown
 
===== Linux =====
Use the ''--depth=1'' option to only get the current objects.
 
As of 20/03/2014 this decreased the number of objects from 226k to 64k
:<code>{{fgdata clone|opt=--depth=1}}</code>
[[Category:Development]]
Callahanp
982

edits

Retrieved from "https://wiki.flightgear.org/Special:MobileDiff/97531"

Navigation menu