Avoiding multiple downloads of FGData on Linux: Difference between revisions

Jump to navigation Jump to search

Avoiding multiple downloads of FGData on Linux (view source)

Revision as of 02:35, 1 July 2021

2,742 bytes removed ,  1 July 2021
Updated to propose an approach to the need for multiple fgdata directories with different checkouts to support multiple builds of flightgear.
(Add “Out of date” template)
(Updated to propose an approach to the need for multiple fgdata directories with different checkouts to support multiple builds of flightgear.)
Line 1: Line 1:
{{Template:Out of date}}
You will find the current recommended approach to this topic here: [[Scripted Compilation on Linux Debian/Ubuntu]]


{{Warning|This article doesn't seem to be up-to-date. You may have better luck [[Scripted Compilation on Linux Debian/Ubuntu#avoiding-multiple-downloads-of-fgdata|here]].}}
This is the original topic, which is thought to be specific to Linux Debian/Ubuntu, mainly because it refers to usage of the download_and_compile.sh script, which is specific to that platform.  However the concepts presented below should apply regardless of platform and regardless of the commands used to build flightgear.  


When building different versions of FlightGear, its important to note that each version has its own version of FGData, and FlightGear will check the version to ensure it is compatible.
{WIP}


Since [[$FG_ROOT#$FG_DATA|FGData]] is so large (At least 1.2 GB as of 07/06/15), it makes sense to download the whole FGData only once. You can keep versions of it separately from the install tree that download_and_compile.sh creates and create symbolic links to them before using the script. That way when you decide to rebuild from scratch, at least the FGData will remain.
Out of date information has been removed, but the article could use examples of using git-worktree for multiple build directories.  


The <code>download_and_compile.sh</code> script can build either the next/master branch, or the stable version of [[FlightGear]].
An up to date version of this page would provide alternative means of handling multiple "copies" of fgdata. One of the best ways to do that is a git feature introduced in 2015, [https://git-scm.com/docs/git-worktree/2.31.0 Git Worktrees].  This git feature allows you to have multiple copies of fgdata with a single master copy attached to sourceforge.  Each copy would have it's own separate checkout.


The instructions below tell how to do this for existing or new uses of <code>download_and_compile.sh</code>.  The technical details are not dependent on where you put your builds.  You can use the directory scheme I use or your own existing scheme.
When building different versions of FlightGear, its important to note that each version has its own version of FGData, and FlightGear will check the version to ensure it is compatible.


The concept as a whole works great.  I've been using it for months. The instructions are a work in progress and have yet to be fully tested.   
Since [[$FG_ROOT#$FG_DATA|FGData]] is so large (At least 1.2 GB as of 07/06/15), it makes sense to download the whole FGData only once.  Reportedly, downloading FGData multiple times from a single IP address can get your IP address banned from SourceForge, simply because of the high volume of repeated data.  (Just because the beer is free, doesn't mean you get to drink the whole keg.)
 
The <code>download_and_compile.sh</code> script can build two and possibly three versions of Flightgear.  It could build quite a number of distinct builds with various features enabled or disabled.


== The main ideas ==
== The main ideas ==
* You may only need one copy of each version of FGData you are working with
* You'll need one directory for each separate FlightGear version you want to build.
* FGData normally lives as a sub-directory in <code>install/fgfs/</code>
* You will need a FlightGear directory that matches the version you built
* Once it exists, <code>install/fgdata</code> can moved and replaced by a symbolic link to FGData's real directory location
* You can use a single downloaded fgdata for everything using git worktrees.
 
* If you are building more than one version of flightgear, install/flightgear is not a particularly good location for the main git archive.
== Technical points and command examples ==
* fgdata normally lives as a sub-directory in <code>install/flightgear/ There is no rule that says it has to stay there.</code>
* <code>mkdir -p install/fgfs</code> will create the install directory and subdirectory if either does not yet exist
* Once it exists, <code>install/flightgear/fgdata</code> can moved to a safe location and replaced by a symbolic link to that location.  The reason for doing this is that you may at some point want to remove your install directory and completely rebuild it.  Having fgdata in a "safe" location prevents it from being deleted.  You'll just have to recreate the install/flghtgear path and re-create the symbolic link before proceeding with your rebuild.
* <code>mv</code>  can move a directory 
* The git worktree feature can be used to create multiple copies of fgdata with a specific version of fgdata checked out for each worktree.  This avoids the necessity of re-downloading fgdata for each variant of flightgear you build.  Having fgdata in that "safe" location is a good idea if you are going to use this feature.
* <code>cp -R</code> will copy a directory recursively.
* <code>ln -s targetname linkname</code>  creates linkname as a symbolic link to the target.  The target may or may not yet exist and that's normally ok. Not in this case, because of the way we create the directory with git.
* The symbolic link must be created <b>after</b> downloading
* Important: You can move the FlightGear source directories and <code>download_and_compile.sh</code> to different folders at will.  Git will not care.  All the Git specific files move right along with the sources.  Its the same for SVN.  However, the same does not hold true for the files created during a build by cmake.  The build and install folders cannot be moved because the files in them have hard coded paths in some cases.  They'll need to be deleted and rebuilt.
 
== Background info ==
I do my builds of different versions and options in separate sub-directories under <code>~/work/fg</code>.  Sometimes I'll create an extra tree for debugging purposes.  I don't  want to have to copy all of FGData just to do the work on a debug version.  I also  copy the sources for the various components to avoid re downloading those as well.  In download_and_compile.sh, Git takes care of any changes needed to switch to between master and stable as needed.
 
== Here's how I handle FGData... ==
Normally fgdata lands in <code>install/fgfs</code>. 
I've moved my 2.10.0 and 2.11.0 FGData versions and replaced them with symbolic links to the new location as needed.
Its as simple as that.
 
== Detailed directions ==
In short the steps are:
# Create a directory for each version of FGData you are going to use
# Create a directory for each version or variation of FlightGear you want to build
# Do you first build of FlightGear
# Retrofit previous use of <code>download_and_compile.sh</code>
# Prepare to build another version
# Building with options
 
=== Create a directory for each version of FGData you are going to use ===
mkdir -p ~/work/fg/fgdata_2.10.0
mkdir -p ~/work/fg/fgdata_2.11.0
 
Eventually a particular version of FGData will be in each
 
=== Create a directory for each version or variation of FlightGear you want to build ===
<code>download_and_compile.sh</code> will be run from these directories
 
Mine are sub-directories of <code>~/work/fg</code> for example:
 
mkdir -p ~/work/fg/master
mkdir -p ~/work/fg/stable
mkdir -p ~/work/fg/master_debug
  mkdir -p ~/work/fg/master_debug_rti
 
Note that I never build at the <code>~/work/fg level</code>.  <code>download_and_compile.sh</code> is always run from a sub-directory
 
=== Do you first build of FlightGear ===
You can skip this step if you've already done one.
 
If you've never used <code>download_and_compile.sh</code> before you're ready to go.
 
cp download_and_compile.sh ~/work/fg/master
cd ~/work/fg/master
./download_and_compile.sh ALL
 
That will build just about everything and will download FGData into <code>~/work/fg/master/install/fgfs/fgdata</code>
 
=== Retrofit previous use of download_and_compile.sh ===
Download_and_compile.sh placed the FGData directory  in the <code>install/fgfs</code> directory.
 
We're going to move it.  N
From the directory where you run <code>download_and_compile.sh</code>:
 
mv install/fgfs/fgdata ~/work/fg/fgdata_2.11.0
ln -s /home/userid/work/fg/fgdata_2.11.0/fgdata  install/fgfs/fgdata
 
 
If you want to adopt my tree naming scheme, then move the source directories for the various components to the appropriate build directory such as <code>~/work/fg/master</code>.  Note that you cannot move the build and install directories.  They'll be rebuilt in the new location.  The rebuild is needed because we use Cmake.
 
Test to see that everything is hooked up.
 
./download_and_compile.sh DATA
 
=== Prepare to build another version ===
In step 1, we created a spot for two FGData versions.  So far only one is actually there, <code>fgdata_2.11.0/fgdata</code>.  Lets fix that so we don't have to download another copy for 2.10.0


  cd ~/work/fg/fgdata_2.8.0
== Technical points and command examples (need example for git worktree) ==
  cp -r ~/work/fg/fgdata_2.11.0 fgdata
*<code>mv</code> can move a directory
*<code>mkdir -p install/fgfs</code> will create the install directory and subdirectory if either does not yet exist
*<code>ln -s targetname linkname</code>  creates linkname as a symbolic link to the target.  The target may or may not yet exist and that's normally ok. Not in this case, because of the way we create the directory with git.  The symbolic link must be created <b>after</b> downloading. you can't link to something that doesn't exist.
* You can move the FlightGear source directories and <code>download_and_compile.sh</code> to different folders at will.  Git will not care.  All the Git specific files move right along with the sources.  Its the same for SVN.  However, the same does not hold true for the files created during a build by cmake. The build and install folders cannot be moved because the files in them have hard coded paths in some cases. They'll need to be deleted and rebuilt.  Take note of that "safe" copy of fgdata mentioned above.


That's right, we just made a copy of 2.11.0's fgdata and are calling it 2.8.0.
== Futures ==
The git workspace command could be applied not just to fgdata, but to any git clone used by the download and compile script.


While we're at it, lets copy the 2.11.0 source code 
One can envision a restructured version of the script that makes use of git worktree by default, and can build multiple variants of Flightgear without much fuss or confusion.


cd ~/work/fg/stable
Consider this:
cp -R ~/work/fg/master/download_and_compile.sh ./
cp -R ~/work/fg/master/fgfs ./
cp -R ~/work/fg/master/plib ./
cp -R ~/work/fg/master/simgear ./
cp -R ~/work/fg/master/OpenSceneGraph ./
cp -R ~/work/fg/master/fgrun ./
cp -R ~/work/fg/master/fgcom ./
cp -R ~/work/fg/master/fgcomgui ./


Directories


And replace the link to the right version
fg/fgdata


mkdir -p  /installfgfs
fg/flightgear
ln -s ~/work/fg/fgdata/2.8.0 install/fgfs
ls -lah install/fgfs/fgdata  # show the symlink to fgdata
ls  install/fgfs/fgdata  # make sure there's something there.


Now you can build stable
fg/simgear


./download_and_compile.sh -s ALL
fg/osg


=== Building with options ===
fg/variant/next
You can use <code>download_and_compile.sh</code> to build master or stable with various options such as <code>-i</code> for OpenRTI  or <code>-g</code> to build a debug version. 


You can do that in your master or stable that you just built without these options by deleting the install directories and using the options you want to rebuild with. (Remember to redo the link as in step 3 if you delete <code>install/fgfs</code>.  The link will go, but <code>~/work/fgdata_*/fgdata</code> will stay.)
fg/variant/stable


An alternative is to use a separate directory.  All that's needed is that link to fgdata before you start.
fg/variant/lts


Three of the <code>download_and_compile.sh</code> options actually build differently.
fg/variant/next_rti_rel_with_debug_info


* <code>-i</code> builds with OpenRTI
Each build runs in a variant subdirectory.  Each build completes but only 1 git clone is ever run for flightgear, simgear, osg and fgdata. They land at the top level, above the variant directories.  Each variant contains a git worktree. To save space, one of the variants could substitute symlinks to the actual git folders, but the rest would have to be worktrees.
* <code>-e</code> builds with –with-eventinput
* <code>-g</code> builds for debugging


If you create a separate directory with the correct <code>install/fgfs/fgdata</code> symbolic link, you can build either master or stable with any of these options, independently of your main builds in master or stable.


You have choices.  You can use the options in <code>~/work/fg/stable</code> and <code>~/work/fg/master</code>, or you can do things in separate directories.  Whatever suits your purpose.
Advantage: Don't need a variant any more, just delete it and re-create it with the script when you need it again.


[[Category:Core developer documentation]]
[[Category:Core developer documentation]]
Callahanp
936

edits

Retrieved from "https://wiki.flightgear.org/Special:MobileDiff/132452"

Navigation menu