Avoiding multiple downloads of FGData on Linux

You will find the current recommended approach to this topic here: Scripted Compilation on Linux Debian/Ubuntu

This is the original topic, which is thought to be specific to Linux Debian/Ubuntu, mainly because it refers to usage of the download_and_compile.sh script, which is specific to that platform. However the concepts presented below should apply regardless of platform and regardless of the commands used to build flightgear.

{{WIP}}

Out of date information has been removed, but the article could use examples of using git-worktree for multiple build directories.

An up to date version of this page would provide alternative means of handling multiple "copies" of fgdata. One of the best ways to do that is a git feature introduced in 2015, Git Worktrees. This git feature allows you to have multiple copies of fgdata with a single master copy attached to sourceforge. Each copy would have it's own separate checkout.

When building different versions of FlightGear, its important to note that each version has its own version of FGData, and FlightGear will check the version to ensure it is compatible.

Since FGData is so large (At least 1.2 GB as of 07/06/15), it makes sense to download the whole FGData only once. Reportedly, downloading FGData multiple times from a single IP address can get your IP address banned from SourceForge, simply because of the high volume of repeated data. (Just because the beer is free, doesn't mean you get to drink the whole keg.)

The download_and_compile.sh script can build two and possibly three versions of Flightgear. It could build quite a number of distinct builds with various features enabled or disabled.

The main ideas

• You'll need one directory for each separate FlightGear version you want to build.
• You will need a FlightGear fgdata directory that matches the version you built
• You can use a single cloned fgdata for everything using git worktrees.
• If you are building more than one version of flightgear, install/flightgear is not a particularly good location for the main fgdata git archive.
• fgdata normally lives as a sub-directory in install/flightgear/. There is no rule that says it has to stay there.
• Once it exists, install/flightgear/fgdata can moved to a safe location and replaced by a symbolic link to that location. The reason for doing this is that you may at some point want to remove your install directory and completely rebuild it. Having fgdata in a "safe" location prevents it from being deleted. You'll just have to recreate the install/flghtgear path and re-create the symbolic link before proceeding with your rebuild.
• The git worktree feature can be used to create multiple additional copies of fgdata with a specific version of fgdata checked out for each worktree. This avoids the necessity of re-downloading fgdata for each variant of flightgear you build. Having fgdata in that "safe" location is a good idea if you are going to use this git feature.

Technical points and command examples (need example for git worktree)

• mv can move a directory
• mkdir -p install/fgfs will create the install directory and subdirectory if either does not yet exist
• ln -s targetname linkname creates linkname as a symbolic link to the target. The target may or may not yet exist and that's normally ok. Not in this case, because of the way we create the directory with git. The symbolic link must be created after downloading. you can't link to something that doesn't exist.
• You can move the FlightGear source directories and download_and_compile.sh to different folders at will. Git will not care. All the Git specific files move right along with the sources. Its the same for SVN. However, the same does not hold true for the files created during a build by cmake. The build and install folders cannot be moved because the files in them have hard coded paths in some cases. They'll need to be deleted and rebuilt. Take note of that "safe" copy of fgdata mentioned above.

Futures

The git workspace command could be applied not just to fgdata, but to any git clone used by the download and compile script.

One can envision a restructured version of the script that makes use of git worktree by default, and can build multiple variants of Flightgear without much fuss or confusion.

Consider this:

Directories

fg/fgdata

fg/flightgear

fg/simgear

fg/osg

fg/variant/next

fg/variant/stable

fg/variant/lts

fg/variant/next_rti_rel_with_debug_info

Each build runs in a variant subdirectory. Each build completes but only 1 git clone is ever run for flightgear, simgear, osg and fgdata. They land at the top level, above the variant directories. Each variant contains a git worktree. To save space, one of the variants could substitute symlinks to the actual git folders, but the rest would have to be worktrees.

Advantage: Don't need a variant any more, just delete it and re-create it with the script when you need it again.