Talk:Scripted Compilation on Linux Debian/Ubuntu: Difference between revisions

Jump to navigation Jump to search

Talk:Scripted Compilation on Linux Debian/Ubuntu (view source)

Revision as of 22:36, 28 September 2013

7,285 bytes removed ,  28 September 2013
no edit summary
No edit summary
Line 1: Line 1:
=== Reorganization of this talk page ===
# Current Status of development on fg-download-and-compile-fgmeta.git
# Overview  -- Discussion of why the script was originally developed and where it might go from here
# Script requirements -- Discussion in general terms of what the script should produce and how a user interacts with it
# Script features -- Specifics of what the script should produce and how
# Different coding approaches to implement the features -- Ways to structure the script so it can be easily understood and maintained
# Proposed features -- New features of the script that have yet to be implemented.
# git usage
# there's a lot of noise on this talk page (re-organize, move to the main page and/or remove it)


=== Current Status of development on the team clone: fg-download-and-compile-fgmeta.git ===
== Current Status of development on the team clone: fg-download-and-compile-fgmeta.git ==


Team clone master branch: 1.9.12 - needs additional testing by more users
Team clone master branch: 1.9.12 - needs additional testing by more users
Line 16: Line 7:
Team clone 1.9.14 branch soon to be deleted and replaced with a "feature branch": feature_directories
Team clone 1.9.14 branch soon to be deleted and replaced with a "feature branch": feature_directories


=== Overview ===
 
=== Script Overview -- Discussion of why the script was originally developed and where it might go from here ===


The download_and_compile.sh script was originally developed as an easy way to get the latest flightgear features in the hands of non-developers and to entice new developers to the project with an easy way to get a workable build environment in place.
The download_and_compile.sh script was originally developed as an easy way to get the latest flightgear features in the hands of non-developers and to entice new developers to the project with an easy way to get a workable build environment in place.
=== Script Requirements ===
=== Script Requirements -- in general terms, what the script should produce ===


Functional requirements:
=== Functional requirements --Specifics of what the script should produce ===
# bash script with options and parameters
# bash script with options and parameters
# install any pre-requisite development packages
# install any pre-requisite development packages
Line 33: Line 25:
# support for building with different versions of underlying packages such as boost, OSG, etc
# support for building with different versions of underlying packages such as boost, OSG, etc


See additional future requirements below:
=== Current Script Features ===
 
=== Script Features ===


The script  has the following major features:
The script  has the following major features:
Line 50: Line 40:
## build and Install of components
## build and Install of components


=== Alternative Coding approaches ===
=== Script implementation notes -- How the script implements features ===
=== Proposed Features ===


Changes in progress, but feel free to jump in. [[User:Callahanp|Callahanp]] ([[User talk:Callahanp|talk]]) 14:24, 28 September 2013 (UTC)
The script currently uses bash associative arrays for determining the version to download and build.
once selected, the version information is stored in bash variables.


=== Older comments and noise to be re-organized into the sections above ===
bash variables are used to
# hold a list of components to build
# specify component container, source, build and install directories.
# determine if
## an update of sources from upstream repositories is desired
## reconfigure is desired
## the build should produce a debug version
##  set flags for debug, build options for use of openrti
#


The team clone has been rebuilt as of 1:15 AM Tuesday, July 30, 2013 (UTC)
== Future Script Versions ==


=== Alternative Coding approaches  to implement the features -- Ways to structure the script so it can be easily understood and maintained===


Recent Maintenance Changes:
The information needed for the work to be done on each component is similar.  We could create a bash associative array for each component with items for each piece of information required:
* git, svn, zip or other file url
* branch and revision information
* update sources or not
* configure or not
* build or not
* install or not
* where to put sources, build and install
* which version of fgdata to use


# added libudev-dev to list of pre-requisite packages,
# fixing a a problem with the -e option.
# Support for 2.12.0
# Corrections to PLIB and OSG svn Urld
# fix the debug option -g
# remove openrti CMakeCache.txt from source dir.
# Changed RELEASE_OR_DEBUG symbols to Release instead of release Debug instead of debug


Recent Enhancements:
# Produce a timestamped log of the build
# Create a rebuild script in the directory the download_and_compile.sh script is run from.  Executing it will rerun Download_and_compile.sh with the same options without having to specify them.
# The title of the window in which the build is run now indicates what variant is being worked on and which component is currently being built. When the build is complete the Window Title says so.
# Optional new directory structure to support multiple variant builds efficiently.


Recent Option additions:
=== Proposed features -- New features of the script that have yet to be implemented ===
# -x Sets the bash -x option
# -v Sets the bash -v option
# -w Sets the make command's verbose option
# -B Selects a Flightgear and Simgear Branch by name.  When used to build Flightgear or Simgear alone, can specify a git commit #
# -V (default=HEAD) can be used to pick a tag from a branch
# -N set the svn client option for building simgear
# -G select an OSG version other than 3.0.1
# -D select the directory usage model.  -D 0 forces the old model -D 1 forces the new model (default is leave the directory model as-i


* No more multiple downloads of fgdata for each separate build. You download only one, and if you need more than one version, it is created locally with copy and git commands. All variant builds use a copy of fgdata from a parent directory
* No more multiple download and builds of plib and osg. You download and build plib-1.8.5 and osg-3.0.1 once. Their sources are kept in a separate directory.
* A single copy of flightgear, simgear and openrti sources that can be used to build whatever versions are needed, one at a time
* Support for additional versions of osg such as 3.1.9
*  Add support for both stable and latest osg.
*  build while downloading
*  download fgdata as a restartable bundle (FGDATA is being broken up so this may be less of an issue from 3.0 forward.
*  download all scenery (or maybe specific regions?)
*  SUSE, Redhat/yum & CENTOS support
*  Distribution builds
*  network setup tests for fgcom, multiplayer and terragear
* torrents
* When updating the sources, its possible to automatically detect if a reconfiguration is needed or not.
* You might want to control which components are rebuilt and which are simply recompiled and re-linked
* You might be in the process of updating a file in one of the components.  How would the build need to change to avoid overwriting your change.
* We currently use the main flightgear git archives for sources.  Its possible to use personal or team clones for one or more components instead?
* Do we need better reporting of the results of the build?
* what can we do with cmake to reduce the bash code used to make decisions about what steps need to be done?
* how much of the scripts logic can we delegate to cmake?


The new directory structure supports:
==  git usage ==
 
=== create a feature branch ===
# No more multiple downloads of fgdata for each separate build. You download only one, and if you need more than one version, it is created locally with copy and git commands. All variant builds use a copy of fgdata from a parent directory
# No more multiple download and builds of plib and osg. You download and build plib-1.8.5 and osg-3.0.1 once. Their sources are kept in a separate directory.
# A single copy of flightgear, simgear and openrti sources that can be used to build whatever versions are needed
# Support for additional versions of osg such as 3.1.9
 
Recommended usage:
 
* Create a top level directory for flightgear builds.
* Create a sub-directory for variant builds
* Create a sub-sub-directory for each variant to be built
* Download_and_compile.sh goes in the top level directory
* cd to a variant directory and run download_and_compile -D 1 and other options to build the variant you want.
 
For example the following will give you two debug versions of flightgear built with the new event model and openrti.  One will be OSG 3.0.1, the other will be OSG 3.1.9:
 
<pre>
mkdir -p fg/variants/master-ei-OSG-3.1.9
mkdir -p fg/variants/master-ei-OSG-3.0.1
cd fg
wget https://www.gitorious.org/+fg-download-and-compile/fg/fg-download-and-compile-fgmeta/blobs/next/download_and_compile.sh
cd variants/master-ei-OSG-3.1.9
../../download_and_compile.sh -p n -d y -g -xvw -D 1 -j 3 -G 3.1.9 -ei ALL
./run_fgrun.sh
 
cd ../../variants/master-ei-OSG-3.0.1
../../download_and_compile.sh -p n -d y -g -xvw -D 1 -j 3 -ei ALL
./run_fgrun.sh
 
</pre>
[[User:Callahanp|Callahanp]] 03:16, 19 July 2013 (UTC)
=== git ===
 
Setup for git local git operations
git clone git://gitorious.org/+fg-download-and-compile/fg/fg-download-and-compile-fgmeta.git
git remote set-url --push origin git@gitorious.org:+fg-download-and-compile/fg/fg-download-and-compile-fgmeta.git
 
Here's how the script_prototype branch was created.  This is done already so you don't have to do it again.  It's included here for instructional purposes.
 
  git branch -f script_prototype # creates a new branch
  git branch -f script_prototype # creates a new branch
  git checkout -f script_prototype
  git checkout -f script_prototype
Line 136: Line 102:
  git push --set-upstream origin script_prototype
  git push --set-upstream origin script_prototype


Making a change on master
=== work on a branch ===
 
git branch -f master
gedit download_and_compile.sh
# make changes
git commit -a
git push
 


Making a change on next
Making a change on branch: feature_logging


  git branch -f next
  git branch -f feature_logging
  gedit download_and_compile.sh
  gedit download_and_compile.sh
  # make changes
  # make feature_logging
  git commit -a
  git commit -a
  git push
  git push
=== merge changes into the main branch ===
=== remove a branch no longer needed ===


=== Development ===
== More Details ==  
 
=== makeffile and cmake idea (needs discussion) ===
The main fgmeta repository script is currently at version 1.9.4 and it DOES NOT WORK without changes.  We need to test what's on Next in a variety of scenaria.
 
Development will be done only on the next branch on https://www.gitorious.org/+fg-download-and-compile/fg/fg-download-and-compile-fgmeta
 
<pre>
mkdir ~/fgfs
cd ~/fgfs
wget http://www.gitorious.org/+fg-download-and-compile/fg/fg-download-and-compile-fgmeta/blobs/next/download_and_compile.sh
chmod 755 download_and_compile.sh
</pre>
:
:For those who can write bash scripts and want to help with development and testing, register as a  user on gitorious.org.  Then log in and send a message on gitorious to users fbrisa or pac1.  You can also contact me by registering on this wiki and sending a message throught the wiki.  The privileged few have my actual email address. 
:
 
At this point only the next branch of the repository is being used.  the version of the script there contains a number of new features, not all of which are complete and tested.
 
The biggest change is the addition of a directory management option.  This option allows efficient use of your disk space, your network connection, and gitorious.org bandwidth usage is kept to a minimum.
 
:-D 0 will cause the script to build flightgear in the same way as it always has.  Each directory you build in gets a complete set of sources, build and install objects, plus a copy of fgdata.
 
: -D 1 will cause the script to convert to using a single set of sources for different builds and to download fgdata only once. 
 
to use -D 1,  simply download the script into a directory the usual way, but before building, create a sub-directory to contain each variety of build.  Even if you are doing just one type of build, there is an advantage to using -D 1.  For example, if you decide to completely rebuild, you can just remove the sub-directory and recreate it.  The sources and fgdata are in the parent directory and remain in place
 
Once the -D 1 option is used, it will remain in effect as long as the fgdata and fgsources directories are present.
 
<pre>
mkdir ~/fgfs
cd ~/fgfs
wget http://www.gitorious.org/+fg-download-and-compile/fg/fg-download-and-compile-fgmeta/blobs/next/download_and_compile.sh
chmod 755 download_and_compile.sh
mkdir -p next
cd next
../download_and_compile.sh -D 1 all
</pre>
 
When the script completes, you will have
 
~/fgfs/download_and_compile.sh
~/fgfs/fgdata_2.11.0/fgdata
~/fgfs/fgsrc
~/fgfs/othersrc
~/fgfs/next/build etc
~/fgfs/next/install etc
 
Additional Changes:
 
# plib-1.8.5 and OpenSceneGraph-3.0.1 are not rebuilt after a successful build.
# you can build next, master, 2.10.0, 2.8.0 etc
# additional script options have been added.
# the window title shows what's being built and what stage things are at
# log file is now named differently and includes a timestamp
 
 
To start working on the script, clone the team clone repository on gitorious.
 
I'll be posting other git instructions as I uncover them.
 
git clone git://gitorious.org/+fg-download-and-compile/fg/fg-download-and-compile-fgmeta.git
git fetch origin
git checkout -f 1.9.4 # for example
 
On the team repository, there are branches for versions 1.9.4 Through 1.9.12 and master.  ignore them.
use next.
 
Additional requested features:
 
#  Add support for both stable and latest osg.
#  build while downloading
#  download fgdata as a restartable bundle
#  download all scenery (or maybe specific regions?)
#  SUSE, Redhat/yum & CENTOS support
#  network setup tests for fgcom, multiplayer and terragear
#  torrents
 
== makeffile and cmake idea (needs discussion) ==


{{cquote|<nowiki>I also did some work (and pushed it) to support CMake superbuilds using fgmeta. It works on Mac and Windows, and probably on Linux with some small tweaks, so this might be a way to simplify future versions of the download_and_compile script, since most of the steps (download & build OSG, download and build PLIB) are handled by the super-build.</nowiki><ref>{{cite web |url=http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg40142.html|title=<nowiki>Re: [Flightgear-devel] new team clone on gitorious.org specifically for download_and_compile.</nowiki>|author=<nowiki>James Turner</nowiki>|date=<nowiki>Sat, 01 Jun 2013 02:16:43 -0700</nowiki>}}</ref>|<nowiki>James Turner</nowiki>}}
{{cquote|<nowiki>I also did some work (and pushed it) to support CMake superbuilds using fgmeta. It works on Mac and Windows, and probably on Linux with some small tweaks, so this might be a way to simplify future versions of the download_and_compile script, since most of the steps (download & build OSG, download and build PLIB) are handled by the super-build.</nowiki><ref>{{cite web |url=http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg40142.html|title=<nowiki>Re: [Flightgear-devel] new team clone on gitorious.org specifically for download_and_compile.</nowiki>|author=<nowiki>James Turner</nowiki>|date=<nowiki>Sat, 01 Jun 2013 02:16:43 -0700</nowiki>}}</ref>|<nowiki>James Turner</nowiki>}}
Line 276: Line 161:




=== General Thoughts on download_and_compile.sh  ===
A build system has set-up steps that can be met as either pre-requisites (steps before the build process) or as requirements of the build system itself.  The approach taken in download_and_compile.sh is to include the set-up steps as part of the process.  This greatly simplifies the user interface in the easiest build cases: stable (one option needed) or next (no options needed).
So what are the set-up steps in addition to building:
* install pre-requisite packages (default option -p y)
* get the sources (default option -d y)
The build for each component steps are:
* configure (default option -r y and not UPDATE
* make  & make install  (default option -c y)
note that the UPDATE is not an option, but a parameter
The description above is how the steps are implemented as of script version 1.9.12
Lets rethink some of this in slightly more detail:
the script needs to deal with the following steps in order:
* installing or updating pre-requisite development packages
then for each component:
* getting copies of sources or updating copies from upstream
* ensuring flightgear package sources are on the desired branch or version for the build in progress
* configuring the packages for building
* compiling
* installing
* reporting results
What could change:
* Its possible to separate the logic that decides what steps to take from the actual execution.  The logic decides what commands to execute and sets up the command list, then a simple process just executes the commands.  This could make a log easier to read.
* When getting the sources, its possible to automatically detect if a reconfiguration is needed or not.
* You might want to control which components are rebuilt and which are simply recompiled and re-linked
* You might be in the process of updating a file in one of the components.  How would the build need to change to avoid overwriting your change.
* We currently use the main flightgear git archives for sources.  Its possible to use personal or team clones for one or more components instead?
* Do we need better reporting of the results of the build?
* what can we do with cmake to reduce the bash code used to make decisions about what steps need to be done?
* how much of the scripts logic can we delegate to cmake?
* should we add a flightgear test run to the script?
* should we add a flightgear test run to the script?


Callahanp
982

edits

Retrieved from "https://wiki.flightgear.org/Special:MobileDiff/62995"

Navigation menu