FlightGear build server: Difference between revisions

From FlightGear wiki
Jump to navigation Jump to search
(added SF release candidate URL)
 
(48 intermediate revisions by 15 users not shown)
Line 1: Line 1:
= Intro =
{{Release}}
The '''FlightGear build server''', sometimes called the '''nightly build server''' or just '''Jenkins''', is a system based on [http://www.jenkins-ci.org Jenkins] which helps core developers to check that their source code compiles correctly for all OS. It is also responsible for building release packages for Linux, Windows and Mac.


A *prototype* of a [http://hudson-ci.org/ Hudson] based [http://en.wikipedia.org/wiki/Continuous_integration build server] for building FG (including OSG and SimGear) can be found at http://zakalawe.ath.cx:8080/
{{Note|
The output files on Jenkins download.flightgear.org and sf.net should always be the same. Jenkins pushes them to download.flightgear.org from where they get distributed to sf.net (with some renaming-magic involved). Direct dowloads from Jenkins should be avoided as much as possible. <ref>{{cite web
  |url    =  https://sourceforge.net/p/flightgear/mailman/message/35539513/  
  |title  =  <nowiki> Re: [Flightgear-devel] [from forum] Torrenting FlightGear </nowiki>
  |author =  <nowiki> Torsten Dreyer </nowiki>
  |date  =  Dec 8th, 2016
  |added  =  Dec 8th, 2016
  |script_version = 0.36
  }}</ref>


This is currently running on a core developer's home box:
}}
"I've found some spare hardware to run the build server (Hudson) on, and it seems bearable for me,
but I suspect less pleasant for people on the other end of my cable connection."[http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg28260.html]


So, the server will need a proper home if it moves beyond the prototype stage.
Some core developers are working with Linux and have no access to a Windows installation in order to test if what they have implemented for Linux will works for Windows or Mac. The build server will automatically start a new compilation after each commit is pushed to the source code repository. We have the ability to compile the source code for Linux, Windows and Mac.


For people who don't know, a build server talks to some slaves, and grabs/builds/tests/packages code. The current server is talking to one slave,
A "nightly build" is one that is automatically generated each night (hence the name) from the most recent code. Since it has not received as much testing as normal FlightGear versions, you might encounter some bugs; on the contrary, you get access to the latest-and-greatest features before they are officially released.
which is an Ubuntu VM (Virtual Machine) which is building  the 'next' branch on Gitorious.
Any slave could be a VM, of course - they use CPU resources while building, but unlike other projects, our commit rate isn't that high - the slaves will be idle most of the time (A Mac slave is also possible, but requires some more work).


'''Note:''' If anyone wishes to volunteer a proper server (with a reasonably symmetric connection) to run Hudson, please get in touch using the [http://www.flightgear.org/mail.html mailing list] or the [http://flightgear.org/forums/ FlightGear forums] - any Unix will do, for Ubuntu/Debian there's an easy apt.get source available. All the setup can be done remotely, given SSH access. The disk, memory, CPU and bandwidth requirements  are pretty moderate, due to our low commit rate.
<ref>{{cite web
  |url    = https://forum.flightgear.org/viewtopic.php?p=283958#p283958
  |title  = <nowiki>Re: Tutorial: How to get Aircraft Center to work</nowiki>
  |author = <nowiki>elgaton</nowiki>
  |date  = May 1st, 2016
  |added = May 7th, 2016
  |script_version = 0.34
  }}
</ref>


It does chew a bit of disk-space, since the master stores the artifacts for the last N builds, where N is configurable. The artifacts are a hundred megabytes or so, since it's all the header files, libs and binaries, though compressed of course.
== News (06/2015) ==
{{Main article|Building FlightGear - Cross Compiling}}
{{FGCquote
  |mingw has came a long way, and i think the MXE openscenegraph package (currently at 3.2.1 on master!!), is beautifully maintained, now it builds almost all core libraries dynamically with some argument passing, even as a static target(MXE_TARGETS{{=}}'i686-w64-mingw32.static'), but it's those plugins again, with their linking errors! i think these are because i'm using the i686-w64-mingw32.static-g++ compiler as opposed to the shared one...<br/>
<br/>
so as soon as i can get the shared build environment running and solve all of it's dependancies for OSG, i think we can have a cross compiller in our hands!
  |{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=242914#p242914
    |title=<nowiki>Re: [SOLVED] Install osgEarth feature on Win7 64b with FG gi</nowiki>
    |author=<nowiki>hamzaalloush</nowiki>
    |date=<nowiki>Thu May 14</nowiki>
  }}
}}


= Hosting Options =
== SourceForge release candidate ==
If you know of any others, please do feel free to add new hosting options here. Some of these are not necessarily useful for directly hosting Hudson, but instead for building FlightGear on different platforms using SSH. This applies in particular to the various build farms.


* http://gcc.gnu.org/wiki/CompileFarm#How_to_Get_Involved.3F
As of 2018.1 release candidates are available from sourceforce https://sourceforge.net/projects/flightgear/files/release-candidate/ <https://sourceforge.net/projects/flightgear/files/release-candidate/>;<br/>
* http://en.opensuse.org/Build_Service
* https://edge.launchpad.net/builders/
* http://hub.opensolaris.org/bin/view/Community+Group+testing/testfarm
* http://www.metamodul.com/10.html
* http://www.gnu.org/software/hurd/public_hurd_boxen.html
* http://www-03.ibm.com/systems/z/os/linux/support/lcds/


= Status 07/2010 =
== Where to find it ==
The FlightGear build server is available at: {{build link}}. There is no artifact available because we don't want copy +1GB to Jenkins master each days. This job is executed once per day.


The Mac build is pretty close to producing a nightly, though - we need to fix a genuine (and long-standing) configuration issue on Mac.


The good news - thanks to some great documentation from Fred, the Windows build  is stable, using MSVC no less. You can grab a fgfs.exe (and .pdb) from any  successful build, and it should 'just work', if you drop it into a current 2.0 install. Future work on some improved packaging of these builds will follow.
You can get the artifact at {project infrastructure|buildserver}}


The better news - all three platforms are now green (i.e, building correctly), and should update reliably (which will be monitored). At present, the server admin gets emailed when a build breaks, and when it goes green again.  
{{FGCquote
These build from Gitorious next, and there will probably be experiments to make it complain to IRC, or even to the mailing list, when the build breaks.
  |We’ve improved the tool around daily build creation. Still some progress to be done but daily builds for Mac and Windows are uploaded to:<br/>
<br/>
https://sourceforge.net/projects/flightgear/files/unstable/
<br/>
(And hence, all the SF mirrors)<br/>
<br/>
And also directly available from:<br/>
<br/>
http://download.flightgear.org/builds/nightly/ <http://download.flightgear.org/builds/nightly/>;<br/>
<br/>
The ‘latest’ files in that directory are stable symlink, I am currently writing a descriptive web-page which will point to them since we expect the patch version number to evolve during 3.5 development (and I will be testing this soon, mostly to see what breaks!)<br/>
<br/>
All the usual caveats about un-tested code apply - although we will probably add some warnings over time to the builds themselves (splash screen etc) to remind people they’re not using a stable release.
  |{{cite web |url=http://sourceforge.net/p/flightgear/mailman/message/33628128/
    |title=<nowiki>[Flightgear-devel] Daily builds update</nowiki>
    |author=<nowiki>James Turner</nowiki>
    |date=<nowiki>2015-03-23</nowiki>
  }}
}}


If desired, it is possible to add a mailing list or other individual addresses to the email  notifications.
== User info ==
Given our commit rate, it is not clear if it warrants a new mailing list or not - it depends how often we break the build :)
[[File:About-dlg-2.10.png|180px|thumb|The about dialog]]


For any build, Hudson uses the Git changelogs to report what (and by whom!) is  new in the build.
As a user you can help developers to track bugs from the latest source code, by tracking bugs you would report them on the [[mailing list]] or [http://sourceforge.net/p/flightgear/codetickets/ bug tracker] (not the forums).
Currently the master is being used to do the Linux builds, because it was easy - no particular reason it has to be done that way, though.


= Goals =
{{FGCquote
  |The [[Integrated Qt5 Launcher |GUI launcher]] included in these builds includes a new dialog to setup (and remember) the FGRoot using file picker. It does some version checking to hopefully make things as robust as possible for the user.
  |{{cite web |url=http://sourceforge.net/p/flightgear/mailman/message/33646906/
    |title=<nowiki>Re: [Flightgear-devel] Download description page</nowiki>
    |author=<nowiki>James Turner</nowiki>
    |date=<nowiki>2015-03-26</nowiki>
  }}
}}


The objective of such systems is that there should be *zero* human steps to create a release - not just out of laziness, but for repeatability. I.e don't write a checklist or 'howto' of creating a release, write a shell script that does the steps. (Or several). And check those scripts into a source control system, too.
{{note|Take care to mention that you're using a nightly build, and to specify when it was built (you can find details in FlightGear's Help > About dialog)}}


In general, such systems are good for capturing how repeatable a build process is -  and the experiences on each of the Linux/mac/Windows slaves seem to confirm this  
If you are interested in doing this please follow these instructions:


In general, when people report that the 'current code doesn't compile', we can direct them to the Hudson page from now on.
=== Linux ===
Well, unfortunately there is nothing interesting for you here :)


= Benefits =
Because Linux users have powerful tools to compile and use the latest source code, take a look at [[Scripted_Compilation_on_Linux_Debian/Ubuntu]]
* lets developers know 'instantly' (within a few minutes) if their change broke 'some other platform', for example 64-bit or Mac (or Windows) (this is the big one, but only matters for developers)
* it can run tests automatically (although right now our test suite is pretty much zero)
* builds can be archived and uploaded somewhere. This doesn't help Linux much, but on Mac (and Windows, when it works), this means anyone can download a latest build and test it, with no need to install compilers, libraries or anything - just download a .zip and run bleeding-edge-FG.


The catch is, for this to be nice, requires some scripting. The current mac slave produces a zip, but you need to know some Terminal magic to actually run the code (set DYLD_LIBRARY_PATH, basically).
=== Mac ===
You need two things to run the latest source code: The compiled source code and the data.


= Issues =
* You can download the compiled source code at: [http://download.flightgear.org/builds/nightly/ FlightGear Mac nightly build]
* The current mac slave produces a zip, but you need to know some terminal magic to actually run the code (set DYLD_LIBRARY_PATH, basically).
* You can download the data with git: <code>{{fgdata clone}}</code>


= Plans =
=== Windows ===
You need two things to run the latest source code: The compiled source code and the data.


The configuration is exportable as XML  files, the server is currently using the official Hudson apt-get package for Ubuntu, so it's a
* You can download the compiled source code at: [http://download.flightgear.org/builds/nightly/ FlightGear Win nightly build download]
fairly repeatable setup. Configuring the Windows slave VM with mingw is proving the biggest hassle - OSG is working.
* You can download the data with git: <code>{{fgdata clone}}</code>


'Soon' there will be a WinXP slave, with a MinGW build. Hopefully this will even extend to a NSIS installer script, if Fred has one lying around. At
== Developer info ==
which point we should have nightly installers available for Windows, and a happier Fred. (A VisualStudio build is also possible, but requires more
Here is a brief summary about the configuration of Jenkins.
interaction with someone else, who has an externally-addressable/tunnel-able box with VS installed).


If need anything you would contact either James, Gene or Clément (the mailing list is a good place to contact them if you don't know where to look at them)


At which point, doing a release means clicking a button on a webpage (on Hudson), and letting the slaves grind away for an hour or so. Magic!
=== master ===
The ''master'' manages the distribution of builds among the slaves.


= Options =
Basically, its job is to be connected to the slaves and ask them to execute the jobs which have been configured.


=== Linux ===
This is the Linux slave based on Fedora 21. In reality it is the same machine than ''master'' but with another login. That's why we want the SSH connection on ''localhost''.


* Another thing the server can do, is email/IRC people when the build breaks on Linux / FreeBSD / Mac / Win due to a commit - obviously very handy for the devs.  
This slave is configured to work in <code>/home/jenkins</code>, that's where we have plenty of free space disk on this machine.


* So, another step is to email committers directly when they break the build. If you're a commiter to FG or SG, you may see some odd emails
=== Mac ===
This is the Mac slave.


* set up a buildser...@flightgear.org address.
=== Windows ===
This is the Windows slave based on Windows Server 2008. It is located in the same LAN than master/Linux slave. There is MSVC2010 installed.


* Yet another thing it can do is run test suites - unfortunately we don't have many such tests at the moment.
This slave is configured to work in <code>G:\jenkins</code>, that's where we have plenty of free space disk on this machine.


* If anyone wants to get into providing nightly .debs or .rpms, that could also be done, but requires people who know those systems, and again can provide a suitable externally address slave to run the builds.
=== Release process ===
The Linux slave has the responsibility to provide the base-package for the Linux-release and Windows-release job.


* If there's other configurations people wish to test (64-bit Linux, in particular), get in touch and they can be added.  
Because the Linux and Windows slave are on the same LAN it is not a problem to exchange 1 GB on the network.


* If it's just for running the monitor, then we probably should talk about putting it onto The MapServer as well
<code>Windows-release</code>, <code>Mac-release</code> and <code>Linux-release</code> have to be triggered by-hand. The result is automatically uploaded to http://fgfs.goneabitbursar.com/releases/.


* Build jobs can run arbitrary shell scripts - they can tag things in CVS or Git, they can create tarballs, upload files to SFTP/FTP servers, the works. So, if Durk/Curt/Fred could codify, somewhere, the steps (in terms of 'things doable in a shell/.bat script') to create an FG pre-release and final-release, the process can be automated.
[[Category:Core developer documentation]]
 
[[Category: Release]]
* Set up a cross compiler version of gcc at flightgear.org to automatically create binary packages (releases) of FlightGear for platforms such as Win32
 
= Related Discussions =
* http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg27592.html
* http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg27918.html
* http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg28260.html

Latest revision as of 10:28, 29 March 2018

Release process

The FlightGear build server, sometimes called the nightly build server or just Jenkins, is a system based on Jenkins which helps core developers to check that their source code compiles correctly for all OS. It is also responsible for building release packages for Linux, Windows and Mac.

Note 

The output files on Jenkins download.flightgear.org and sf.net should always be the same. Jenkins pushes them to download.flightgear.org from where they get distributed to sf.net (with some renaming-magic involved). Direct dowloads from Jenkins should be avoided as much as possible. [1]


Some core developers are working with Linux and have no access to a Windows installation in order to test if what they have implemented for Linux will works for Windows or Mac. The build server will automatically start a new compilation after each commit is pushed to the source code repository. We have the ability to compile the source code for Linux, Windows and Mac.

A "nightly build" is one that is automatically generated each night (hence the name) from the most recent code. Since it has not received as much testing as normal FlightGear versions, you might encounter some bugs; on the contrary, you get access to the latest-and-greatest features before they are officially released.

[2]

News (06/2015)

1rightarrow.png See Building FlightGear - Cross Compiling for the main article about this subject.

Cquote1.png mingw has came a long way, and i think the MXE openscenegraph package (currently at 3.2.1 on master!!), is beautifully maintained, now it builds almost all core libraries dynamically with some argument passing, even as a static target(MXE_TARGETS='i686-w64-mingw32.static'), but it's those plugins again, with their linking errors! i think these are because i'm using the i686-w64-mingw32.static-g++ compiler as opposed to the shared one...


so as soon as i can get the shared build environment running and solve all of it's dependancies for OSG, i think we can have a cross compiller in our hands!


Cquote2.png

SourceForge release candidate

As of 2018.1 release candidates are available from sourceforce https://sourceforge.net/projects/flightgear/files/release-candidate/ <https://sourceforge.net/projects/flightgear/files/release-candidate/>;

Where to find it

The FlightGear build server is available at: http://build.flightgear.org:8080/. There is no artifact available because we don't want copy +1GB to Jenkins master each days. This job is executed once per day.


You can get the artifact at {project infrastructure|buildserver}}

Cquote1.png We’ve improved the tool around daily build creation. Still some progress to be done but daily builds for Mac and Windows are uploaded to:


https://sourceforge.net/projects/flightgear/files/unstable/
(And hence, all the SF mirrors)

And also directly available from:

http://download.flightgear.org/builds/nightly/ <http://download.flightgear.org/builds/nightly/>;

The ‘latest’ files in that directory are stable symlink, I am currently writing a descriptive web-page which will point to them since we expect the patch version number to evolve during 3.5 development (and I will be testing this soon, mostly to see what breaks!)

All the usual caveats about un-tested code apply - although we will probably add some warnings over time to the builds themselves (splash screen etc) to remind people they’re not using a stable release.


— James Turner (2015-03-23). [Flightgear-devel] Daily builds update.
(powered by Instant-Cquotes)
 Cquote2.png

User info

The about dialog

As a user you can help developers to track bugs from the latest source code, by tracking bugs you would report them on the mailing list or bug tracker (not the forums).

Cquote1.png The GUI launcher included in these builds includes a new dialog to setup (and remember) the FGRoot using file picker. It does some version checking to hopefully make things as robust as possible for the user.
— James Turner (2015-03-26). Re: [Flightgear-devel] Download description page.
(powered by Instant-Cquotes)
 Cquote2.png
Note  Take care to mention that you're using a nightly build, and to specify when it was built (you can find details in FlightGear's Help > About dialog)

If you are interested in doing this please follow these instructions:

Linux

Well, unfortunately there is nothing interesting for you here :)

Because Linux users have powerful tools to compile and use the latest source code, take a look at Scripted_Compilation_on_Linux_Debian/Ubuntu

Mac

You need two things to run the latest source code: The compiled source code and the data.

  • You can download the compiled source code at: FlightGear Mac nightly build
  • You can download the data with git: git clone git://git.code.sf.net/p/flightgear/fgdata/

Windows

You need two things to run the latest source code: The compiled source code and the data.

Developer info

Here is a brief summary about the configuration of Jenkins.

If need anything you would contact either James, Gene or Clément (the mailing list is a good place to contact them if you don't know where to look at them)

master

The master manages the distribution of builds among the slaves.

Basically, its job is to be connected to the slaves and ask them to execute the jobs which have been configured.

Linux

This is the Linux slave based on Fedora 21. In reality it is the same machine than master but with another login. That's why we want the SSH connection on localhost.

This slave is configured to work in /home/jenkins, that's where we have plenty of free space disk on this machine.

Mac

This is the Mac slave.

Windows

This is the Windows slave based on Windows Server 2008. It is located in the same LAN than master/Linux slave. There is MSVC2010 installed.

This slave is configured to work in G:\jenkins, that's where we have plenty of free space disk on this machine.

Release process

The Linux slave has the responsibility to provide the base-package for the Linux-release and Windows-release job.

Because the Linux and Windows slave are on the same LAN it is not a problem to exchange 1 GB on the network.

Windows-release, Mac-release and Linux-release have to be triggered by-hand. The result is automatically uploaded to http://fgfs.goneabitbursar.com/releases/.

  1. ↑ Torsten Dreyer  (Dec 8th, 2016).  Re: [Flightgear-devel] [from forum] Torrenting FlightGear .
  2. ↑ elgaton (May 1st, 2016). Re: Tutorial: How to get Aircraft Center to work.
Retrieved from "https://wiki.flightgear.org/w/index.php?title=FlightGear_build_server&oldid=114773"