FlightGear build server: Difference between revisions

From FlightGear wiki
Jump to navigation Jump to search
m (reodering and adding more details about the benefits of automated builds)
Line 17: Line 17:
It does chew a bit of disk-space, since the master stores the artifacts for the last N builds, where N is configurable. The artifacts are a hundred megabytes or so, since it's all the header files, libs and binaries, though compressed of course.
It does chew a bit of disk-space, since the master stores the artifacts for the last N builds, where N is configurable. The artifacts are a hundred megabytes or so, since it's all the header files, libs and binaries, though compressed of course.


= Hosting Options =
= Goals =
If you know of any others, please do feel free to add new hosting options here. Some of these are not necessarily useful for directly hosting Hudson, but instead for building FlightGear on different platforms using SSH. This applies in particular to the various build farms.
 
The objective of such systems is that there should be *zero* human steps to create a release - not just out of laziness, but for repeatability. I.e don't write a checklist or 'howto' of creating a release, write a shell script that does the steps. (Or several). And check those scripts into a source control system, too.
 
In general, such systems are good for capturing how repeatable a build process is -  and the experiences on each of the Linux/mac/Windows slaves seem to confirm this.
 
Quoting [http://www.joelonsoftware.com/articles/fog0000000043.html The Joel Test: 12 Steps to Better Code]:
 
2. Can you make a build in one step?
By this I mean: how many steps does it take to make a shipping build from the latest source snapshot? On good teams, there's a single script you can 
run that does a full checkout from scratch, rebuilds every line of code, makes the EXEs, in all their various versions, languages, and #ifdef
combinations, creates the installation package, and creates the final media -- CDROM layout, download website, whatever.
If the process takes any more than one step, it is prone to errors. And when you get closer to shipping, you want to have a very fast cycle of fixing
the "last" bug, making the final EXEs, etc. If it takes 20 steps to compile the code, run the installation builder, etc., you're going to go crazy and
you're going to make silly mistakes.
For this very reason, the last company I worked at switched from WISE to InstallShield: we required that the installation process be able to run, from
a script, automatically, overnight, using the NT scheduler, and WISE couldn't run from the scheduler overnight, so we threw it out. (The kind folks at
WISE assure me that their latest version does support nightly builds.)
3. Do you make daily builds?
When you're using source control, sometimes one programmer accidentally checks in something that breaks the build. For example, they've added a new
source file, and everything compiles fine on their machine, but they forgot to add the source file to the code repository. So they lock their machine
and go home, oblivious and happy. But nobody else can work, so they have to go home too, unhappy.
Breaking the build is so bad (and so common) that it helps to make daily builds, to insure that no breakage goes unnoticed. On large teams, one good
way to insure that breakages are fixed right away is to do the daily build every afternoon at, say, lunchtime. Everyone does as many checkins as
possible before lunch. When they come back, the build is done. If it worked, great! Everybody checks out the latest version of the source and goes on 
working. If the build failed, you fix it, but everybody can keep on working with the pre-build, unbroken version of the source.
On the Excel team we had a rule that whoever broke the build, as their "punishment", was responsible for babysitting the builds until someone else
broke it. This was a good incentive not to break the build, and a good way to rotate everyone through the build process so that everyone learned how
it worked.
 
For additional information on the benefits of automated builds, please see: [http://www.joelonsoftware.com/articles/fog0000000023.html Daily Builds Are Your Friend (Joel on Software)].
 
In general, when people report that the 'current code doesn't compile', we can direct them to the Hudson page from now on.


* http://gcc.gnu.org/wiki/CompileFarm#How_to_Get_Involved.3F
* http://en.opensuse.org/Build_Service
* https://edge.launchpad.net/builders/
* http://hub.opensolaris.org/bin/view/Community+Group+testing/testfarm
* http://www.metamodul.com/10.html
* http://www.gnu.org/software/hurd/public_hurd_boxen.html
* http://www-03.ibm.com/systems/z/os/linux/support/lcds/


= Status 07/2010 =
= Status 07/2010 =
Line 43: Line 72:
Currently the master is being used to do the Linux builds, because it was easy - no particular reason it has to be done that way, though.
Currently the master is being used to do the Linux builds, because it was easy - no particular reason it has to be done that way, though.


= Goals =


The objective of such systems is that there should be *zero* human steps to create a release - not just out of laziness, but for repeatability. I.e don't write a checklist or 'howto' of creating a release, write a shell script that does the steps. (Or several). And check those scripts into a source control system, too.  
= Hosting Options =
If you know of any others, please do feel free to add new hosting options here. Some of these are not necessarily useful for directly hosting Hudson, but instead for building FlightGear on different platforms using SSH. This applies in particular to the various build farms.


In general, such systems are good for capturing how repeatable a build process is -  and the experiences on each of the Linux/mac/Windows slaves seem to confirm this
* http://gcc.gnu.org/wiki/CompileFarm#How_to_Get_Involved.3F
* http://en.opensuse.org/Build_Service
* https://edge.launchpad.net/builders/
* http://hub.opensolaris.org/bin/view/Community+Group+testing/testfarm
* http://www.metamodul.com/10.html
* http://www.gnu.org/software/hurd/public_hurd_boxen.html
* http://www-03.ibm.com/systems/z/os/linux/support/lcds/


In general, when people report that the 'current code doesn't compile', we can direct them to the Hudson page from now on.


= Benefits =
= Benefits =

Revision as of 12:26, 2 July 2010

Intro

A *prototype* of a Hudson based build server for building FG (including OSG and SimGear) can be found at http://zakalawe.ath.cx:8080/

This is currently running on a core developer's home box:

"I've found some spare hardware to run the build server (Hudson) on, and it seems bearable for me,
but I suspect less pleasant for people on the other end of my cable connection."[1]

So, the server will need a proper home if it moves beyond the prototype stage.

For people who don't know, a build server talks to some slaves, and grabs/builds/tests/packages code. The current server is talking to one slave, which is an Ubuntu VM (Virtual Machine) which is building the 'next' branch on Gitorious. Any slave could be a VM, of course - they use CPU resources while building, but unlike other projects, our commit rate isn't that high - the slaves will be idle most of the time (A Mac slave is also possible, but requires some more work).

Note: If anyone wishes to volunteer a proper server (with a reasonably symmetric connection) to run Hudson, please get in touch using the mailing list or the FlightGear forums - any Unix will do, for Ubuntu/Debian there's an easy apt.get source available. All the setup can be done remotely, given SSH access. The disk, memory, CPU and bandwidth requirements are pretty moderate, due to our low commit rate.

It does chew a bit of disk-space, since the master stores the artifacts for the last N builds, where N is configurable. The artifacts are a hundred megabytes or so, since it's all the header files, libs and binaries, though compressed of course.

Goals

The objective of such systems is that there should be *zero* human steps to create a release - not just out of laziness, but for repeatability. I.e don't write a checklist or 'howto' of creating a release, write a shell script that does the steps. (Or several). And check those scripts into a source control system, too.

In general, such systems are good for capturing how repeatable a build process is - and the experiences on each of the Linux/mac/Windows slaves seem to confirm this.

Quoting The Joel Test: 12 Steps to Better Code:

2. Can you make a build in one step?
By this I mean: how many steps does it take to make a shipping build from the latest source snapshot? On good teams, there's a single script you can  
run that does a full checkout from scratch, rebuilds every line of code, makes the EXEs, in all their various versions, languages, and #ifdef 
combinations, creates the installation package, and creates the final media -- CDROM layout, download website, whatever.

If the process takes any more than one step, it is prone to errors. And when you get closer to shipping, you want to have a very fast cycle of fixing 
the "last" bug, making the final EXEs, etc. If it takes 20 steps to compile the code, run the installation builder, etc., you're going to go crazy and 
you're going to make silly mistakes.

For this very reason, the last company I worked at switched from WISE to InstallShield: we required that the installation process be able to run, from 
a script, automatically, overnight, using the NT scheduler, and WISE couldn't run from the scheduler overnight, so we threw it out. (The kind folks at 
WISE assure me that their latest version does support nightly builds.)

3. Do you make daily builds?
When you're using source control, sometimes one programmer accidentally checks in something that breaks the build. For example, they've added a new 
source file, and everything compiles fine on their machine, but they forgot to add the source file to the code repository. So they lock their machine 
and go home, oblivious and happy. But nobody else can work, so they have to go home too, unhappy.

Breaking the build is so bad (and so common) that it helps to make daily builds, to insure that no breakage goes unnoticed. On large teams, one good 
way to insure that breakages are fixed right away is to do the daily build every afternoon at, say, lunchtime. Everyone does as many checkins as 
possible before lunch. When they come back, the build is done. If it worked, great! Everybody checks out the latest version of the source and goes on  
working. If the build failed, you fix it, but everybody can keep on working with the pre-build, unbroken version of the source.

On the Excel team we had a rule that whoever broke the build, as their "punishment", was responsible for babysitting the builds until someone else 
broke it. This was a good incentive not to break the build, and a good way to rotate everyone through the build process so that everyone learned how 
it worked. 

For additional information on the benefits of automated builds, please see: Daily Builds Are Your Friend (Joel on Software).

In general, when people report that the 'current code doesn't compile', we can direct them to the Hudson page from now on.


Status 07/2010

The Mac build is pretty close to producing a nightly, though - we need to fix a genuine (and long-standing) configuration issue on Mac.

The good news - thanks to some great documentation from Fred, the Windows build is stable, using MSVC no less. You can grab a fgfs.exe (and .pdb) from any successful build, and it should 'just work', if you drop it into a current 2.0 install. Future work on some improved packaging of these builds will follow.

The better news - all three platforms are now green (i.e, building correctly), and should update reliably (which will be monitored). At present, the server admin gets emailed when a build breaks, and when it goes green again. These build from Gitorious next, and there will probably be experiments to make it complain to IRC, or even to the mailing list, when the build breaks.

If desired, it is possible to add a mailing list or other individual addresses to the email notifications. Given our commit rate, it is not clear if it warrants a new mailing list or not - it depends how often we break the build :)

For any build, Hudson uses the Git changelogs to report what (and by whom!) is new in the build. Currently the master is being used to do the Linux builds, because it was easy - no particular reason it has to be done that way, though.


Hosting Options

If you know of any others, please do feel free to add new hosting options here. Some of these are not necessarily useful for directly hosting Hudson, but instead for building FlightGear on different platforms using SSH. This applies in particular to the various build farms.


Benefits

  • lets developers know 'instantly' (within a few minutes) if their change broke 'some other platform', for example 64-bit or Mac (or Windows) (this is the big one, but only matters for developers)
  • it can run tests automatically (although right now our test suite is pretty much zero)
  • builds can be archived and uploaded somewhere. This doesn't help Linux much, but on Mac (and Windows, when it works), this means anyone can download a latest build and test it, with no need to install compilers, libraries or anything - just download a .zip and run bleeding-edge-FG.

The catch is, for this to be nice, requires some scripting. The current mac slave produces a zip, but you need to know some Terminal magic to actually run the code (set DYLD_LIBRARY_PATH, basically).

Issues

  • The current mac slave produces a zip, but you need to know some terminal magic to actually run the code (set DYLD_LIBRARY_PATH, basically).

Plans

The configuration is exportable as XML files, the server is currently using the official Hudson apt-get package for Ubuntu, so it's a fairly repeatable setup. Configuring the Windows slave VM with mingw is proving the biggest hassle - OSG is working.

'Soon' there will be a WinXP slave, with a MinGW build. Hopefully this will even extend to a NSIS installer script, if Fred has one lying around. At which point we should have nightly installers available for Windows, and a happier Fred. (A VisualStudio build is also possible, but requires more interaction with someone else, who has an externally-addressable/tunnel-able box with VS installed).


At which point, doing a release means clicking a button on a webpage (on Hudson), and letting the slaves grind away for an hour or so. Magic!

Options

  • Another thing the server can do, is email/IRC people when the build breaks on Linux / FreeBSD / Mac / Win due to a commit - obviously very handy for the devs.
  • So, another step is to email committers directly when they break the build. If you're a commiter to FG or SG, you may see some odd emails
  • set up a buildser...@flightgear.org address.
  • Yet another thing it can do is run test suites - unfortunately we don't have many such tests at the moment.
  • If anyone wants to get into providing nightly .debs or .rpms, that could also be done, but requires people who know those systems, and again can provide a suitable externally address slave to run the builds.
  • If there's other configurations people wish to test (64-bit Linux, in particular), get in touch and they can be added.
  • If it's just for running the monitor, then we probably should talk about putting it onto The MapServer as well
  • Build jobs can run arbitrary shell scripts - they can tag things in CVS or Git, they can create tarballs, upload files to SFTP/FTP servers, the works. So, if Durk/Curt/Fred could codify, somewhere, the steps (in terms of 'things doable in a shell/.bat script') to create an FG pre-release and final-release, the process can be automated.
  • Set up a cross compiler version of gcc at flightgear.org to automatically create binary packages (releases) of FlightGear for platforms such as Win32

Related Discussions