TerraGear scenery build server

From FlightGear wiki
Revision as of 17:23, 12 January 2012 by Hooray (Talk | contribs) (collect everything related to a scenery build server (frequently brought up discussion on the forums))

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Problem

Honestly compiling TerraGear could be the biggest issue

The main trouble is in compiling TerraGear and the fact it doesn't always play nicely. I think the fact the tutorial on the wiki is segmented doesn't help either - it's far easier to put all your elevation data in one folder, all your shapefiles in another folder, your airport data in another folder and then start running the process.

Many people (esp Windows users) have reported problems getting TerraGear tools to run. This is probably a significant factor limiting community contribution to FG custom scenery.

A 'build server' is one solution to this problem -- a server that accepts public-contributed shapefiles, builds the tile in an automated process (using the latest bugfixed TerraGear tools), then provides a way for the user to download the resulting .btg files to check his work.

I thought of a ready to use vm that contains the necessary tools (terragear) + a python script that connects to a server, downloads the needed data / scripts and starts compiling the scenery. After the task is done the data would be uploaded back to the server.

The server could run a python script too, that manages the connected compile slaves. The script could take the compile slave capabilities (disk io, processors, line bandwidth) into account and assign tasks based on this to the slave.

All that is needed is someone to take the time setting up a Linux distro that comes with TerraGear and all required tools pre-built and pre-configured, using SuseStudio this isn't too complicated (the TG process however is, so you may need help from experienced TG users such as statto et al), this would get you a downloadable ISO file (CD/DVD or a USB boot image), which could in turn be installed by people, either on a dedicated box (or in a dual boot setup), or simply within vmware/virtualbox.

Finally, we could add a web frontend (perl,php or python) so that people could upload their own scenery to be built.

Let's pool our know-how and resources together and aim to setup a scenery build server!

Background

For what it's worth, the primary bottleneck for scenery generation is disk I/O -- reading and writing files. There is a computational aspect, but on the whole it's disk IO that will limit things.

Back in the day when I was cranking out the world scenery myself, I had a cluster of linux machines running in parallel -- all talking to an nfs server -- I thought it was a pretty slick setup at the time, but I'm sure a fast single modern PC could outperform the cluster of 20-25 PC's I was using at the time.

Compiling a detailed area of 3 square degrees across only takes a couple hours running on a not-very-powerful server. Admittedly that's not that large of an area, but I've never found the build process to be unreasonable, though I only work with small batches.

Objective

This is an idea that has been repeatedly brought up on the FG forums (see references below). The idea is to make it easier for people to build scenery, by providing a scenery build service (server actually) where people can upload their scenery to be built and get their scenery compiled remotely, without needing to be intimately familiar with TerraGear and without requiring TerraGear to be installed locally. This should also make it easier for people to use TerraGear, without having to build it from source.

Virtualization

Scenery compilation is demanding, so setting up a VM on a powerful system with scenery compilation tools would be a good idea. This might come in handy for people whose system is not as powerful and enable them to contribute to scenery building without requiring a powerful computer.

Feedback

  • Yes, that would indeed be interesting, with a small web interface to interact with the user. Always wanted to do it, never found the time...
  • Certainly sounds interesting, would mean I could build scenery while away from home which would be a huge bonus for me, as I spend most my time on my laptop!
  • There's certainly interest in this sort of thing, and I am pretty sure you'd get to see a number of feature requests for such a thing.
  • The neat thing about a VM image (i.e. VirtualBox-based) would be that it could not only be provided as a server, but that it would also be possible to create a bootable "live CD/DVD" - so that people could easily run and install this things locally.
  • I think that using virtualization is an excellent opportunity here to simplify participating in FG development, if we had "live CDs" for most criticial FG processes, we'd surely have more people able to contribute.
  • Though, I am really not sure if starting with TerraGear is the simplest approach to pursue, I think there are several potential pitfalls - especially given that it is currently being worked on, but also due it not really being the most intuitive piece of software in the FG eco system. For people without solid TG experience, this will be pretty challenging.
  • If you're just looking to get started, you might just as well look into creating VM images to set up a multiplayer server (fgms), this is well documented in the wiki.Even today, many people find it not so easy to set up everything properly. So having a VM image to easily run fgms might be a nice practice to get started? And you'd surely gather valuable experience for your TerraGear image, too.
  • just trying the concept by coming up with a fully virtualized setup of fgms might be another option, that should be much easier to accomplish, but would be a nice exercise to see how things can be deployed, configured and run - sort of like a "beta test" to gather more experience with virtualizing FG related software.
  • The neat thing about this whole VM-based approach is that everything would be neatly encapsulated - so that would also make it possible to run several VMs at once (i.e. a fgms & terragear servers at once) while still controlling everything from the central host system, this could even be done using a simple GUI to easily configure and control all VMs.
  • If you manage to get this working, it would surely be a significant contribution - especially once people actually start supporting it to simplify things even further.
  • For example, the "TerraGear GUI" could be changed to support a "remote mode" where a build server can be used, so that scenery is built remotely. In the same sense, a web frontend could also be provided to allow people to easily use the server.
  • A QGIS/GRASS plugin might be another interesting option.

Crowd-compilation

  • A way to distribute work to a bunch of vm scenery bulding servers would be nice. Something people willing to help and with some extra disk space could run on their system in the normal downtime to share computing power to the scenery generating effort.
  • Yes, we also talked about this last year, it seems there are some SSH-based solutions available, i.e. they would use standard "scp" to copy stuff via ssh to a remote location. If I remember correctly, we also looked for a Python-based solution so that the TerraGear GUI could be modified accordingly.
  • I understand that generating the scenery from the raw data takes quite some CPU time. How about creating a BOINC project which could harness the idle CPU-power of many users to assist with this job?
  • Actually I dont have a clue about this (generating scenery) and I think there're many others like me out there. So a BOINC-Project would be a way to contribute to FG really ease! I would join this.

BOINC

Also in that case, a BOINC project would help because then for a million IOs there are a million PCs doing each 1 IO at the same time instead of 1 PC queueing up a million IOs ^^. But automatizing the whole process a little bit could also help. As I mentioned in another thread, I have 10MBit/s sym inet, a fairly powerful server which runs 24/7 and has RAIDs (IO-speed!) and now I upgraded my workstation also. If it's a BOINC project or just an automated script which pulls the stuff and goes through the workflow, I would be happy to contribute to the rendering of the scenery. Just familiarizing myself with geospatial sciences, getting into various GIS programs and doing the process by hand is a dealbreaker for me. I just don't have the time to get into all this in depth, unfortunately.

Maybe it actually would be a good idea for the Terragear people to concentrate on toolchain and workflow for a moment and come up with something that in basic can work fairly automatically. That would for sure be an investment into the future, because toolchains and workflows which need extensive manual intervention not only eat up resources unnecessarily but also include inherent risks for the whole project because these kinds of manual operations require specific knowledge which is incorporated in persons. If the persons leave the project for what reason ever, the whole project is in jeopardy.

Remastering

Also note that when we looked into this last year, we found a number of Linux distributions that are specifically designed to support "re-mastering", i.e. you can create a custom Linux distribution just by setting everything up as needed, and then create a bootable live CD automatically (i.e. scripted or GUI based) 100% based on the running "live session".

Basically, this would allow you to download such a distro's ISO image, then install it into a VM - set up everything as needed, and create a completely new ISO/VM image based on your customizations.

Note that you can do all of this easily using the free SuseSTUDIO service provided by OpenSUSE. This is a completely web-driven wizard to create OpenSUSE-based live CD/DVD images, completely set up and customized to your own needs.

You can even test and run the images remotely, but you can obviously also download the images once things are working. It is also possible to set up "projects" - so that several people can work together.

Approach

Use a remastered linux distribution (i.e. using http://www.susestudio.com/ ), which already contains TerraGear and all required tools/dependencies in a pre-built and pre-configured fashion.

Concerns and open issues

  • Another thing worth keeping in mind is compatibility, i.e. 32 bit vs. 64 bit OS. In the case of TG, 64 bit architectures offer theoretical benefits, regarding memory use - OTOH, many people these days are still running 32 bit OS.

References