TerraGear scenery build server: Difference between revisions

From FlightGear wiki
Jump to navigation Jump to search
No edit summary
No edit summary
Line 7: Line 7:


[[File:TerraGear-BuildServer-Prototype.png|thumb|TerraGear BuildServer Experiments]]
[[File:TerraGear-BuildServer-Prototype.png|thumb|TerraGear BuildServer Experiments]]
[[File:TerraGear-GUI-Remote-Mode.png|thumb|Prototyping a remote mode for [[TerraGear]] as part of [[TerraGear GUI]]]]


* Last update 01/2012
* Last update 01/2012

Revision as of 07:20, 15 February 2014

Note: While this article is based on considerable community feedback, there's nobody working on this currently.
So if you'd like to help in one way or another, please get in touch or just help improve the article in the meantime!
Useful Skills:
Virtualization, Linux, TerraGear, SSH, Qt, TurnKeyLinux


People:

Mentors: Hooray(get in touch to learn more)
It's possible that this article hasn't been updated in a while, so to catch up with the latest developments, you are advised not to start working on anything directly related to this without first coordinating your ideas with fellow FlightGear contributors using the FlightGear developers mailing list or the FlightGear forums. See also the talk page.


This article is a stub. You can help the wiki by expanding it.
TerraGear BuildServer Experiments
Prototyping a remote mode for TerraGear as part of TerraGear GUI
  • Last update 01/2012
  • Looking for people familiar with TerraGear
  • Looking for people who are familiar with Linux, virtualization

Problem

Honestly compiling TerraGear could be the biggest issue

The main trouble is in compiling TerraGear and the fact it doesn't always play nicely. I think the fact the tutorial on the wiki is segmented doesn't help either - it's far easier to put all your elevation data in one folder, all your shapefiles in another folder, your airport data in another folder and then start running the process.

Many people (esp Windows users) have reported problems getting TerraGear tools to run. This is probably a significant factor limiting community contribution to FG custom scenery.

A 'build server' is one solution to this problem -- a server that accepts public-contributed shapefiles, builds the tile in an automated process (using the latest bugfixed TerraGear tools), then provides a way for the user to download the resulting .btg files to check his work.

I thought of a ready to use vm that contains the necessary tools (terragear) + a python script that connects to a server, downloads the needed data / scripts and starts compiling the scenery. After the task is done the data would be uploaded back to the server.

The server could run a python script too, that manages the connected compile slaves. The script could take the compile slave capabilities (disk io, processors, line bandwidth) into account and assign tasks based on this to the slave.

All that is needed is someone to take the time setting up a Linux distro that comes with TerraGear and all required tools pre-built and pre-configured, using SuseStudio this isn't too complicated (the TG process however is, so you may need help from experienced TG users such as statto et al), this would get you a downloadable ISO file (CD/DVD or a USB boot image), which could in turn be installed by people, either on a dedicated box (or in a dual boot setup), or simply within vmware/virtualbox.

Finally, we could add a web frontend (perl,php or python) so that people could upload their own scenery to be built.

Let's pool our know-how and resources together and aim to setup a scenery build server!

Background

For what it's worth, the primary bottleneck for scenery generation is disk I/O -- reading and writing files. There is a computational aspect, but on the whole it's disk IO that will limit things.

Back in the day when I was cranking out the world scenery myself, I had a cluster of linux machines running in parallel -- all talking to an nfs server -- I thought it was a pretty slick setup at the time, but I'm sure a fast single modern PC could outperform the cluster of 20-25 PC's I was using at the time.

Compiling a detailed area of 3 square degrees across only takes a couple hours running on a not-very-powerful server. Admittedly that's not that large of an area, but I've never found the build process to be unreasonable, though I only work with small batches.

Objective

This is an idea that has been repeatedly brought up on the FG forums (see references below). The idea is to make it easier for people to build scenery, by providing a scenery build service (server actually) where people can upload their scenery to be built and get their scenery compiled remotely, without needing to be intimately familiar with TerraGear and without requiring TerraGear to be installed locally. This should also make it easier for people to use TerraGear, without having to build it from source.

Virtualization

Scenery compilation is demanding, so setting up a VM on a powerful system with scenery compilation tools would be a good idea. This might come in handy for people whose system is not as powerful and enable them to contribute to scenery building without requiring a powerful computer.

Plan / Status

The first step would probably be finding someone familiar with installing/setting up TG on Linux, and then also someone familiar with virtualization, so that a Linux distro like TurnkeyLinux can be set up with TerraGear - from that point on, everything should be fairly straightforward actually, assuming that we have someone willing to work on a web-interface -or API- to expose TerraGear as a web service, e.g. using Python, Php, Ruby or Perl - in fact, one could even use Nasal for this

Hooray is probably not going to work on this anytime soon, but is willing to help work out the details to get this started, so if you're interested in helping, please do get in touch!

One of the most straightforward ways to proceed here, would be just setting up TerraGear and using SSH/SCP to interact with the service - obviously, that would require either setting up accounts for each user, or setting up a shared/trusted account - in turn, this could then be supported by Gijs' TerraGear GUI - i.e. kinda like a "remote operating" mode that works over SSH - that would allow us to reuse the existing GUI, without having to come up with a new WebService API.

Qt has various APIs to handle SSH, e.g. see: http://doc.qt.digia.com/qtcreator-extending/utils-sshconnection.html

  • Download & Install Virtualbox Done Done
  • Download TurnKeyLinux Done Done
  • Set up new VM Done Done
  • Read up on TKLDev 20}% completed
  • Install SimGear in Headless mode Building_using_CMake#SimGear_Build_Options Not done Not done
  • Install TerraGear inside the VM using the Building FlightGear - Debian#TerraGear script 10}% completed
  • Get in touch with Gijs to patch TerraGear GUI to support operating over SSH Not done Not done
  • Get in touch with people volunteering server hardware Not done Not done
  • Get in touch with the TerraGear guys (psadro_gm & papillon81) to help optimize TerraGear for server-focused operation Not done Not done
  • Investigate clustering options, i.e. via distcc Not done Not done

Feedback

  • Yes, that would indeed be interesting, with a small web interface to interact with the user. Always wanted to do it, never found the time...
  • Certainly sounds interesting, would mean I could build scenery while away from home which would be a huge bonus for me, as I spend most my time on my laptop!
  • There's certainly interest in this sort of thing, and I am pretty sure you'd get to see a number of feature requests for such a thing.
  • The neat thing about a VM image (i.e. VirtualBox-based) would be that it could not only be provided as a server, but that it would also be possible to create a bootable "live CD/DVD" - so that people could easily run and install this things locally.
  • I think that using virtualization is an excellent opportunity here to simplify participating in FG development, if we had "live CDs" for most criticial FG processes, we'd surely have more people able to contribute.
  • Though, I am really not sure if starting with TerraGear is the simplest approach to pursue, I think there are several potential pitfalls - especially given that it is currently being worked on, but also due it not really being the most intuitive piece of software in the FG eco system. For people without solid TG experience, this will be pretty challenging.
  • If you're just looking to get started, you might just as well look into creating VM images to set up a multiplayer server (fgms), this is well documented in the wiki.Even today, many people find it not so easy to set up everything properly. So having a VM image to easily run fgms might be a nice practice to get started? And you'd surely gather valuable experience for your TerraGear image, too.
  • just trying the concept by coming up with a fully virtualized setup of fgms might be another option, that should be much easier to accomplish, but would be a nice exercise to see how things can be deployed, configured and run - sort of like a "beta test" to gather more experience with virtualizing FG related software.
  • The neat thing about this whole VM-based approach is that everything would be neatly encapsulated - so that would also make it possible to run several VMs at once (i.e. a fgms & terragear servers at once) while still controlling everything from the central host system, this could even be done using a simple GUI to easily configure and control all VMs.
  • If you manage to get this working, it would surely be a significant contribution - especially once people actually start supporting it to simplify things even further.
  • For example, the "TerraGear GUI" could be changed to support a "remote mode" where a build server can be used, so that scenery is built remotely. In the same sense, a web frontend could also be provided to allow people to easily use the server.
  • A QGIS/GRASS plugin might be another interesting option.

Crowd-compilation

  • A way to distribute work to a bunch of vm scenery bulding servers would be nice. Something people willing to help and with some extra disk space could run on their system in the normal downtime to share computing power to the scenery generating effort.
  • Yes, we also talked about this last year, it seems there are some SSH-based solutions available, i.e. they would use standard "scp" to copy stuff via ssh to a remote location. If I remember correctly, we also looked for a Python-based solution so that the TerraGear GUI could be modified accordingly.
  • I understand that generating the scenery from the raw data takes quite some CPU time. How about creating a BOINC project which could harness the idle CPU-power of many users to assist with this job?
  • Actually I dont have a clue about this (generating scenery) and I think there're many others like me out there. So a BOINC-Project would be a way to contribute to FG really ease! I would join this.

BOINC

Also in that case, a BOINC project would help because then for a million IOs there are a million PCs doing each 1 IO at the same time instead of 1 PC queueing up a million IOs ^^. But automatizing the whole process a little bit could also help. As I mentioned in another thread, I have 10MBit/s sym inet, a fairly powerful server which runs 24/7 and has RAIDs (IO-speed!) and now I upgraded my workstation also. If it's a BOINC project or just an automated script which pulls the stuff and goes through the workflow, I would be happy to contribute to the rendering of the scenery. Just familiarizing myself with geospatial sciences, getting into various GIS programs and doing the process by hand is a dealbreaker for me. I just don't have the time to get into all this in depth, unfortunately.

Maybe it actually would be a good idea for the Terragear people to concentrate on toolchain and workflow for a moment and come up with something that in basic can work fairly automatically. That would for sure be an investment into the future, because toolchains and workflows which need extensive manual intervention not only eat up resources unnecessarily but also include inherent risks for the whole project because these kinds of manual operations require specific knowledge which is incorporated in persons. If the persons leave the project for what reason ever, the whole project is in jeopardy.

Remastering

Also note that when we looked into this last year, we found a number of Linux distributions that are specifically designed to support "re-mastering", i.e. you can create a custom Linux distribution just by setting everything up as needed, and then create a bootable live CD automatically (i.e. scripted or GUI based) 100% based on the running "live session".

Basically, this would allow you to download such a distro's ISO image, then install it into a VM - set up everything as needed, and create a completely new ISO/VM image based on your customizations.

Note that you can do all of this easily using the free SuseSTUDIO service provided by OpenSUSE. This is a completely web-driven wizard to create OpenSUSE-based live CD/DVD images, completely set up and customized to your own needs.

You can even test and run the images remotely, but you can obviously also download the images once things are working. It is also possible to set up "projects" - so that several people can work together.

Approach

Use a remastered linux distribution (i.e. using http://www.susestudio.com/ ), which already contains TerraGear and all required tools/dependencies in a pre-built and pre-configured fashion.

Concerns and open issues

  • Another thing worth keeping in mind is compatibility, i.e. 32 bit vs. 64 bit OS. In the case of TG, 64 bit architectures offer theoretical benefits, regarding memory use - OTOH, many people these days are still running 32 bit OS.

References