TerraGear scenery build server
|
Note:
This project is currently under active development. And we're looking for volunteers interested in contributing.
Mentors: Hooray(get in touch to learn more) |
|
terragear tools are not exactly the most robust :) Building them can be a challenge, and getting useful output is just as hard.
|
|
|
|
I run genapts850 on linux 64bit with no problem. I've got a new machine, now and Windows 64 running on a virtual machine, but I haven't done any development work on Windows for 18 years now. It's really not very high on the priority list, as the world scenery is also generated on linux.
|
|
|
|
Remember, Terragear was never intended to used by an end user. It was designed to be run on a Linux server. It's been almost 3 years since I've started looking at this code. It took almost a full year of hacking on it before I was confident enough to commit changes. It is not user friendly. It was once my intention to make it easier to use and add features. Over time, it became obvious that this was optimistic. Bug fixes pretty much takes all of my time. |
|
|
|
User friendliness, and support for windows, 32 bit, etc have dropped to very low priority, as the server that Martin uses when generating the official scenery is well known to me.
|
|
|
|
If you feel something is poorly designed, and should be done differently, go ahead and try to improve it. If others can be convinced of the merit, it most likely will become the 'new way'. So many examples of this in the past 4 years of development.
— psadro_gm (Wed Jan 29). Re: Why are cockpits so integrated with planes?.
(powered by Instant-Cquotes) |
|
Reeed's concept is more like this:
| This article is a stub. You can help the wiki by expanding it. |
| This article describes content/features that may not yet be available in the latest stable version of FlightGear (2020.3). You may need to install some extra components, use the latest development (Git) version or even rebuild FlightGear from source, possibly from a custom topic branch using special build settings: This feature is scheduled for FlightGear 4.x.  If you'd like to learn more about getting your own ideas into FlightGear, check out Implementing new features for FlightGear. |
 | |
| Started in | 02/2014 |
|---|---|
| Description | Build Server for TerraGear |
| Maintainer(s) | looking for volunteers |
| Contributor(s) | looking for volunteers (since 02/2014), |
| Status | prototyping |
- Last update 02/2014
- Looking for people familiar with TerraGear
- Looking for people who are familiar with Linux, virtualization
News
As part of working on a long-standing feature request, namely having a central shared TerraGear scenery build server, saiarcot895 has created Linux packages for Debian/Ubuntu (deb/ppa) to install precompiled TerraGear binaries. This now greatly simplifies installing TerraGear, because people no longer need to manually set up a complete Linux build environment and all SimGear/TerraGear dependencies. Everything is now done in an automated fashion.
Note To test the precomiled packages on a debian/wheezy based distro: edit /etc/apt/sources.list.d/sources.list and add the TG repo provided by saiarcot895:
deb http://128.61.105.135/apt/wheezy main deb http://128.61.105.135/apt/wheezy contrib deb http://128.61.105.135/apt/wheezy non-free On Ubuntu Precise and Quantal the repo paths can be put in a new list file, create /etc/apt/sources.list.d/saiarcot895-terragear.wheezy.list and insert the the following. The format needs to be deb http://128.61.105.135/apt wheezy main deb http://128.61.105.135/apt wheezy contrib deb http://128.61.105.135/apt wheezy non-free Next, run apt-get update and then apt-get install terragear as root: # apt-get update && apt-get install terragear |
Note: This only installs the Terragear tool chain and not the GUI or any src. It by default installs it in /usr/bin folder unlike saiarcot895's download_and_install.sh (link may be broken [2]) which installs everything including the GUI and src, wherever you want to put it.
Next, we're hoping to use this to install TerraGear on a public server for which people can ask for remote shell access (SSH). People interested in exploring this, should be getting in touch via the forum or the wiki.
Obviously, users will still need to be familiar with TerraGear, but they may benefit from reduced bandwidth restrictions and/or more horsepower in comparison to running TerraGear locally (e.g. one user offered to contribute hosting on a 32gb RAM and 8-core server). So this could be a great opportunity for people to run scripted/unattended jobs, without having to go through the hassle of downloading/building/installing and configuring TerraGear. But so far, being familiar with TerraGear and Linux is going to be a prerequisite still.
Once that is working, we'll investigate making the setup reproducible by using TurnKeyLinux. Once we have a working TKL distro, we can install a full TG setup in just a few minutes by downloading an ISO file and installing it in a VM (VMWare/VirtualBox). This would basically allow people to easily download/install TerraGear locally, either installed next to their OS, or as a virtual machine.
The long term idea is to hook up TerraGear GUI to it, so that the GUI front-end talks to TerraGear across SSH.
This is currently still a use-case for which TG wasn't designed for, and the TG developers mentioned already on the forums that there may be some roadblocks ahead, so everything here is still highly experimental. But ultimately we hope to provide a front-end to a Linux-based TerraGear VM, either by reusing TerraGear GUI or by coming up with a custom web-based front-end (please get in touch if you can help with this!).
If you're interested in helping or learning more, please get in touch via the forum or via TerraGear scenery build server.
Problem
Honestly compiling TerraGear could be the biggest issue
The main trouble is in compiling TerraGear and the fact it doesn't always play nicely. I think the fact the tutorial on the wiki is segmented doesn't help either - it's far easier to put all your elevation data in one folder, all your shapefiles in another folder, your airport data in another folder and then start running the process.
Many people (esp Windows users) have reported problems getting TerraGear tools to run. This is probably a significant factor limiting community contribution to FG custom scenery.
A 'build server' is one solution to this problem -- a server that accepts public-contributed shapefiles, builds the tile in an automated process (using the latest bugfixed TerraGear tools), then provides a way for the user to download the resulting .btg files to check his work.
I thought of a ready to use vm that contains the necessary tools (terragear) + a python script that connects to a server, downloads the needed data / scripts and starts compiling the scenery. After the task is done the data would be uploaded back to the server.
The server could run a python script too, that manages the connected compile slaves. The script could take the compile slave capabilities (disk io, processors, line bandwidth) into account and assign tasks based on this to the slave.
All that is needed is someone to take the time setting up a Linux distro that comes with TerraGear and all required tools pre-built and pre-configured, using SuseStudio this isn't too complicated (the TG process however is, so you may need help from experienced TG users such as statto et al), this would get you a downloadable ISO file (CD/DVD or a USB boot image), which could in turn be installed by people, either on a dedicated box (or in a dual boot setup), or simply within vmware/virtualbox.
Finally, we could add a web frontend (perl,php or python) so that people could upload their own scenery to be built.
Let's pool our know-how and resources together and aim to setup a scenery build server!
|
|
I run windows, and have followed the winding toolchain trail to a brick wall. TerraGear for windows is apparently abandoned, broken, or obsolete.
— Kabuki (Sat Jul 05). Is it possible to create a new airport in Windows?.
(powered by Instant-Cquotes) |
|
|
|
What's the best option for me? Install some VM software and learn to use that so I can do fake linux? Or, run Ubuntu only for the sake of TerraGear? Or maybe create some airport data and hand it off to someone who can use TerraGear?
— Kabuki (Sat Jul 05). Is it possible to create a new airport in Windows?.
(powered by Instant-Cquotes) |
|
Background
For what it's worth, the primary bottleneck for scenery generation is disk I/O -- reading and writing files. There is a computational aspect, but on the whole it's disk IO that will limit things.
Back in the day when I was cranking out the world scenery myself, I had a cluster of linux machines running in parallel -- all talking to an nfs server -- I thought it was a pretty slick setup at the time, but I'm sure a fast single modern PC could outperform the cluster of 20-25 PC's I was using at the time.
Compiling a detailed area of 3 square degrees across only takes a couple hours running on a not-very-powerful server. Admittedly that's not that large of an area, but I've never found the build process to be unreasonable, though I only work with small batches.
Objective
This is an idea that has been repeatedly brought up on the FG forums (see references below). The idea is to make it easier for people to build scenery, by providing a scenery build service (server actually) where people can upload their scenery to be built and get their scenery compiled remotely, without needing to be intimately familiar with TerraGear and without requiring TerraGear to be installed locally. This should also make it easier for people to use TerraGear, without having to build it from source.
Virtualization
Scenery compilation is demanding, so setting up a VM on a powerful system with scenery compilation tools would be a good idea. This might come in handy for people whose system is not as powerful and enable them to contribute to scenery building without requiring a powerful computer.
Plan / Status (03/2014)
The first step would probably be finding someone familiar with installing/setting up TG on Linux, and then also someone familiar with virtualization, so that a Linux distro like TurnkeyLinux can be set up with TerraGear - from that point on, everything should be fairly straightforward actually, assuming that we have someone willing to work on a web-interface -or API- to expose TerraGear as a web service, e.g. using Python, Php, Ruby or Perl - in fact, one could even use Nasal for this
Hooray is probably not going to work on this anytime soon, but is willing to help work out the details to get this started, so if you're interested in helping, please do get in touch!
One of the most straightforward ways to proceed here, would be just setting up TerraGear and using SSH/SCP to interact with the service - obviously, that would require either setting up accounts for each user, or setting up a shared/trusted account - in turn, this could then be supported by Gijs' TerraGear GUI - i.e. kinda like a "remote operating" mode that works over SSH - that would allow us to reuse the existing GUI, without having to come up with a new WebService API.
Qt has various APIs to handle SSH, e.g. see: http://doc.qt.digia.com/qtcreator-extending/utils-sshconnection.html
- Download & Install Virtualbox
Done - Download TurnKeyLinux Done
- Set up new VM Done
- Read up on TKLDev
- Install SimGear in Headless mode Building_using_CMake#SimGear_Build_Options Done
- Install TerraGear inside the VM using the Building FlightGear - Debian#TerraGear script Done
- Investigate packaing SimGear headless (saiarcot895) Done
- Investigate packaging TerraGear as a deb/ppa (saiarcot895 ) Done
- Use Using TerraGear to come up with a test script
- Get in touch with Gijs to patch TerraGear GUI to support operating over SSH
Not done - Get in touch with people volunteering hosting Not done
- Get in touch with the TerraGear guys (psadro_gm & papillon81) to help optimize TerraGear for server-focused operation Not done
- Investigate clustering options, i.e. via distcc Not done
Feedback
- Yes, that would indeed be interesting, with a small web interface to interact with the user. Always wanted to do it, never found the time...
- Certainly sounds interesting, would mean I could build scenery while away from home which would be a huge bonus for me, as I spend most my time on my laptop!
- There's certainly interest in this sort of thing, and I am pretty sure you'd get to see a number of feature requests for such a thing.
- The neat thing about a VM image (i.e. VirtualBox-based) would be that it could not only be provided as a server, but that it would also be possible to create a bootable "live CD/DVD" - so that people could easily run and install this things locally.
- I think that using virtualization is an excellent opportunity here to simplify participating in FG development, if we had "live CDs" for most criticial FG processes, we'd surely have more people able to contribute.
- Though, I am really not sure if starting with TerraGear is the simplest approach to pursue, I think there are several potential pitfalls - especially given that it is currently being worked on, but also due it not really being the most intuitive piece of software in the FG eco system. For people without solid TG experience, this will be pretty challenging.
- If you're just looking to get started, you might just as well look into creating VM images to set up a multiplayer server (fgms), this is well documented in the wiki.Even today, many people find it not so easy to set up everything properly. So having a VM image to easily run fgms might be a nice practice to get started? And you'd surely gather valuable experience for your TerraGear image, too.
- just trying the concept by coming up with a fully virtualized setup of fgms might be another option, that should be much easier to accomplish, but would be a nice exercise to see how things can be deployed, configured and run - sort of like a "beta test" to gather more experience with virtualizing FG related software.
- The neat thing about this whole VM-based approach is that everything would be neatly encapsulated - so that would also make it possible to run several VMs at once (i.e. a fgms & terragear servers at once) while still controlling everything from the central host system, this could even be done using a simple GUI to easily configure and control all VMs.
- If you manage to get this working, it would surely be a significant contribution - especially once people actually start supporting it to simplify things even further.
- For example, the "TerraGear GUI" could be changed to support a "remote mode" where a build server can be used, so that scenery is built remotely. In the same sense, a web frontend could also be provided to allow people to easily use the server.
- A QGIS/GRASS plugin might be another interesting option.
Crowd-compilation
- A way to distribute work to a bunch of vm scenery bulding servers would be nice. Something people willing to help and with some extra disk space could run on their system in the normal downtime to share computing power to the scenery generating effort.
- Yes, we also talked about this last year, it seems there are some SSH-based solutions available, i.e. they would use standard "scp" to copy stuff via ssh to a remote location. If I remember correctly, we also looked for a Python-based solution so that the TerraGear GUI could be modified accordingly.
- I understand that generating the scenery from the raw data takes quite some CPU time. How about creating a BOINC project which could harness the idle CPU-power of many users to assist with this job?
- Actually I dont have a clue about this (generating scenery) and I think there're many others like me out there. So a BOINC-Project would be a way to contribute to FG really ease! I would join this.
BOINC
| Note In its current form, this section/article is largely based on quotes collected from various related discussions/channels (devel list, forum etc) using the Instant-Cquotes script. Wiki users and other contributors are encouraged to help rewrite/edit contents accordingly to help get rid of unnecessary quoting (i.e. while the wiki is not intended to be a collection of quotes, quotes are sometimes the best/easiest way to bootstrap new articles, while also providing a good way to link back to related discussions in the archives).
While rewriting usually only entails changing first person speech to 3rd person. However, please try to retain references/links to the original discussion whenever possible. |
|
|
For what it's worth, the primary bottleneck for scenery generation is disk I/O -- reading and writing files. There is a computational aspect, but on the whole it's disk IO that will limit things.
Back in the day when I was cranking out the world scenery myself, I had a cluster of linux machines running in parallel -- all talking to an nfs server -- I thought it was a pretty slick setup at the time, but I'm sure a fast single modern PC could outperform the cluster of 20-25 PC's I was using at the time. |
|
|
|
I understand that generating the scenery from the raw data takes quite some CPU time.
How about creating a BOINC project which could harness the idle CPU-power of many users to assist with this job? — sgofferj (Dec 9th, 2011). Create a BOINC project for scenery generation?.
(powered by Instant-Cquotes) |
|
|
|
The only alternative to do it more quickly, in my opinion, would be using a distributed architecture - e.g. BOINC - or a cluster of servers.) |
|
|
|
when our main scenery resource will be gone in a couple of weeks, we will have a hard time to create a new world scenery. Generating the entire world was a job that ran continuously for days - if not weeks, even on a powerful machin with almost unlimited resources. As we have to re-think the entire process anyway, I just had the idea to use our users resources for that, probably by using something like BOINC: https://boinc.berkeley.edu/trac/wiki/AppIntro We sure need to do many tweaks within the tool-chain, but I can't think of any reason why this should not work. Does anybody have experience with BOINC or alike? — Torsten Dreyer (Dec 3rd, 2015). [Flightgear-devel] Continous, Distributed Generation of Terrain.
(powered by Instant-Cquotes) |
|
|
|
we could adopt an even simpler approach:
|
|
|
|
I thought of a ready to use vm that contains the necessary tools (terragear) + a python script that connects to a server, downloads the needed data / scripts and starts compiling the scenery.
After the task is done the data would be uploaded back to the server. The server could run a python script too, that manages the connected compile slaves. The script could take the compile slave capabilities (disk io, processors, line bandwidth) into account and assign tasks based on this to the slave. |
|
Also in that case, a BOINC project would help because then for a million IOs there are a million PCs doing each 1 IO at the same time instead of 1 PC queueing up a million IOs ^^. But automatizing the whole process a little bit could also help. As I mentioned in another thread, I have 10MBit/s sym inet, a fairly powerful server which runs 24/7 and has RAIDs (IO-speed!) and now I upgraded my workstation also. If it's a BOINC project or just an automated script which pulls the stuff and goes through the workflow, I would be happy to contribute to the rendering of the scenery. Just familiarizing myself with geospatial sciences, getting into various GIS programs and doing the process by hand is a dealbreaker for me. I just don't have the time to get into all this in depth, unfortunately.
Maybe it actually would be a good idea for the Terragear people to concentrate on toolchain and workflow for a moment and come up with something that in basic can work fairly automatically. That would for sure be an investment into the future, because toolchains and workflows which need extensive manual intervention not only eat up resources unnecessarily but also include inherent risks for the whole project because these kinds of manual operations require specific knowledge which is incorporated in persons. If the persons leave the project for what reason ever, the whole project is in jeopardy.
Remastering
Also note that when we looked into this last year, we found a number of Linux distributions that are specifically designed to support "re-mastering", i.e. you can create a custom Linux distribution just by setting everything up as needed, and then create a bootable live CD automatically (i.e. scripted or GUI based) 100% based on the running "live session".
Basically, this would allow you to download such a distro's ISO image, then install it into a VM - set up everything as needed, and create a completely new ISO/VM image based on your customizations.
Note that you can do all of this easily using the free SuseSTUDIO service provided by OpenSUSE. This is a completely web-driven wizard to create OpenSUSE-based live CD/DVD images, completely set up and customized to your own needs.
You can even test and run the images remotely, but you can obviously also download the images once things are working. It is also possible to set up "projects" - so that several people can work together.
Approach
Use a remastered linux distribution (i.e. using http://www.susestudio.com/ ), which already contains TerraGear and all required tools/dependencies in a pre-built and pre-configured fashion.
Concerns and open issues
- Another thing worth keeping in mind is compatibility, i.e. 32 bit vs. 64 bit OS. In the case of TG, 64 bit architectures offer theoretical benefits, regarding memory use - OTOH, many people these days are still running 32 bit OS.