FlightGear wiki

TerraGear scenery build server: Difference between revisions

Page Discussion

TerraGear scenery build server (view source)

Revision as of 14:37, 5 June 2016

12,621 bytes added ,  5 June 2016
→‎Crowd-compilation
m (reeed's concept schematic)
 
(23 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Template:Mentored Volunteer Effort
{{Template:Mentored Volunteer Effort
|mentors=D-YETI (server/hosting[http://forum.flightgear.org/viewtopic.php?f=5&t=14163&start=15#p200802]), reeed, saiarcot895 (packaging), itismike, Hooray(get in touch to learn more)
|developers=D-YETI (server/hosting[http://forum.flightgear.org/viewtopic.php?f=5&t=14163&start=15#p200802]), reeed, F-JJTH(Web frontend), saiarcot895 (packaging), statto (testing), itismike
|skills=Virtualization, Linux, [[TerraGear]], [[TerraGear GUI]] SSH, Qt, TurnKeyLinux}}
|mentors=Hooray(get in touch to learn more)
|skills=Virtualization, Linux, [[TerraGear]] (psadro_gm & papillon81), [[TerraGear GUI]](Gijs), SSH, Qt, TurnKeyLinux}}
 
{{FGCquote
  |terragear tools are not exactly the most robust :)  Building them can be a challenge, and getting useful output is just as hard.
  |{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=224616#p224616
    |title=<nowiki>Re: Terragear, compilation errors-warnings</nowiki>
    |author=<nowiki>psadro_gm</nowiki>
    |date=<nowiki>Sun Nov 16</nowiki>
  }}
}}
 
{{FGCquote
  |I run genapts850 on linux 64bit with no problem.  I've got a new machine, now and Windows 64 running on a virtual machine, but I haven't done any development work on Windows for 18 years now.  It's really not very high on the priority list, as the world scenery is also generated on linux.
  |{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=167433#p167433
    |title=<nowiki>Re: TerraGear GUI</nowiki>
    |author=<nowiki>psadro_gm</nowiki>
    |date=<nowiki>Wed Sep 26</nowiki>
  }}
}}
 
{{FGCquote
  |Remember, Terragear was never intended to used by an end user.  It was designed to be run on a Linux server.  It's been almost 3 years since I've started looking at this code.  It took almost a full year of hacking on it before I was confident enough to commit changes.<br/>
It is not user friendly.  It was once my intention to make it easier to use and add features.  Over time, it became obvious that this was optimistic.  Bug fixes pretty much takes all of my time.<br/>
<br/>
That's not to say that I can't imagine an easy to use toolchain,  unfortunately, we just don't have the resources to work toward this goal.  If anyone is interested, I could help explain how the tools work, and how the code works.
  |{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=195883#p195883
    |title=<nowiki>Re: Terragear for beginners</nowiki>
    |author=<nowiki>psadro_gm</nowiki>
    |date=<nowiki>Sun Dec 08</nowiki>
  }}
}}
 
 
 
 
{{FGCquote
  |User friendliness, and support for windows, 32 bit, etc have dropped to very low priority, as the server that Martin uses when generating the official scenery is well known to me.<br/>
<br/>
Hence, this is the reason we went to the web page setup.  There was also some ideas for a VM, but that would still require a 64 bit host.
  |{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=224616#p224616
    |title=<nowiki>Re: Terragear, compilation errors-warnings</nowiki>
    |author=<nowiki>psadro_gm</nowiki>
    |date=<nowiki>Sun Nov 16</nowiki>
  }}
}}
 
{{FGCquote
  |If you feel something is poorly designed, and should be done differently, go ahead and try to improve it.  If others can be convinced of the merit, it most likely will become the 'new way'.  So many examples of this in the past 4 years of development.<br/>
<br/>
advanced weather, random buildings, Rembrandt, ALS, canvas and OSGEarth.  These are features that at one point, someone said - I wish FG could do xxx.  Then, they sat down, and figured out HOW FG could do it.
  |{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=199226#p199226
    |title=<nowiki>Re: Why are cockpits so integrated with planes?</nowiki>
    |author=<nowiki>psadro_gm</nowiki>
    |date=<nowiki>Wed Jan 29</nowiki>
  }}
}}
 
 
 


[[File:TerraGear-Build-Server-Workflow.png|950px|TerraGear Build Server Workflow when using [[TerraGear GUI]] over SSH]]
[[File:TerraGear-Build-Server-Workflow.png|950px|TerraGear Build Server Workflow when using [[TerraGear GUI]] over SSH]]
Line 17: Line 76:




{{Template:Non-stable}}
{{Template:Non-stable|version=4.x|progress=40}}


{{infobox subsystem
{{infobox subsystem
Line 35: Line 94:
* Looking for people familiar with [[TerraGear]]
* Looking for people familiar with [[TerraGear]]
* Looking for people who are familiar with Linux, virtualization
* Looking for people who are familiar with Linux, virtualization
= News =
{{See also|TerraGear BOINC}}
As part of working on a long-standing feature request, namely having a central shared [[TerraGear scenery build server]], saiarcot895 has created Linux packages for Debian/Ubuntu (deb/ppa) to install precompiled TerraGear binaries. This now greatly simplifies installing TerraGear, because people no longer need to manually set up a complete Linux build environment and all SimGear/TerraGear dependencies. Everything is now done in an automated fashion.
{{Note|To test the precomiled packages on a debian/wheezy based distro: edit '''/etc/apt/sources.list.d/sources.list''' and add the TG repo provided by saiarcot895:
<pre>
deb http://128.61.105.135/apt/wheezy main
deb http://128.61.105.135/apt/wheezy contrib
deb http://128.61.105.135/apt/wheezy non-free
</pre>
On Ubuntu Precise and Quantal the repo paths can be put in a new list file, create '''/etc/apt/sources.list.d/saiarcot895-terragear.wheezy.list''' and insert the the following. The format needs to be
<pre>
deb http://128.61.105.135/apt wheezy main
deb http://128.61.105.135/apt wheezy contrib
deb http://128.61.105.135/apt wheezy non-free
</pre>
Next, run '''apt-get update''' and then '''apt-get install terragear''' as root:
<pre>
# apt-get update && apt-get install terragear
</pre>
}}
Note:
This only installs the Terragear tool chain and not the GUI or any src. It by default installs it in /usr/bin folder unlike saiarcot895's download_and_install.sh (link may be broken [http://www.ggbhydropower.fr/flightgear/download_and_compile_tg.sh]) which installs everything including the GUI and src, wherever you want to put it.
[[File:TerraGear-hgtchop-VM.png|thumb|TerraGear hgtchop running in a 64bit TurnKeyLinux VM using saiarcot895's debian/wheezy packages]]
Next, we're hoping to use this to install TerraGear on a public server for which people can ask for remote shell access (SSH). People interested in exploring this, should be  getting in touch via the forum or the wiki.
Obviously, users will still need to be familiar with TerraGear, but they may benefit from reduced bandwidth restrictions and/or more horsepower in comparison to running TerraGear locally (e.g. one user offered to contribute hosting on a 32gb RAM and 8-core server). So this could be a great opportunity for people to run scripted/unattended jobs, without having to go through the hassle of downloading/building/installing and configuring TerraGear.
But so far, being familiar with TerraGear and Linux is going to be a prerequisite still.
Once that is working, we'll investigate making the setup reproducible by using TurnKeyLinux. Once we have a working TKL distro, we can install a full TG setup in just a few minutes by downloading an ISO file and installing it in a VM (VMWare/VirtualBox). This would basically allow people to easily download/install TerraGear locally, either installed next to their OS, or as a virtual machine.
[[File:F-jjth-tggui-prototype.jpg|thumb|Web-based TerraGear GUI prototype developed by F-JJTH in 03/2014]]
The long term idea is to hook up [[TerraGear GUI]] to it, so that the GUI front-end talks to TerraGear across SSH.
This is currently still a use-case for which TG wasn't designed for, and the TG developers mentioned already on the forums that there may be some roadblocks ahead, so everything here is still highly experimental. But ultimately we hope to provide a front-end to a Linux-based TerraGear VM, either by reusing [[TerraGear GUI]] or by coming up with a custom web-based front-end (please get in touch if  you can help with this!).
If you're interested in helping or learning more, please get in touch via the forum or via [[TerraGear scenery build server]].


= Problem =
= Problem =
Line 55: Line 157:


Let's pool our know-how and resources together and aim to setup a scenery build server!
Let's pool our know-how and resources together and aim to setup a scenery build server!
{{FGCquote
  |I run windows, and have followed the winding toolchain trail to a brick wall. TerraGear for windows is apparently abandoned, broken, or obsolete.
  |{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=214155#p214155
    |title=<nowiki>Is it possible to create a new airport in Windows?</nowiki>
    |author=<nowiki>Kabuki</nowiki>
    |date=<nowiki>Sat Jul 05</nowiki>
  }}
}}
{{FGCquote
  |What's the best option for me? Install some VM software and learn to use that so I can do fake linux? Or, run Ubuntu only for the sake of TerraGear?  Or maybe create some airport data and hand it off to someone who can use TerraGear?
  |{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=214155#p214155
    |title=<nowiki>Is it possible to create a new airport in Windows?</nowiki>
    |author=<nowiki>Kabuki</nowiki>
    |date=<nowiki>Sat Jul 05</nowiki>
  }}
}}


= Background =
= Background =
Line 70: Line 190:
Scenery compilation is demanding, so setting up a VM on a powerful system with scenery compilation tools would be a good idea. This might come in handy for people whose system is not as powerful and enable them to contribute to  scenery building without requiring a powerful computer.
Scenery compilation is demanding, so setting up a VM on a powerful system with scenery compilation tools would be a good idea. This might come in handy for people whose system is not as powerful and enable them to contribute to  scenery building without requiring a powerful computer.


= Plan / Status =
 
= Plan / Status (03/2014) =
 
The first step would probably be finding someone familiar with installing/setting up TG on Linux, and then also someone familiar with virtualization, so that a Linux distro like TurnkeyLinux can be set up with TerraGear - from that point on, everything should be fairly straightforward actually, assuming that we have someone willing to work on a web-interface -or API- to expose TerraGear as a web service, e.g. using Python, Php, Ruby or Perl - in fact, one could even use Nasal for this  
The first step would probably be finding someone familiar with installing/setting up TG on Linux, and then also someone familiar with virtualization, so that a Linux distro like TurnkeyLinux can be set up with TerraGear - from that point on, everything should be fairly straightforward actually, assuming that we have someone willing to work on a web-interface -or API- to expose TerraGear as a web service, e.g. using Python, Php, Ruby or Perl - in fact, one could even use Nasal for this  


Line 85: Line 207:
* Install SimGear in Headless mode [[Building_using_CMake#SimGear_Build_Options]] {{Done}}
* Install SimGear in Headless mode [[Building_using_CMake#SimGear_Build_Options]] {{Done}}
* Install TerraGear inside the VM using the [[Building FlightGear - Debian#TerraGear]] script {{Done}}
* Install TerraGear inside the VM using the [[Building FlightGear - Debian#TerraGear]] script {{Done}}
* Investigate packaing SimGear headless (saiarcot895) {{Done}}
* Investigate packaging TerraGear as a deb/ppa  (saiarcot895 ) {{Done}}
* Use [[Using TerraGear]] to come up with a test script {{Progressbar|20}}
* Get in touch with Gijs to patch [[TerraGear GUI]] to support operating over SSH {{Not done}}
* Get in touch with Gijs to patch [[TerraGear GUI]] to support operating over SSH {{Not done}}
* Get in touch with people volunteering hosting {{Not done}}
* Get in touch with people volunteering hosting {{Not done}}
Line 105: Line 230:


= Crowd-compilation  =
= Crowd-compilation  =
{{Merge|TerraGear BOINC}}
* A way to distribute work to a bunch of vm scenery bulding servers would be nice. Something people willing to help and with some extra disk space could run on their system in the normal downtime to share computing power to the scenery generating effort.
* A way to distribute work to a bunch of vm scenery bulding servers would be nice. Something people willing to help and with some extra disk space could run on their system in the normal downtime to share computing power to the scenery generating effort.
* Yes, we also talked about this last year, it seems there are some SSH-based solutions available, i.e. they would use standard "scp" to copy stuff via ssh to a remote location. If I remember correctly, we also looked for a Python-based solution so that the TerraGear GUI could be modified accordingly.
* Yes, we also talked about this last year, it seems there are some SSH-based solutions available, i.e. they would use standard "scp" to copy stuff via ssh to a remote location. If I remember correctly, we also looked for a Python-based solution so that the TerraGear GUI could be modified accordingly.
Line 111: Line 237:


== BOINC ==
== BOINC ==
{{Merge|TerraGear BOINC}}
{{DeQuote}}
{{FGCquote
|1= For what it's worth, the primary bottleneck for scenery generation is disk I/O -- reading and writing files.  There is a computational aspect, but on the whole it's disk IO that will limit things.
Back in the day when I was cranking out the world scenery myself, I had a cluster of linux machines running in parallel -- all talking to an nfs server -- I thought it was a pretty slick setup at the time, but I'm sure a fast single modern PC could outperform the cluster of 20-25 PC's I was using at the time.
|2= {{cite web
  | url    = http://forum.flightgear.org/viewtopic.php?p=144405#p144405
  | title  = <nowiki>Re: Create a </nowiki>
  | author = <nowiki>curt</nowiki>
  | date  = Dec 9th, 2011
  | added  = Dec 9th, 2011
  | script_version = 0.23
  }}
}}
{{FGCquote
|1= I understand that generating the scenery from the raw data takes quite some CPU time.
How about creating a [http://boinc.berkeley.edu/ BOINC] project which could harness the idle CPU-power of many users to assist with this job?
|2= {{cite web
  | url    = http://forum.flightgear.org/viewtopic.php?p=144361#p144361
  | title  = <nowiki>Create a BOINC project for scenery generation?</nowiki>
  | author = <nowiki>sgofferj</nowiki>
  | date  = Dec 9th, 2011
  | added  = Dec 9th, 2011
  | script_version = 0.23
  }}
}}
{{FGCquote
|1= The only alternative to do it more quickly, in my opinion, would be using a distributed architecture - e.g. BOINC - or a cluster of servers.)
|2= {{cite web
  | url    = http://forum.flightgear.org/viewtopic.php?p=225099#p225099
  | title  = <nowiki>Re: Howto create the parking positions</nowiki>
  | author = <nowiki>elgaton</nowiki>
  | date  = Nov 21st, 2014
  | added  = Nov 21st, 2014
  | script_version = 0.23
  }}
}}
{{FGCquote
|1= when our main scenery resource will be gone in a couple of weeks, we will have a hard time to create a new world scenery. Generating the entire world was a job that ran continuously for days - if not weeks, even on a powerful machin with almost unlimited resources. As we have to re-think the entire process anyway, I just had the idea to use our users resources for that, probably by using something like BOINC: https://boinc.berkeley.edu/trac/wiki/AppIntro We sure need to do many tweaks within the tool-chain, but I can't think of any reason why this should not work. Does anybody have experience with BOINC or alike?
|2= {{cite web
  | url    = http://sourceforge.net/p/flightgear/mailman/message/34666765/
  | title  = <nowiki>[Flightgear-devel] Continous, Distributed Generation of Terrain</nowiki>
  | author = <nowiki>Torsten Dreyer</nowiki>
  | date  = Dec 3rd, 2015
  | added  = Dec 3rd, 2015
  | script_version = 0.23
  }}
}}
{{FGCquote
|1= we could adopt an even simpler approach:
<ol style="list-style-type: decimal">#  improve the existing tools (and add new ones) to support regenerating a single tile without depending on the surrounding ones;
#  expand the FlightGear Scenery Database webforms, if needed, to allow any FG user to submit improvements that can not be done via the Airport Gateway or OSM;
#  periodically grab all improvements from OSM/the Airport Gateway via their APIs;
#  regenerate the single tiles using a distributed system like BOINC (see [[TerraGear scenery build server|this page on the wiki]]);
#  publish the scenery via HTTP (from a single website or a network of mirrors).
|2= {{cite web
  | url    = http://forum.flightgear.org/viewtopic.php?p=236906#p236906
  | title  = <nowiki>Re: Google code colse there doors</nowiki>
  | author = <nowiki>elgaton</nowiki>
  | date  = Mar 26th, 2015
  | added  = Mar 26th, 2015
  | script_version = 0.23
  }}
}}
{{FGCquote
|1= I thought of a ready to use vm that contains the necessary tools (terragear) + a python script that connects to a server, downloads the needed data / scripts and starts compiling the scenery.
After the task is done the data would be uploaded back to the server.
The server could run a python script too, that manages the connected compile slaves. The script could take the compile slave capabilities (disk io, processors, line bandwidth) into account and assign tasks based on this to the slave.
|2= {{cite web
  | url    = http://forum.flightgear.org/viewtopic.php?p=144459#p144459
  | title  = <nowiki>Re: Create a </nowiki>
  | author = <nowiki>ot-666</nowiki>
  | date  = Dec 10th, 2011
  | added  = Dec 10th, 2011
  | script_version = 0.23
  }}
}}
Also in that case, a BOINC project would help because then for a million IOs there are a million PCs doing each 1 IO at the same time instead of 1 PC queueing up a million IOs ^^.
Also in that case, a BOINC project would help because then for a million IOs there are a million PCs doing each 1 IO at the same time instead of 1 PC queueing up a million IOs ^^.
But automatizing the whole process a little bit could also help. As I mentioned in another thread, I have 10MBit/s sym inet, a fairly powerful server which runs 24/7 and has RAIDs (IO-speed!) and now I upgraded my workstation also. If it's a BOINC project or just an automated script which pulls the stuff and goes through the workflow, I would be happy to contribute to the rendering of the scenery. Just familiarizing myself with geospatial sciences, getting into various GIS programs and doing the process by hand is a dealbreaker for me. I just don't have the time to get into all this in depth, unfortunately.
But automatizing the whole process a little bit could also help. As I mentioned in another thread, I have 10MBit/s sym inet, a fairly powerful server which runs 24/7 and has RAIDs (IO-speed!) and now I upgraded my workstation also. If it's a BOINC project or just an automated script which pulls the stuff and goes through the workflow, I would be happy to contribute to the rendering of the scenery. Just familiarizing myself with geospatial sciences, getting into various GIS programs and doing the process by hand is a dealbreaker for me. I just don't have the time to get into all this in depth, unfortunately.
Retrieved from "https://wiki.flightgear.org/Special:MobileDiff/67669...99299"