Talk:TerraGear scenery build server

From FlightGear wiki
Jump to navigation Jump to search

Server & Hosting (D-YETI)

By the way I personally would give some computing time and IO /Diskspace, at the server listed below, too, because I hope much more pople will benefit from it than running it for me alone. Since Terragear is not so easy to install and even with the gui doesn't always give the result I was expecting.

  • Intel® Core™ i5-4570 CPU @ 3.20GHz × 4
  • 32GB RAM
  • SSD + RAID
  • but rather slow Internet connection. (600kbit/s dn / 128kbit/s up)

Potential TerraGear Issues (psadro_gm)

An issue you will may hit is that some terragear tools don't behave very well when run at the same time (genapt and ogrdecode, in particular). this may work with two separate work directories, but I have not tried it. If using the same work directory, You can hit some directory / file creation collisions. To recitify these, I added a Boost global mutex so one can write at a time. With multiple users, this may become a bottleneck.

Packaging (saiarcot895 )

Hooray:Obviously, one major part of this would involve installing/packaging SimGear and TerraGear as *.deb files. This would greatly simplify the installation - we do have several instructions related to this, and even a script - so we could change the script to build SG/TG and then package things afterwards.

Ideally, SimGear would be built in "headless" mode - i.e. minus all the OSG dependencies, no sound etc required: http://wiki.flightgear.org/Building_using_CMake#SimGear_Build_Options

So we would first of all end up with something like "libsimgear-headless-dev" which could be used to build TerraGear. The next step would be creating a package for TerraGear itself that depends on SimGear-headless-dev

Once these two steps are completed, setting up TerraGear on Linux should become much more straightforward. And we could use these packages to set up a TerraGear build server.

All this will be based on TurnKeyLinux, which is a debian/wheezy based distro, intended for "remastering" - so that new virtual appliances can be easily created, and exported - so that others can install an ISO file to get a working TG setup.

For now, I simply built SG headless (libsimgearcore) and installed it system-wide and it's working properly.

But obviously it would be better to come up with packages for both, SG and TG (64bit). Which would be useful for all debian/ubuntu users, not just this particular effort.

Also, we would need to publish those packages, so that people only need to add the repo to download updates automatically.

Overall, this would help greatly simplify TG deployment obviously, no matter if it's about having a build server or not.

Looking at SG/FG, the best option seems to be to use cmake's cpack support to create packages: http://www.cmake.org/Wiki/CMake:CPackPackageGenerators

saiarcot895: I think I can help here.
Part of the work may have been already done. We are distributing simgear in two packages (since there are two libraries): libsimgearcore and libsimgearscene. libsimgearcore doesn't have any graphical dependencies; it only depends on libc6. libexpat, libgcc1, libstdc++6, and zlib1g. The only drawback here is that the -dev package includes headers for both libraries, which would require both libraries be installed. Would libsimgearcore be equivalent to the library produced by SIMGEAR_HEADLESS?
saiarcot895: I just uploaded the terragear package to my flightgear-edge PPA. It should be built and available in about 20-30 minutes for users of Ubuntu Precise, Quantal, and Trusty, but not Saucy (someone decided to place the Boost libraries on Saucy at a weird place, and so I get to find a fix to that). If you use Debian, I think you can download the debs for the Quantal version and use that.
Hooray: There seems to be a dependency issue regarding libtiff according to [1] ? --Hooray (talk) 19:15, 15 February 2014 (UTC)
saiarcot895: Already on it. I got another dependency issue with libgdal (apparently, they changed the development package name right after Precise). Flyhigh (talk) 21:10, 15 February 2014 (UTC)
Hooray: Would it be a lot of work to set up launchpad such that it also provides a debian wheezy (64bit) package ? That's what I am currently using - otherwise, I'd need to change some things here - so it depends on the amount of work involved obviously. Thanks for looking into it! --Hooray (talk) 04:09, 16 February 2014 (UTC)
saiarcot895: Sorry for the delay. The Launchpad PPAs are Ubuntu-only, but I can easily create deb files for Wheezy. It's just a matter of publishing it somewhere. Also, I might have a patch to send regarding having an explicit #include <std::string> line in a file. Something is different in Precise that an explicit #include is necessary. Also, I finally fixed the annoying bug in Saucy; it was actually a bug in CGAL having the incorrect Boost location in its config file. Flyhigh (talk) 16:14, 8 March 2014 (UTC)
saiarcot895: I've built simgear and terragear for the unstable/sid repo, and the packages are available at http://128.61.105.135/apt. To get it down to wheezy, however, I'll have to bump up (at least) OpenSceneGraph from unstable because wheezy still has 3.0. Flyhigh (talk) 15:30, 9 March 2014 (UTC)
Hooray: Sorry for not getting back to this earlier, thanks for the updates and all your work here. You raised a few good points - I am wondering if we can avoid the OSG issue in this particular instance for now, because we really only require the "headless" part of SimGear for TG, i.e. OSG should not even be required normally, right ? Otherwise, you're right that it would make sense to also provide deb packages for wheezy. But just for TG, I think, we don't need to touch any OSG related stuff ? --Hooray (talk) 16:13, 9 March 2014 (UTC)
saiarcot895: You're right, I forgot about that. I've built simgear (in headless mode) for unstable and will build it for testing (jessie) and stable (wheezy), and will build terragear for all three releases once simgear is done. This should be done in a few hours. Flyhigh (talk) 21:36, 10 March 2014 (UTC)
saiarcot895: Simgear for all releases is done, and Terragear for unstable is done. However, Terragear on testing failed with an error in a boost include saying uintptr_t wasn't found...which doesn't make sense to me since boost is from the main Debian archives. Flyhigh (talk) 00:28, 11 March 2014 (UTC)
Hooray: Thank you, I will be testing this shortly and have forwarded instructions to D-YETI who offered to provide hosting for this.--Hooray (talk) 17:32, 11 March 2014 (UTC)

Ok, I added your URL to apt/source.lst on Deb Wheezy (64 bit) and ran apt-get update, when doing "apt-cache search" (simgear/terragear) your repo shows up correctly with the SG packages:

  • core
  • dev
  • dbg

And installation worked without problems, however terragear does not currently show up as a separate package yet, and also isn't listed in the files on the server apparently ? --Hooray (talk) 20:02, 11 March 2014 (UTC)

saiarcot895: Refresh it and check it now. It should be there. Flyhigh (talk) 20:37, 11 March 2014 (UTC)
Hooray: Confirmed, this works now properly - just took 3 seconds to install TG here now on a fresh system, surprisingly not even SG was required (static build?) Very good job indeed! Now, we'll need to document this and spread the good news (article/newsletter). I'll also try to get in touch with D-YETI to see if he still can contribute hosting, which would also be a good way to host the actual TG packages. I guess we could ask Curt to set up a subdomain like packages.flightgear.org to host deb/ppa packages there for i386/amd64. Did you prepare those packages manually, or where you able to use cmake's cpack support for this ? I am asking because I guess it would be a good idea to strive towards using cmake/cpack for this, so that package creation can be automatically supported for SG/FG/TG for each upcoming release, without involving manual work - possibly by running things via the build server (nightly snapshots/prereleases).--Hooray (talk) 20:58, 11 March 2014 (UTC)
saiarcot895: Strange; it has a dependency on libsimgearcore3.1.0, and it's not a static build. Ah well; if it works, you're fine. I did a mix of both manual stuff and scripts, but updates can be fully scripted (which is what I'm doing for my edge PPA). I used scripts from the debhelper package to generate the source package (I think you can get the source package and the debian folder using "sudo apt-get source terragear"), and then used sbuild to recreate a Debian Wheezy chroot and build it in that chroot. It looks like CPack asks for much of the same things as the debian/control file. Also, I'm not sure if CPack generates the .dsc and .changes file, which is necessary for publishing it to a repo. Flyhigh (talk) 21:33, 11 March 2014 (UTC)

saiarcot895: I'm getting a build failure with TerraGear git commit 25294ee and simgear git commit 1948198.

/build/buildd/terragear-2.1.0~1200+git25294ee/src/Airports/GenAirports850/airport.cxx:1144:9: error: 'class SGBinObject' has no member named 'set_pts_v'

/build/buildd/terragear-2.1.0~1200+git25294ee/src/Airports/GenAirports850/airport.cxx:1145:9: error: 'class SGBinObject' has no member named 'set_pts_n'

/build/buildd/terragear-2.1.0~1200+git25294ee/src/Airports/GenAirports850/airport.cxx:1146:9: error: 'class SGBinObject' has no member named 'set_pt_materials'

/build/buildd/terragear-2.1.0~1200+git25294ee/src/Airports/GenAirports850/airport.cxx:1147:9: error: 'class SGBinObject' has no member named 'set_tris_v'

/build/buildd/terragear-2.1.0~1200+git25294ee/src/Airports/GenAirports850/airport.cxx:1148:9: error: 'class SGBinObject' has no member named 'set_tris_n'

/build/buildd/terragear-2.1.0~1200+git25294ee/src/Airports/GenAirports850/airport.cxx:1149:9: error: 'class SGBinObject' has no member named 'set_tris_tc'

/build/buildd/terragear-2.1.0~1200+git25294ee/src/Airports/GenAirports850/airport.cxx:1150:9: error: 'class SGBinObject' has no member named 'set_tri_materials'

/build/buildd/terragear-2.1.0~1200+git25294ee/src/Airports/GenAirports850/airport.cxx:1151:9: error: 'class SGBinObject' has no member named 'set_strips_v'

/build/buildd/terragear-2.1.0~1200+git25294ee/src/Airports/GenAirports850/airport.cxx:1152:9: error: 'class SGBinObject' has no member named 'set_strips_n'

/build/buildd/terragear-2.1.0~1200+git25294ee/src/Airports/GenAirports850/airport.cxx:1153:9: error: 'class SGBinObject' has no member named 'set_strips_tc'

/build/buildd/terragear-2.1.0~1200+git25294ee/src/Airports/GenAirports850/airport.cxx:1154:9: error: 'class SGBinObject' has no member named 'set_strip_materials'

/build/buildd/terragear-2.1.0~1200+git25294ee/src/Airports/GenAirports850/airport.cxx:1155:9: error: 'class SGBinObject' has no member named 'set_fans_v'

/build/buildd/terragear-2.1.0~1200+git25294ee/src/Airports/GenAirports850/airport.cxx:1156:9: error: 'class SGBinObject' has no member named 'set_fans_n'

/build/buildd/terragear-2.1.0~1200+git25294ee/src/Airports/GenAirports850/airport.cxx:1157:9: error: 'class SGBinObject' has no member named 'set_fans_tc'

/build/buildd/terragear-2.1.0~1200+git25294ee/src/Airports/GenAirports850/airport.cxx:1158:9: error: 'class SGBinObject' has no member named 'set_fan_materials'

Does someone need to update the code here? (cause of error is SG commit 5b2b420) Flyhigh (talk) 01:27, 8 April 2014 (UTC)

Please use git branch scenery/ws2.0 for building packages - it is the current stable branch. We are working on new features that need some simgear coordination in master. This work is temporarily stalled for Simgear feature freeze. --Psadro gm (talk) 13:03, 7 July 2014 (UTC)

Thank you for the update, Psadro gm. Currently, I'm bringing up-to-date both simgear and terragear. The latest simgear is in sid and jessie (testing), and terragear will be going in soon. I'll need to build a few additional packages in sid (stable), since OpenSceneGraph 3.2 isn't in sid. Flyhigh (talk) 02:20, 9 July 2014 (UTC)

Just keep in mind: as per our discussion above, TG/SG can be built in "headless" mode and shouldn't require OSG necessarily. Thanks again for your efforts here, really appreciated ! --Hooray (talk) 02:55, 9 July 2014 (UTC)

TerraGear & TerraGear GUI (reeed)

this damned build server idea doesn't want to go away!

very quickly now, to get me up to speed:

  • is a hosting server available for our use right away? or who are the probable candidate(s)?
  • who maintains the latest, compileable and functional TerraGear?
  • Is the above TG packaged with Gijs' GUI? If not, which one is packaged?

TurnkeyLinux (Hooray)

WebService / UI (F-JJTH, Gijs, Hooray, reeed )

40}% completed (proof-of-concept by F-JJTH)

Gijs:It'd be nicer/better to have a web interface. That'll allow you to keep the interface and backend in sync (without asking people to recompile/redownload the GUI om each change), save people from downloading yet another tool and make it work/look the same on all systems.

Hooray: As I mentioned on the forum, I am not at all opposed to having a webservice, I just consider it much more straightforward to modify TerraGear GUI for now. if there's anybody interested in working on a web interface that would obviously be great, but short of that, I would simply modify TerraGear GUI to support two modes local (default) and remote (build server) over SSH. I already looked at the code briefly and the main issue is the amount of Qt code that runs locally, i.e. to download stuff for example - if that could be moved out into external programs, preferably part of TerraGear itself, then it would be trivial to support both, local AND remote TG installations. Currently, there's just too much stuff directly done, without calling external tools - so that would need to be factored out.
Overall, the TerraGear GUI is not such a complex code base, so I am not sure if keeping the frontend/backends in sync is such a huge issue ?? I mean, people ALREADY need to have a matching TG/TGGUI versions after all ...
I am certainly not going to stop anybody from coming up with a webservice - it's just something that I am probably not going to work on anytime soon - while modifying TGGUI accordingly would just take a few hours and involve well-understood steps.
So I wasn't thinking of forking TGGUI, but more about extending it to support local AND remote installations - selected/configured during startup.
Obviously you are much more familiar with the code in question here - but based on what I've seen, it would not be such a difficult undertaking to turn TGGUI into a "remote control" for a TG build server setup.
If frontend changes are such an issue, I would actually suggest to get rid of the hardcoded GUI and use Qt's scripting support for those *.ui files instead - that would allow you to automatically download QtScript while booting.
Finally, coming up with a working web service is in my opinion more difficult than using SSH to teach TGGUI to run a handful of commands remotely and fetch the results via SFTP - so yeah, I am kinda lazy here

Standalone GUI (via SSH)

If the consensus should end up being not to use TerraGear GUI, the most straightforward option for coming up with a simple GUI would be Python using SSH and GUI bindings:

Terragear Build Issues

(I decided to create a new section instead of continuing in an off-topic section)

I found another build error, although this might be due to changes in upstream. Here's the build log: 

cd "/«BUILDDIR»/terragear-2.1.0~1202+git47db776/build/src/Prep/DemChop" && /usr/bin/cmake -E cmake_link_script CMakeFiles/hgtchop.dir/link.txt --verbose=1
/usr/bin/c++   -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2  -frounding-math -Wall  -D_REENTRANT -DNO_OPENSCENEGRAPH_INTERFACE -O3 -DNDEBUG   -Wl,-z,relro -Wl,--as-needed  CMakeFiles/hgtchop.dir/hgtchop.cxx.o  -o hgtchop -rdynamic -lmpfr -lgmp -lCGAL_Core -lCGAL -lboost_thread -lboost_system -lpthread ../../Lib/HGT/libHGT.a -lz -lSimGearCore -lpthread -lz -lrt -lmpfr -lgmp -lCGAL_Core -lCGAL -lboost_thread -lboost_system -lpthread -lSimGearCore -lpthread -lrt 
In file included from /«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_contour.hxx:12:0,
                 from /«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_polygon.hxx:64,
                 from /«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_chopper.hxx:3,
                 from /«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_chopper.cxx:9:
/«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_unique_geod.hxx: In constructor 'SGGeodIndex::SGGeodIndex(SGGeod)':
/«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_unique_geod.hxx:44:27: warning: large integer implicitly truncated to unsigned type [-Woverflow]
                 FNV_prime =    1099511628211ULL;
                           ^
/«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_unique_geod.hxx:45:30: warning: large integer implicitly truncated to unsigned type [-Woverflow]
                 offset_basis = 14695981039346656037ULL;
                              ^
make[3]: Leaving directory '/«BUILDDIR»/terragear-2.1.0~1202+git47db776/build'
/usr/bin/cmake -E cmake_progress_report "/«BUILDDIR»/terragear-2.1.0~1202+git47db776/build/CMakeFiles"  33
[ 25%] Built target hgtchop
/usr/bin/cmake -E cmake_progress_report "/«BUILDDIR»/terragear-2.1.0~1202+git47db776/build/CMakeFiles" 46
[ 26%] Building CXX object src/Lib/terragear/CMakeFiles/terragear.dir/tg_contour.cxx.o
cd "/«BUILDDIR»/terragear-2.1.0~1202+git47db776/build/src/Lib/terragear" && /usr/bin/c++   -DCGAL_USE_GMP -DCGAL_USE_MPFR -DHAVE_CONFIG_H -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2  -frounding-math -Wall  -D_REENTRANT -DNO_OPENSCENEGRAPH_INTERFACE -O3 -DNDEBUG -isystem /usr/include/i386-linux-gnu -I"/«BUILDDIR»/terragear-2.1.0~1202+git47db776/build" -I"/«BUILDDIR»/terragear-2.1.0~1202+git47db776/src" -I"/«BUILDDIR»/terragear-2.1.0~1202+git47db776/build/src" -I"/«BUILDDIR»/terragear-2.1.0~1202+git47db776/build/src/Include" -I"/«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib" -I/usr/include/gdal    -o CMakeFiles/terragear.dir/tg_contour.cxx.o -c "/«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_contour.cxx"
/usr/bin/cmake -E cmake_progress_report "/«BUILDDIR»/terragear-2.1.0~1202+git47db776/build/CMakeFiles" 47
[ 28%] Building CXX object src/Lib/terragear/CMakeFiles/terragear.dir/tg_misc.cxx.o
cd "/«BUILDDIR»/terragear-2.1.0~1202+git47db776/build/src/Lib/terragear" && /usr/bin/c++   -DCGAL_USE_GMP -DCGAL_USE_MPFR -DHAVE_CONFIG_H -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2  -frounding-math -Wall  -D_REENTRANT -DNO_OPENSCENEGRAPH_INTERFACE -O3 -DNDEBUG -isystem /usr/include/i386-linux-gnu -I"/«BUILDDIR»/terragear-2.1.0~1202+git47db776/build" -I"/«BUILDDIR»/terragear-2.1.0~1202+git47db776/src" -I"/«BUILDDIR»/terragear-2.1.0~1202+git47db776/build/src" -I"/«BUILDDIR»/terragear-2.1.0~1202+git47db776/build/src/Include" -I"/«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib" -I/usr/include/gdal    -o CMakeFiles/terragear.dir/tg_misc.cxx.o -c "/«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_misc.cxx"
/usr/bin/cmake -E cmake_progress_report "/«BUILDDIR»/terragear-2.1.0~1202+git47db776/build/CMakeFiles" 48
[ 29%] Building CXX object src/Lib/terragear/CMakeFiles/terragear.dir/tg_nodes.cxx.o
cd "/«BUILDDIR»/terragear-2.1.0~1202+git47db776/build/src/Lib/terragear" && /usr/bin/c++   -DCGAL_USE_GMP -DCGAL_USE_MPFR -DHAVE_CONFIG_H -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2  -frounding-math -Wall  -D_REENTRANT -DNO_OPENSCENEGRAPH_INTERFACE -O3 -DNDEBUG -isystem /usr/include/i386-linux-gnu -I"/«BUILDDIR»/terragear-2.1.0~1202+git47db776/build" -I"/«BUILDDIR»/terragear-2.1.0~1202+git47db776/src" -I"/«BUILDDIR»/terragear-2.1.0~1202+git47db776/build/src" -I"/«BUILDDIR»/terragear-2.1.0~1202+git47db776/build/src/Include" -I"/«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib" -I/usr/include/gdal    -o CMakeFiles/terragear.dir/tg_nodes.cxx.o -c "/«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_nodes.cxx"
In file included from /«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_contour.hxx:12:0,
                 from /«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_polygon.hxx:64,
                 from /«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_accumulator.hxx:4,
                 from /«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_contour.cxx:6:
/«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_unique_geod.hxx: In constructor 'SGGeodIndex::SGGeodIndex(SGGeod)':
/«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_unique_geod.hxx:44:27: warning: large integer implicitly truncated to unsigned type [-Woverflow]
                 FNV_prime =    1099511628211ULL;
                           ^
/«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_unique_geod.hxx:45:30: warning: large integer implicitly truncated to unsigned type [-Woverflow]
                 offset_basis = 14695981039346656037ULL;
                              ^
In file included from /usr/include/CGAL/assertions.h:36:0,
                 from /usr/include/CGAL/basic.h:42,
                 from /usr/include/CGAL/Cartesian/Cartesian_base.h:28,
                 from /usr/include/CGAL/Simple_cartesian.h:28,
                 from /«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_nodes.hxx:10,
                 from /«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_nodes.cxx:2:
/usr/include/CGAL/Search_traits_adapter.h: In instantiation of 'class CGAL::Search_traits_adapter<boost::tuples::tuple<CGAL::Point_2<CGAL::Simple_cartesian<double> >, double>, My_point_property_map, CGAL::Search_traits_2<CGAL::Simple_cartesian<double> > >':
/«BUILDDIR»/terragear-2.1.0~1202+git47db776/src/Lib/terragear/tg_nodes.hxx:35:29:   required from here
/usr/include/CGAL/Search_traits_adapter.h:75:3: error: invalid application of 'sizeof' to incomplete type 'boost::STATIC_ASSERTION_FAILURE<false>'
   BOOST_STATIC_ASSERT( ( boost::is_same< boost::lvalue_property_map_tag,

The version of CGAL is 4.4-1+b1, and the version of Boost is 1.55.0+dfsg-2. This actually might be an incompatibility in CGAL with Boost.

I'm also having this issue with CGAL 4.4 from thee Debian repositories and the official CGAL website. I tried downloading and installing CGAL 4.3 and everything worked as expected. It seems it is an incompatibility of CGAL 4.4 with Boost 1.55 --Ludomo (talk) 12:00, 27 July 2014 (UTC)

Retrieved from "https://wiki.flightgear.org/w/index.php?title=Talk:TerraGear_scenery_build_server&oldid=74227"