Talk:Building FlightGear - Cross Compiling

From FlightGear wiki
Jump to navigation Jump to search

Build Errors

is it a good idea to update the page with the build errors encountered and the possible solutions(like for example with OSG) so as to make the status more informative? -- Hamzaalloush (talk) 18:22, 18 May 2015 (EDT)

sure, why not - we should add anything here that may help others at some point, even if it's just for historic reasons, i.e. the project having a history before it gets a visible commit log. Besides openscenegraph.mk still seems to contain a bunch of static-build related build flags as far as I can tell, are you sure that dynamic/shared builds are intended to be supported ?--Hooray (talk) 18:38, 18 May 2015 (EDT)
yes, you have to change the flags to build dynamically, and this is how you finally get shared libraries into /bin, else they get built into /lib with the *.a prefix as static. actually some shared libraries actually do get installed while some require linking with the plugins before encountering error , this is what i intend to work on -- Hamzaalloush (talk) 18:52, 18 May 2015 (EDT)
It is usually a good idea to document problems and how they got solved. If not for else then for to not do them again.
It could also potentially help similar endeavors later on.
Johan G (Talk | contribs) 06:38, 19 May 2015 (EDT)

Required OSG Plugins

can we get a subset of required plugin needed by flightgear? we can simply disable the faulty ones of the time bieng if they are not needed? mxe allows you to apply patches to source from the *.mk file as well for this -- Hamzaalloush (talk) 18:58, 18 May 2015 (EDT)

sounds like a plan, I'd use grep -R and the OSG_USEPLUGIN macro in $SG_SRC and $FG_SRC to see what's important [1]. --Hooray (talk) 19:06, 18 May 2015 (EDT)

Minimizing the needed number of MXE packages

the default 'all-filtered' make target, builds 367 packages in a clean environment, only 107 out of 367 are built using MXE's own dependency walker in the Makefile if 'openscenegraph' is passed as a parameter to 'make'. few packages can be excluded by disabling uneeded OSG plugins, and dependencies of further packages. -- Hamzaalloush (talk) 21:16, 22 May 2015 (EDT)

mxe specific tool chain

as far as I know, mxe should be quite able to only build those dependencies that are needed for FG - i.e. plib/openal/simgear/osg which should in turn pick up their own dependencies automatically. So it's mainly a matter of making sure that dependency resolution is working correctly. If you are really referring to the underlying tool chain and not just dependencies, I am not sure what might be optional ? But I guess it would make sense to come up with 2-3 *.mk files for the main FG dependencies, so that the a corresponding build environment can be automatically created by the mxe installer, and which can be used by the Superbuild afterwards. --Hooray (talk) 13:55, 23 May 2015 (EDT)

as i experienced this works very well(95% of the time, MXE pulls only required packages), but due to some packages that introduce dependency bloat, such as for instance OSG compiles with ffmpeg(we don't need this), so this nice package pulls all the audio tools such as x264, flac encoders etc. we can minimize this in our fork, such that ffmpeg is disabled completley, further elimination of a dozen packages consisting of only encoders built for ffmpeg, i'm sure i can find other examples like this as well. FG main dependancies such as simgear will be included in the 'requirements' for fgfs.mk, so whenever you call 'make fgfs' it builds it. for the Superbuild, i'm thinking an extra automation step is required preferably by a script in $(MXE_ROOT)/tools that copies all the dll's and binaries to the expected Building_using_CMake_-_Windows#Directory_tree. -- Hamzaalloush (talk) 16:40, 23 May 2015 (EDT)
I do agree, I have also seen optional dependencies being "defaulted" to "yes" for the sake of simplicity - while ffmpeg isn't currently needed by SG/FG, a few FG devs (especially those doing UAV stuff) have been contemplating to support ffmpeg-based video streaming in FG at some point (see supporting streaming in FG for details/references), so I guess it would be better to identify such osg/mxe issues (I think some osg examples require ffmpeg), and make the corresponding dependency optional by adding proper dependency resolution, e.g. by using the corresponding OSG magic to detect such dependencies (osg-config, or even just cmake configuration flags). I do however agree that it makes sense to first get a basic build working, and only look at all the other optional stuff later on - we will need some kind of solution to deal with those anyway (think fgcom, profiler, osgEarth, HLA/RTI etc).--Hooray (talk) 13:49, 24 May 2015 (EDT)
if that's the case, i might ignore this for now and keep ffmpeg, unless it gives error and then i'll proceed with a basic build. in static-mingw, until now only one OSG plugin has been making problems which was when building 'gdal', which i assume necessary for osg-earth, I’ll write the specifics in the Issues/OSG topic when i'm done so we can resolve it later. -- Hamzaalloush (talk) 19:35, 24 May 2015 (EDT)
true, gdal is needed for osgEarth, but would also be useful for supporting mxe-based building of TerraGear --Hooray (talk) 06:55, 28 May 2015 (EDT)

3rd party deps/directory structure

I still need to check the Superbuild/CMakeLists.txt files, but I assume that those assumptions are not Windows specific, but primarily MSVC specific - so may not even be applicable to a gcc-based cross-compiler ? --Hooray (talk) 13:54, 24 May 2015 (EDT)

yes your right! :-) after i checked, this was to resolve MSVC linking to 3rd parties. that dir structure might not even be needed, still, i think it is a good idea to at least compile the respective versions of those 3rd party dependencies as provided in the fgmeta/3rdparty branch. -- Hamzaalloush (talk) 19:30, 24 May 2015 (EDT)
it definitely is a good idea, but looking at the two CMakeLists.txt in SG/FG, those 3rd party deps are there mainly for those systems not providing "system-wide" versions of those - so we could just as well review existing mxe *.mk files and directly add support for the equivalent libs to mxe, to ensure that there both options are provided. As far as I understand, system-libs would be chosen by default on *nix platforms anyway, so we could just as well ignore 3rd party deps when doing cross-compilation, by simply using the corresponding mxe equivalents for those. Overall, I think that "3rdparty.zip" is a workaround addressing shortcomings in the Windows build process - so if it is easier to directly build the proper packages as part of mxe, I would consider doing that instead, as it feels more "natural" to me. --Hooray (talk) 15:33, 28 May 2015 (EDT)
certainly this was my thought process recently too. as FG is known to compile against development version of those. so i will leave them as is, and probably even remove that objective from the roadmap. it makes sense to continue unless we have problems. -- Hamzaalloush (talk) 16:28, 28 May 2015 (EDT)

Testing & Development

it would be helpful if there was automation step involved, to routinely cross-compile crucial MXE element for current status, as for instance when certain MXE package "$(PKG)_URL" elements becomes unavailable upstream. sort of like a Jenkins implementation. -- Hamzaalloush (talk) 05:29, 27 May 2015 (EDT)

I've been in touch with a few contributors - unless we can get the FlightGear Build Server maintainers involved, we do have at least one volunteer to provide hosting on a VPS, where mxe and jenkins could be set up, with full root access provided as well. Alternatively, the OpenSuse Build Service (OBS) does provide Jenkins integration, too: https://en.opensuse.org/openSUSE:OpenStack_and_Crowbar_development_process
Given all the recent progress, and the increasing number of contributors wanting to help with different aspects of this effort, I do think that it would make sense to either register an OBS account, and/or, sign up for gcc compile farm access: https://gcc.gnu.org/wiki/CompileFarm - any preferences ? --Hooray (talk) 15:30, 28 May 2015 (EDT)
no i do not. which infrastructure is more easier to use? i think i can leave this decision in the hands of someone with more experience than me, so please feel free to suggest any ideas or someone to consult/contribute? -- Hamzaalloush (talk) 16:27, 28 May 2015 (EDT)
The "gcc compile farm" is just a bunch of dedicated servers reachable via SSH where you can download/build and run stuff, so fairly easy to use for people familiar with Linux - the OBS is a different beast, and pretty sophisticiated, too - i.e. it can even be used to automatically package up software for different distros automatically (think deb, ppa, rpm etc), and it comes with tons of support and documentation, too - while also being possible to integrate with Jenkins based CI servers and a few IDEs. --Hooray (talk) 16:50, 28 May 2015 (EDT)

simgear.mk/fgfs.mk

i think i can prototype something right now for this, should i parallel with (FlyHigh -saiarcot895? what is his status right now? -- Hamzaalloush (talk) 16:35, 28 May 2015 (EDT)

having a prototype (for SimGear headless) would make sense in my opinion, and it coud be used as a template for other (non-headless) stuff, including both, SG & FG as a whole, but also other dependencies (think openrti). So given that SimGear is the only common dependency here, I would suggest to prototype a well-commented example and get this committed, so that others can take a look and learn from it, e.g. to help refine SG/FG support and come up with modules for optional dependencies. The Superbuild would be a different beast anyway, because it is trying to do everything (including the source check outs), so that would require some thinking - and having experience with dedicated simgear.mk and flightgear.mk files should come in very handy for that.--Hooray (talk) 16:47, 28 May 2015 (EDT)
I have a preliminary simgear.mk that has most of the dependencies and the building, but haven't reached testing stage yet because I'm having trouble compiling netcdf. -- Flyhigh (talk) 21:47, 28 May 2015 (EDT)
Well, I found at least half of the problem: portablexdr has an empty build command for x86_64-w64-mingw32, which I was building in. Also, I've uploaded the simgear.mk to here -- Flyhigh (talk) 22:21, 28 May 2015 (EDT)
yeah, some clean-up required for these x86_64-w64-mingw32 entries there. i suggest removing them as i did removing $(PKG)_BUILD_SHARED complety whenever i get a [no-build] result for shared package. nice work on getting the preliminary simgear.mk up, i'll test with it, but let me know when you are finished working that x86_64-w64-mingw32 toolchain to produce openscenegraph, please commit your progress for each package fixed. also, if you want i can merge that simgear.mk commit into my fork, or you do a pull request? whichever you prefer. i'll be following your repo as a remote. -- Hamzaalloush (talk) 04:22, 29 May 2015 (EDT)


Superbuild

This isn't relevant at the moment, but referring to the Superbuild related comments above: it seems we could end up with a viable workaround by figuring out if/how and where those *.mk files are getting the value for -DCMAKE_TOOLCHAIN_FILE='$(CMAKE_TOOLCHAIN_FILE)' from - i.e. CMAKE_TOOLCHAIN_FILE specifically, and directly reuse that in the Superbuild (need to check if that'd work). But if it's too much of a hassle, let's just focus on *.mk files for starters. --Hooray (talk) 17:06, 28 May 2015 (EDT)

sure, it's stored in mxe/usr/${triplet-name}/share/cmake/mxe-conf.cmake, i'm sorry if this is not the place to include here, but here is it's contents anyway, autotools has similar implementation with $(MXE_CONFIGURE_OPTS), for autotools that is contained in the Makefile IIRC. mxe-conf.cmake: -- Hamzaalloush (talk) 17:14, 28 May 2015 (EDT)
set(CMAKE_SYSTEM_NAME Windows)
set(MSYS 1)
set(BUILD_SHARED_LIBS ON)
set(LIBTYPE SHARED)
set(CMAKE_BUILD_TYPE Release)
set(CMAKE_FIND_ROOT_PATH /home/hamza/build/hamza-mxe/usr/i686-w64-mingw32.shared)
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_C_COMPILER /home/hamza/build/hamza-mxe/usr/bin/i686-w64-mingw32.shared-gcc)
set(CMAKE_CXX_COMPILER /home/hamza/build/hamza-mxe/usr/bin/i686-w64-mingw32.shared-g++)
set(CMAKE_Fortran_COMPILER /home/hamza/build/hamza-mxe/usr/bin/i686-w64-mingw32.shared-gfortran)
set(CMAKE_RC_COMPILER /home/hamza/build/hamza-mxe/usr/bin/i686-w64-mingw32.shared-windres)
set(CMAKE_MODULE_PATH "/home/hamza/build/hamza-mxe/src/cmake" ${CMAKE_MODULE_PATH}) # For mxe FindPackage scripts
set(CMAKE_INSTALL_PREFIX /home/hamza/build/hamza-mxe/usr/i686-w64-mingw32.shared CACHE PATH "Installation Prefix")
set(CMAKE_BUILD_TYPE Release CACHE STRING "Debug|Release|RelWithDebInfo|MinSizeRel")
set(CMAKE_CROSS_COMPILING ON) # Workaround for http://www.cmake.org/Bug/view.php?id=14075
set(CMAKE_RC_COMPILE_OBJECT "<CMAKE_RC_COMPILER> -O coff <FLAGS> <DEFINES> -o <OBJECT> <SOURCE>") # Workaround for buggy windres rules
set(PKG_CONFIG_EXECUTABLE /home/hamza/build/hamza-mxe/usr/bin/i686-w64-mingw32.shared-pkg-config)
set(HDF5_C_COMPILER_EXECUTABLE /home/hamza/build/hamza-mxe/usr/bin/i686-w64-mingw32.shared-h5cc)
set(HDF5_CXX_COMPILER_EXECUTABLE /home/hamza/build/hamza-mxe/usr/bin/i686-w64-mingw32.shared-h5c++)
set(Boost_THREADAPI "win32")
set(QT_QMAKE_EXECUTABLE /home/hamza/build/hamza-mxe/usr/i686-w64-mingw32.shared/qt/bin/qmake)
thanks, you spared me quite a bit of searching here - I think this should already be a good way to play with getting the Superbuild to work at some point, without it necessarily having to invoke/involve mxe *.mk files. Basically, we would need the equivalent of some kind of mxe-config script that provides the corresponding path for such keys (e.g. the CMAKE_TOOLCHAIN_FILE) so that this can be reused elsewhere, without having to hard-code any paths.--Hooray (talk) 17:19, 28 May 2015 (EDT)

.deb packaging

what should the .deb package consist of? a freeze of a successful build with all it's packages? it's totaling 3 GB's right now on my system. -- Hamzaalloush (talk) 16:35, 28 May 2015 (EDT)

right, I guess we do have different target audiences: those just wanting to cross-compile FG (without all the setup overhead) would definitely benefit most from everything pre-compiled included, so I would suggest to keep things simple for now and just turn the whole shebang into a deb/ppa package to ensure that FG (including all deps) can be built. I can also help and turn this into a virtual appliance for VirtualBox (possibly including a working fg source tree) But I think we only need to look into this once the other items can be scratched off from the roadmap - i.e. once SG/FG actually build & link correctly. In the meantime, it should probably be safe to assume that all parties interested in this, are able to get/set up and use mxe from your github branch ? --Hooray (talk) 16:44, 28 May 2015 (EDT)
yup, you can clone this from my branch and get it going with only 'git clone -b master-oldgcc https://github.com/hamzaalloush/mxe-clone.git' && 'make openscenegraph', but i do not guarantee all packages will be working on non-Ubuntu system. i have a back-up rig that has Debian Wheezy in it, and was unable to compile with it. -- Hamzaalloush (talk) 16:50, 28 May 2015 (EDT)

OSG Status

Given that OSG now builds correctly, I would suggest to come up with a dedicated osgearth.mk file for testing purposes, which we can also use as a template for SG/FG flightgear-osgearth.mk afterwards, while also preparing support for the osgearth branch poweroftwo has been maintaining. In addition, osgearth comes with a bunch of osgearth applications that should be useful for testing purposes (OSG vs. osgearth support, both being required for linking) --Hooray (talk) 00:32, 29 May 2015 (EDT)

good post-coffee afternoon :) sure, i'll look into that since Flyhigh is unto the sf/fg mk files for now. -- Hamzaalloush (talk) 04:25, 29 May 2015 (EDT)