View source for FlightGear and OpenGL ES

{{Stub}}

{{infobox subsystem
|image       = 
|name        = OpenGL ES compatible subset of FlightGear
|started     = 04/2020 
|description = identify, patch and build a subset of FlightGear compatible with GLES
|status      = RFC
|developers  = talks
}}

{{Portability Navbar}}
{{See also|Howto:Build and run FlightGear on Raspberry Pi 4}}

Some FG users are very interested in seeing how FlightGear runs on lower end/embedded hardware (think RPi-style) using {{Wikipedia|OpenGL ES}}.

The bigger issue here is we need to ditch [[PUI]] (which is in progress) and some OpenGL 1.0 code (HUD, 2D panels especially - can be #ifdef for now) so we can enable Core profile on Mac - since Mac 4.x support (we only hit about 4.3 alas, but with some extensions to get in sight of 4.5) is Core profile only, no Compatability mode.

I *believe* the new open-source Intel-Mesa drivers on Linux (which are supposed to decent quality, and even fast) might be in the same situation.<ref>https://sourceforge.net/p/flightgear/mailman/message/36341578/</ref>

Probably to do it initially, GLES1 would have to be used - I think GLES2 has no fixed function - and simgear/flightgear would have to be patched to use only OSG calls (if possible), e.g. by excluding features/subsystems that still use legacy OpenGL code incompatible with GLES.

OSG can have GLES{1,2} support without windowing compiled in.

OpenGL 1 and 2 can be mixed but OpenGL ES 1 and OpenGL ES 2 can't. There is a performance price to pay for this backward compatibility. For this reason, we should get rid of PLib because this (old) code is all OpenGL 1 that may slowdown the all rendering pipeline.

For historical reasons, Flightgear and Simgear are mixing OpenGL 1
(old code) and OpenGL 2 (more recent code).<ref>https://sourceforge.net/p/flightgear/mailman/message/36980923/</ref>

== Background ==
{{Main article|Howto:Optimizing FlightGear for mobile devices}}

The OpenSceneGraph port initiated in 2006 has never been fully completed, so that there is a certain amount of code making use of legacy OpenGL calls, which complicates modernizing the renderer. 

In particular, this means that [[Supporting multiple renderers]] is unnecessarily complicated, [[Unifying the 2D rendering backend via canvas|unifying the 2D rendering back-end]] is a long standing challenge. It's only since just very recently, that supporting different renderers is being worked on thanks to the [[Compositor]] effort.

But even then, phasing out legacy code or porting it, still needs to be addressed sooner or later.

Being able to run fgfs on such, comparatively low-powered, systems using OpenGL ES can actually be a good thing for fgfs as a whole - it can help us understand bottlenecks that are hardly visible on typical gaming/developer rigs, but that may still show up over time (think leaking listeners/memory) - this sort of thing can also be considered the prerequisite for people wanting to target/build/run fgfs on other embedded hardware, such as thin clients with integrated GPUs or even mobile phones/tablets (think Android)

we need to remove PUI and change the Canvas not to use Shiva, to be ES2 compatible or Core-profile compatible.<ref>https://sourceforge.net/p/flightgear/mailman/message/36194973/</ref>

In other words, if the right people were to team up to specifically target such hardware, this could also mean significant performance improvements for people on powerful gaming rigs.

== Approach ==
{{See also|Unifying the 2D rendering backend via canvas}}

For starters, we can try the [[FlightGear Headless]] option to build a fgfs version without showing any graphics at all.

The next step will be identifying and excluding problematic sources (those containing legacy/raw OpenGL code, e.g. using <code>glBegin()</code> or <code>glEnable()</code> respectively):

https://sourceforge.net/p/flightgear/flightgear/ci/next/tree/src/Cockpit/render_area_2d.cxx
<syntaxhighlight lang="cpp">
void RenderArea2D::RenderQuad( const SGVec2f *p) {
    glBegin(GL_QUADS);
        glNormal3f(0.0f, 0.0f, 0.0f);
        glVertex2fv( p[0].data() );
        glVertex2fv( p[1].data() );
        glVertex2fv( p[2].data() );
        glVertex2fv( p[3].data() );
    glEnd();
}
</syntaxhighlight>

Whenever FlightGear sources contain such or similar code, it is pretty safe to assume that we will need to port/exclude such modules from compilation to ensure that no legacy OpenGL code is executed at runtime.

<syntaxhighlight>
Instrumentation/HUD/HUD_runway.cxx
Instrumentation/HUD/HUD_tbi.cxx
Instrumentation/HUD/HUD.hxx
Instrumentation/HUD/HUD_instrument.cxx
Instrumentation/HUD/HUD_ladder.cxx
Instrumentation/HUD/HUD_dial.cxx
Instrumentation/HUD/HUD_tape.cxx
Cockpit/render_area_2d.cxx
Cockpit/panel.cxx
GUI/WaypointList.cxx
GUI/CanvasWidget.cxx
GUI/MapWidget.cxx
</syntaxhighlight>

This means, we'll need to review/disable the compilation of the following folders in $FG_SRC and patch up all hard-coded references to these systems:
* Instrumentation/HUD
* Cockpit
* GUI

In addition, there's Viewer/PUICamera.cxx which references glEnable()

== Example: Disabling PUI ==
{{Main article|Developing using CMake}}

We should add separate build options to explicitly disable certain features individually.

The bigger issue here is we need to ditch PUI (which is in progress) and some OpenGL 1.0 code (HUD, 2D panels especially - can be #ifdef for now) so we can enable Core profile on Mac - since Mac 4.x support (we only hit about 4.3 alas, but with some extensions to get in sight of 4.5) is Core profile only, no Compatability mode.<ref>https://sourceforge.net/p/flightgear/mailman/message/36341578/</ref>

Knowing that [[PUI]] contains legacy OpenGL code and knowing it's scheduled to be removed anyway because it isn't compatible with modern OpenGL, we will disable it completely without 
reviewing/porting individual PUI files. This means opening $FG_SRC/CMakeLists.txt to add a new option to disable PUI.


Successfully disabling a feature means primarily:
* adding a corresponding new build option to the top-level CMakeLists.txt (e.g. DISABLE_PUI)
* opening fg_init.cxx and navigating to the lines where the feature/subsystem is initialized
* wrapping the corresponding code in between <code>#ifdef...#endif</code> blocks
* locating any remaining references to the subsystem in question and repeating the last step there to ensure that removed subsystems are not accessed at runtime

https://sourceforge.net/p/flightgear/flightgear/ci/next/tree/CMakeLists.txt
<syntaxhighlight lang="cmake">
option(SYSTEM_CPPUNIT    "Set to ON to build Flightgear with the system's CppUnit library")
option(DISABLE_PUI    "Set to ON to build Flightgear without PUI support")

if(DISABLE_PUI)
      add_definitions(-DDISABLE_PUI)
endif(DISABLE_PUI)

</syntaxhighlight>

We will be adding <code>#ifdef</code> macros to the sources in question, and update CMakeLists.txt accordingly.

Ideally, in conjunction with a feature-specific build option to disable the corresponding feature (think [[PUI]] or the [[HUD]]).

Thus, after editing CMakeLists.txt, we need to open fg_init.cxx to prevent initalization of PUI.


https://sourceforge.net/p/flightgear/flightgear/ci/next/tree/src/Main/fg_init.cxx#l1043
<syntaxhighlight lang="cpp">
    ////////////////////////////////////////////////////////////////////
    // Create and register the XML GUI.
    ////////////////////////////////////////////////////////////////////
#ifndef DISABLE_PUI
    globals->add_subsystem("gui", new NewGUI, SGSubsystemMgr::INIT);
#endif
</syntaxhighlight>


So, after editing fg_init.cxx to prevent the PUI GUI from getting initialized by FlightGear, we will need to find remaining hard-coded references to it, to fix those up and deal with PUI not being available. This means grepping $FG_SRC for any references to <code>"pui"</code> to locate remaining get_subsystem() calls. (FIXME: new subsystem lookups use templates), this will include code in unrelated modules, e.g. fgcommands (think menu bindings) accessing the GUI via something like <code>get_subsystem("gui");</code>

https://sourceforge.net/p/flightgear/flightgear/ci/next/tree/src/Main/fg_scene_commands.cxx#l277
<syntaxhighlight lang="cpp">
    NewGUI * gui = (NewGUI *)globals->get_subsystem("gui");
    if (!gui) {
      return false;
    }

</syntaxhighlight>

Thus, it makes sense to intercept registration of these fgcommands at the bottom of the file by wrapping these inside ifdef macros:

https://sourceforge.net/p/flightgear/flightgear/ci/next/tree/src/Main/fg_scene_commands.cxx#l497
<syntaxhighlight lang="cpp">
/**
 * Table of built-in commands.
 *
 * New commands do not have to be added here; any module in the application
 * can add a new command using globals->get_commands()->addCommand(...).
 */
static struct {
  const char * name;
  SGCommandMgr::command_t command;
} built_ins [] = {
    { "exit", do_exit },
    { "reset", do_reset },
    { "reposition", do_reposition },
    { "switch-aircraft", do_switch_aircraft },
    { "panel-load", do_panel_load },
    { "preferences-load", do_preferences_load },
    { "toggle-fullscreen", do_toggle_fullscreen },
    { "screen-capture", do_screen_capture },
    { "hires-screen-capture", do_hires_screen_capture },
    { "tile-cache-reload", do_tile_cache_reload },
#ifndef DISABLE_PUI
    { "dialog-new", do_dialog_new },
    { "dialog-show", do_dialog_show },
    { "dialog-close", do_dialog_close },
    { "dialog-update", do_dialog_update },
    { "dialog-apply", do_dialog_apply },
    { "open-browser", do_open_browser },
    { "gui-redraw", do_gui_redraw },
#endif
    { "add-model", do_add_model },
    { "presets-commit", do_presets_commit },
    { "press-cockpit-button", do_press_cockpit_button },
    { "release-cockpit-button", do_release_cockpit_button },
    { "dump-scenegraph", do_dump_scene_graph },
    { "dump-terrainbranch", do_dump_terrain_branch },
    { "print-visible-scene", do_print_visible_scene_info },
    { "reload-shaders", do_reload_shaders },
    { "reload-materials", do_materials_reload },
    { "open-launcher", do_open_launcher },
    { 0, 0 }			// zero-terminated
};

</syntaxhighlight>

This will ensure that fgcommands that are PUI related won't be available in a non-PUI build.

We will also need to look for other files accessing the "NewGUI" subsystem <code>grep -nr "NewGUI" -l</code>:
<syntaxhighlight>
Main/subsystemFactory.cxx
Main/fg_scene_commands.cxx
ATC/atcdialog.cxx
ATC/atcdialog.hxx
Autopilot/route_mgr.cxx
</syntaxhighlight>

Since we have excluded $FG_ROOT/GUI from the build and updated fg_init.cxx, we will only need to review/patch these files to ensure that there are no hard-coded references to PUI if it's not available, using the same ifdef based approach.  

Obviously, fgdata level resources like menu bindings and/or Nasal code may still try to execute such bindings.

Finally, there may still be other PUI specific references in the source tree ($FG_SRC), so that it does help to check the other folders next: <code>grep -nr "PUI" -l
</code>

Since we have excluded $FG_SRC/GUI from the compilation, we can safely ignore any references to that folder, with the remaining ones being:
<syntaxhighlight>
Input/FGMouseInput.cxx
Main/fg_os.hxx
Main/bootstrap.cxx
Viewer/renderer_compositor.hxx
Viewer/PUICamera.cxx
Viewer/renderer_legacy.cxx
Viewer/CMakeLists.txt
Viewer/PUICamera.hxx
Viewer/renderer_compositor.cxx
Viewer/renderer_legacy.hxx
Viewer/GraphicsWindowQt5.cpp
Canvas/canvas_mgr.hxx
</syntaxhighlight>

The renderer is basically just referencing the PUI camera so that the GUI can be drawn, whereas canvas_mgr merely acccesses the PUI subsystem to be able to render canvas based textures. In other words, both references can be easily removed for testing purposes.

== Affected features and sources ==

{| class="wikitable"
|-
! Feature !! Directory !! Notes !! Status 
|-
| [[PUI]] || $FG_SRC/GUI || trivial to disable, can be replaced via [[Phi]] || {{Progressbar|70}}
|-
| [[FGPanel|2D Panels]] || $FG_SRC/Cockpit || legacy 2D panels (could use [[Canvas]]), actually as of 04/2020, there is now a dedicated OpenGL ES based version of the standalone fgpanel port available which could be backported (mainly contains hard-coded shader updates) <ref>https://sourceforge.net/p/flightgear/flightgear/ci/dc6fb535962eff8214bc5b1708886178cceb482e/log/?path=/utils/fgpanel</ref> See also merge request {{Merge-request|project=flightgear|id=191|text=#191:HLA update + fgpanel on Raspberry Pi}} and the discussion <ref>https://sourceforge.net/p/flightgear/mailman/message/36978512/</ref>

There's also a [https://sourceforge.net/p/flightgear/flightgear/ci/topics/panel-hacking/~/tree/ branch] (which is a few of years old) contains WIP on implementing 2D panels as canvas. It uses the same loading / updating logic in C++ (for 100% compatibility), but rather than building custom rendering, it builds up a Canvas element hierarchy in C++.<ref>https://sourceforge.net/p/flightgear/mailman/message/36980724/</ref><ref>https://sourceforge.net/p/flightgear/mailman/message/37042351/</ref>
|| {{Pending}}
|-
| [[Canvas Path]] (Shiva) || $SG_SRC/canvas || needs some thinking, probably native OpenVG on RPi <ref>https://github.com/ajstarks/openvg</ref> <ref>http://mindchunk.blogspot.com/2012/09/openvg-on-raspberry-pi.html</ref> or an OpenGL based implementation like nanovg <ref>https://www.raspberrypi.org/forums/viewtopic.php?t=247586#p1516785</ref> || {{Pending}}
|-
| [[Effects]] || $FG_ROOT/Effects || not important for the time being, will include [[Shader]]s and probably involve a custom [[Compositor]] pipeline || {{Pending}}
|-
| [[OSGText Issues]] || ... || If you have a chance to replace OSGText by canvas - do so. OSGText (even with the custom OSG repository) does not play nice with our Effects and Shaders framework and will continue to have issues for all eternity.<ref>https://forum.flightgear.org/viewtopic.php?f=37&t=36287&p=353351&#p353351</ref><ref>https://sourceforge.net/p/flightgear/codetickets/2199/#f702</ref> <ref>https://sourceforge.net/p/flightgear/mailman/message/37042476/</ref>|| {{Progressbar|10}}
|}

In addition, [[SimGear]] needs special treatment, too <ref>https://sourceforge.net/p/flightgear/mailman/message/36978098/</ref>.
Using the same heuristics as before (based on grepping [[$SG_SRC]] for <code>glBegin</code> and <code>glEnable</code>, we end up with the following sources (ignoring [[Canvas Path ]]/shivavg, which is dealt with already above):

<syntaxhighlight>
simgear/scene/tgdb/SGVasiDrawable.cxx:42:    glBegin(GL_POINTS);
simgear/scene/model/shadanim.cxx
</syntaxhighlight>


{| class="wikitable"
|-
! Feature !! Directory !! Notes !! Status 
|-
|}

== Porting ==
{{Wikipedia|OpenGL ES}}

=== Vertex buffer objects ===
{{See also|Uniform Buffer Objects}}

=== Canvas.Path (OpenVG) ===
Will likely need to replace shiva vg with an OpenGL 2.0 based implementation like nanovg <ref>https://github.com/memononen/nanovg</ref>.

== Community talks ==
* [https://forum.flightgear.org/viewtopic.php?f=45&t=36922&p=364937#p364937 Download_and_compile.sh & Raspberry Pi4]
* [https://sourceforge.net/p/flightgear/mailman/search/?q=%22opengl+es%22 OpenGL ES (devel list)]

== References ==
{{Appendix}}

== Related content ==
* [[FlightGear Headless]]
* [[Howto:Optimizing FlightGear for mobile devices]]
* [[Graphics card profiles]]
* [[Canvas and OpenGL ES]]
* [[FlightGear and old Hardware]]
* [[Compositor]]