View source for Compositor

{{Template:Non-stable|version=2019.3|progress=80}}


{{infobox subsystem
|image       = Clouds experiment.jpg
|name        = Compositor Subsystem
|started     = 01/2018
|description = Dynamic rendering pipeline configured via the property tree and XML
|status      = * experimental as of 02/2018
* pending review (06/2018) <ref>https://sourceforge.net/p/flightgear/mailman/message/36355833/</ref> 
* put up for review/discussion as of 08/2018
* pending merge-request (02/2019) <ref>https://sourceforge.net/p/flightgear/mailman/message/36578956/</ref>
|maintainers = none
|developers  = Icecode
|topic-sg = https://sourceforge.net/u/fgarlin/simgear/ci/new-compositor/tree/
|topic-fg = https://sourceforge.net/u/fgarlin/flightgear-src/ci/new-compositor/tree/
|topic-fgdata = https://sourceforge.net/u/fgarlin/flightgear/ci/new-compositor/tree/
|merge-requests = 
* https://sourceforge.net/p/flightgear/simgear/merge-requests/47/
* https://sourceforge.net/p/flightgear/flightgear/merge-requests/149/
* https://sourceforge.net/p/flightgear/fgdata/merge-requests/131/
}}

The '''Compositor''' aims to bring multi-pass rendering to FlightGear. It encapsulates a rendering pipeline and exposes its parameters to the [[Property Tree]].
The pipeline used on each physical viewport defined on the [[Howto:Configure camera view windows|CameraGroup settings]] can be changed via a <code><compositor/></code> tag containing a valid path to the Compositor XML file (using the [[PropertyList XML File]] format).

The Compositor introduces a new dedicated fgdata directory for new/custom rendering pipelines: $FG_ROOT/Compositor.

This is also where the default pipeline can be found, namely: $FG_ROOT/Compositor/default.xml

== Background ==
{{See also|Supporting multiple renderers|Howto:Canvas View Camera Element}}

First discussed in 03/2012 during the early [[Rembrandt]] days, Zan (Lauri Peltonen) came up with a set of patches demonstrating how to create an XML-configurable rendering pipeline.

Back then, this work was considered to look pretty promising <ref>{{cite web
  |url    =  https://sourceforge.net/p/flightgear/mailman/message/28946515/ 
  |title  =  <nowiki> Re: [Flightgear-devel] [Rembrandt] the plan </nowiki> 
  |author =  <nowiki> Mathias Fröhlich </nowiki> 
  |date   =  Mar 7th, 2012 
  |added  =  Mar 7th, 2012 
  |script_version = 0.36 
  }}</ref> and at the time plans were discussed to unify this with the ongoing Rembrandt implementation (no longer maintained).

Adopting Zan's approach would have meant that efforts like [[Rembrandt]] (deferred rendering) could have been implemented without requiring C++ space modifications, i.e. purely in [[Base package]] space.

Rembrandt's developer (FredB) suggested to extend the format to avoid duplicating the stages when you have more than one viewport, i.e.  specifying a pipeline as a template, with conditions like in effects, and have the current camera layout refer the pipeline that would be duplicated, resized and positioned for each declared viewport <ref>{{cite web
  |url    =  https://sourceforge.net/p/flightgear/mailman/message/28944773/ 
  |title  =  <nowiki> Re: [Flightgear-devel] [Rembrandt] the plan </nowiki> 
  |author =  <nowiki> Frederic Bouvier </nowiki> 
  |date   =  Mar 7th, 2012 
  |added  =  Mar 7th, 2012 
  |script_version = 0.36 
  }}</ref>


Zan's original patches can still be found in his newcameras branches which allow the user to define the rendering pipeline in preferences.xml:

* [https://gitorious.org/fg/zans-flightgear?p=fg:zans-flightgear.git;a=shortlog;h=refs/heads/newcameras FlightGear]
* [https://gitorious.org/fg/zans-simgear?p=fg:zans-simgear.git;a=log;h=refs/heads/newcameras SimGear]

At that point, it didn't have everything Rembrandt's pipeline needs, but most likely could be easily enhanced to support those things. 

Basically, the original version added support for multiple camera passes, texture targets, texture formats, passing textures from one pass to another etc, while preserving the standard rendering line if user wants that. <ref>{{cite web
  |url    =  https://sourceforge.net/p/flightgear/mailman/message/28944733/ 
  |title  =  <nowiki> [Flightgear-devel] [Rembrandt] the plan </nowiki> 
  |author =  <nowiki> Lauri Peltonen </nowiki> 
  |date   =  Mar 7th, 2012 
  |added  =  Mar 7th, 2012 
  |script_version = 0.36 
  }}</ref>

== Status ==
'''02/2019'''

The Compositor is in an usable state right now: it works but there are no effects or pipelines developed for it. There are also some bugs and features that don't work as expected because of some hardcoded assumptions in the FlightGear Viewer code.

The corresponding set of patches (topic branches) were put up for review in 08/2018 to discuss the underlying approach and hopefully get this merged in 2019. <ref>https://sourceforge.net/p/flightgear/mailman/message/36355833/</ref>

As of 02/2019, a set of merge requests has been created for the Compositor framework for SimGear, FlightGear and FGData respectively.

The Compositor is disabled by default at compilation time in
FlightGear (not SimGear). Both Rembrandt and the forward pipeline are
still available if this flag is disabled.

(There is a known issue where FG will crash with a segfault if a valid
Compositor XML isn't found.)

The FGData merge request only includes the Low-spec pipeline, which
imitates the current near/far camera forward pipeline.<ref>https://sourceforge.net/p/flightgear/mailman/message/36578956/</ref>

James and Richard agreed that śince compilation is disabled by default, it might be easier to merge more or less as-is (unless there’s something really awkward), and then make any adjustments on top? 

That also means we can enable this option on Jenkins to get platform coverage.<ref>https://sourceforge.net/p/flightgear/mailman/message/36579111/</ref> <ref>https://sourceforge.net/p/flightgear/mailman/message/36583393/</ref>

And this would make things easier for other people willing to collaborate.

Everything is structured so nothing is affected when the compositor is
disabled at compile time, so there shouldn't be any breakage from the
commit.<ref>https://sourceforge.net/p/flightgear/mailman/message/36583493/</ref>

== Features ==

* Completely independent of other parts of the simulator, i.e. it's part of SimGear and can be used in a standalone fashion if needed, ala Canvas.
* Although independent, its aim is to be fully compatible with the current rendering framework in FG. This includes the Effects system, CameraGroup, Rembrandt and ALS (and obviously the Canvas).
* Its functionality overlaps Rembrandt: what can be done with Rembrandt can be done with the Compositor, but not vice versa.
* Fully configurable via an XML interface without compromising performance (ala Effects, using PropertyList files).
* Flexible, expandable and compatible with modern graphics.
* It doesn't increase the hardware requirements, it expands the hardware range FG can run on. People with integrated GPUs (Intel HD etc) can run a Compositor with a single pass that renders directly to the screen like before, while people with more powerful cards can run a Compositor that implements deferred rendering, for example.

== Elements ==

=== Buffers ===

A buffer represents a texture or, more generically, a region of GPU memory. It can have the following parameters:

; name
: Passes will be able to address the buffer by this name.
; type
: Any texture type allowed by OpenGL: 1d, 2d, 2d-array, 2d-multisample, 3d, rect or cubemap.
; width
: Texture width. It's possible to write 'screen' to use the physical viewport width.
; screen-width-scale
: If 'screen' was used, this controls the width scaling factor.
; height
: Texture height. It's possible to write 'screen' to use the physical viewport height.
; screen-height-scale
: If 'screen' was used, this controls the height scaling factor.
; depth
: Texture depth.
; internal-format, source-format and source-type
: They specify the texture format. They correspond to the arguments ''internalformat'', ''format'' and ''type'' respectively of the OpenGL function ''glTexImage2D''.
; min-filter and mag-filter (Optional)
: They change the minification and magnification filtering respectively. Possible values are: linear, linear-mipmap-linear, linear-mipmap-nearest, nearest, nearest-mipmap-linear and nearest-mipmap-nearest. The default value for both filters is linear.
; wrap-s, wrap-t and wrap-r (Optional)
: They change the wrap mode for each coordinate. Possible values are: clamp, clamp-to-edge, clamp-to-border, repeat and mirror. The default value for every coordinate is clamp-to-border.

A typical [[PropertyList XML File|property tree structure]] describing a buffer may be as follows:

<syntaxhighlight lang="xml">
<buffer>
  <name>color</name>
  <type>2d</type>
  <width>screen</width>
  <height>screen</height>
  <screen-width-scale>1.5</screen-width-scale>
  <screen-height-scale>1.5</screen-height-scale>
  <internal-format>rgba8</internal-format>
  <source-format>rgba</source-format>
  <source-type>ubyte</source-type>
  <min-filter>nearest-mipmap-nearest</min-filter>
  <mag-filter>nearest-mipmap-nearest</mag-filter> 
  <wrap-s>repeat</wrap-s>
  <wrap-t>repeat</wrap-t>
</buffer>
</syntaxhighlight>

=== Passes ===

A pass wraps around an [http://public.vrac.iastate.edu/vancegroup/docs/OpenSceneGraphReferenceDocs-3.0/a00089.html osg::Camera]. Passes all have some common parameters:

; condition
: A valid XML expression that enables the pass when met.
; clear-color, clear-accum, clear-depth and clear-stencil
: Default values are black, black, 1.0 and 0 respectively.
; clear-color-bit, clear-accum-bit, clear-depth-bit and clear-stencil-bit
: Default values are true, false, true, false, respectively.
; effect-override
: Use the specified effect to draw every object. This is useful for special passes like G-buffer and shadow map optimization.

There are specific pass types, each with their own set of custom parameters:

* '''scene'''
Renders the scene from the point of view given by the CameraGroup.
; cull-mask
: A 32 bit number that specifies the cull mask to be used. See <tt>simgear/scene/util/RenderConstants.hxx</tt> to know which bits enable what.
; z-near and z-far
: They change the depth range to be used. If both of them are zero, the default Z range in the CameraGroup is used.
; clustered-forward
: Enables the use of clustered forward rendering for this pass.
; cubemap-face
: Ignores the given view and projection matrices and uses a custom one that renders the scene as if it was seen from inside a cubemap looking towards the specified face.

* '''quad'''
Renders a fullscreen quad with an optional [[Effects|effect]] applied. Useful for screen space shaders (like SSAO, Screen Space Reflections or bloom) and deferred rendering.
; geometry
: Specifies the x, y, width and height of the fullscreen quad inside the viewport using normalized coordinates.
; effect
: The quad will use this effect.

* '''shadow-map'''
Renders the scene from a light's point of view.
; light-num
: The OpenGL light number to use for this shadow map.
; near-m and far-m
: They specify the range of the shadow map.

Passes can render to a buffer (Render to Texture), to several buffers (Multiple Render Targets) or directly to the framebuffer. This is accomplished by the <code><attachment/></code> tag. Possible parameters of an attachment are:

; buffer
: The name of the buffer to output to.
; component
: FBO attachment point. Possible values are color0 to color15, depth, stencil and depth-stencil.
; level (Optional)
: Controls the mip map level of the texture that is attached. Default value is 0.
; face (Optional)
: Controls the face of texture cube map or z level of 3d texture. Default value is 0.
; mipmap-generation (Optional)
: Controls whether mipmap generation should be done for texture. Default value is false.
; multisample-samples (Optional)
: MSAA samples. Default value is 0.
; multisample-color-samples (Optional)
: MSAA color samples. Default value is 0.

Passes can also receive buffers as input and use them in their shaders. This is accomplished by the <code><binding/></code> tag, which has the following parameters:

; buffer
: The name of the buffer to bind.
; unit
: The texture unit to place the texture on. Effects will be able to access the buffer on this texture unit.

Example XML for a ''scene'' type pass:

<syntaxhighlight lang="xml">
<pass>
  <name>forward-lighting</name>
  <type>scene</type>
  <clear-color type="vec4d">0 0 0 0</clear-color>

  <clustered-forward/>

  <binding>
    <buffer>shadowmap</buffer>
    <unit>10</unit>
  </binding>
  <attachment>
    <buffer>color</buffer>
    <component>color0</component>
    <multisample-samples>4</multisample-samples>
    <multisample-color-samples>4</multisample-color-samples>
  </attachment>
  <attachment>
    <buffer>depth</buffer>
    <component>depth</component>
  </attachment>
</pass>
</syntaxhighlight>

== Pipelines ==

=== Low-Spec pipeline ===

A fixed function forward rendering pipeline mainly targeted to low spec systems. It imitates the classic forward pipeline used before multi-pass rendering was introduced by using two near/far cameras rendering directly to the screen.

[[File:Low-spec Compositor pipeline.png|thumb|Screenshot showing OSG stats of the Compositor-based low-spec rendering pipeline.]]

== Canvas integration ==

[[File:Blur_and_blue_filter_applied_to_a_buffer.png|thumb|Post processed image (right) after applying gaussian blur and a blue filter to a scene pass (left).]]

Apart from serving as a debugging tool for visualizing the contents of a buffer, integrating the Compositor with [[Canvas]] allows aircraft developers to access RTT capabilities. Compositor buffers can be accessed within Canvas via a new custom Canvas Image protocol '''buffer://'''. For example, the path <code>buffer://test-compositor/test-buffer</code> displays the buffer test-buffer declared in test-compositor.

<syntaxhighlight lang="nasal">
var (width,height) = (612,612);
var title = 'Compositor&Canvas test';
var window = canvas.Window.new([width,height],"dialog")
 .set('title',title);
var myCanvas = window.createCanvas().set("background", canvas.style.getColor("bg_color"));
var root = myCanvas.createGroup();
var path = "buffer://test-compositor/test-buffer";
var child = root.createChild("image")
    .setTranslation(50, 50)
    .setSize(512, 512)
    .setFile(path);
</syntaxhighlight>

== TODO ==

* Bring back distortion correction.
* Finish Clustered Forward Rendering.
* Some kind of versioning system to be able to make breaking changes in the future if/when the compositor is updated in any significant way, without people having to manually update their configs.
* Coherently support static conditional branching (<condition> tags available for both passes and buffers), as well as using property tree parameters (via a <property> tag ala Effects). For simplicity and to avoid major architectural changes, these conditional checks should only be performed on startup (no dynamic branching). For example, the user should be able to select if they want shadows and their size in the launcher, not on the in-sim Rendering dialog.

== Known Issues ==

* Setting a buffer scale factor different from 1.0 and rendering to it might not scale the splash screen correctly.
* No error handling when the Compositor wasn't found. FlightGear should quit gracefully but right now it segfaults.
* computeIntersections() always uses pass 0, which might fail. The scene pass should be used (which one if there are several?).

== Related ==
=== Articles ===
* [[Uniform Buffer Objects]]
* [[CompositeViewer Support]]
* [[FlightGear CIGI Support (Common Image Generator Interface)]]
=== Discussions ===
* [https://forum.flightgear.org/viewtopic.php?f=47&t=35095 Clustered Forward Rendering] (12/2018)
* [https://forum.flightgear.org/viewtopic.php?f=47&t=33045 Getting started with RTT]
* [https://forum.flightgear.org/viewtopic.php?f=47&t=32846 Canvas::View Development]
* [https://forum.flightgear.org/viewtopic.php?f=71&t=23929 Gear View in Cockpit]
* [https://forum.flightgear.org/viewtopic.php?f=37&t=20057 Rear-View mirror]
* [https://forum.flightgear.org/viewtopic.php?f=30&t=18905 Progress on synthetic terrain]
* [https://forum.flightgear.org/viewtopic.php?f=71&t=17184 Instruments with heightmaps]


{{Appendix}}