Difference between revisions of "Compositor"

From FlightGear wiki
Jump to navigation Jump to search
Line 1: Line 1:
 
+
{{Template:Non-stable|version=2019.3|progress=80}}
{{Template:Non-stable|version=2019.3|progress=60}}
 
 
 
  
 
{{infobox subsystem
 
{{infobox subsystem
Line 17: Line 15:
 
}}
 
}}
  
A '''Compositor''' is a rendering pipeline configured by the [[Property Tree]]. Every configurable element of the pipeline wraps an OpenGL/OSG object and exposes its parameters to the property tree. This way, the rendering pipeline becomes dynamically-reconfigurable at runtime.
+
The '''Compositor''' aims to bring multi-pass rendering to FlightGear. It encapsulates a rendering pipeline and exposes its parameters to the [[Property Tree]]. The pipeline used on each physical viewport defined on the [[Howto:Configure camera view windows||CameraGroup settings]] can be changed via the <compositor/> tag containing a valid path to the Compositor XML file.
 
 
Related efforts: [[Howto:Canvas View Camera Element]]
 
  
 
== Background ==
 
== Background ==
Line 62: Line 58:
 
   |script_version = 0.36  
 
   |script_version = 0.36  
 
   }}</ref>
 
   }}</ref>
 
In 02/2018, Icecode GL re-implemented a more generic version of Zan's idea using Canvas concepts, which means the rendering pipeline is not just xml-configurable, but using listeners to be dynamically-reconfigurable at runtime via properties.
 
  
 
The corresponding set of patches (topic branches) were put up for review in 08/2018 to discuss the underlying approach and hopefully get this merged in 2019.
 
The corresponding set of patches (topic branches) were put up for review in 08/2018 to discuss the underlying approach and hopefully get this merged in 2019.
  
== Status ==
+
== Features ==
'''08/2018'''
 
 
 
Fernando has been working on and off on multi-pass rendering support for
 
FlightGear since late 2017. It went through several iterations and
 
design changes, but he think it's finally on the right track. It's
 
heavily based on the Ogre3D Compositor and inspired by many
 
data-driven rendering pipelines. Its features include:
 
  
 
* Completely independent of other parts of the simulator, i.e. it's part of SimGear and can be used in a standalone fashion if needed, ala Canvas.
 
* Completely independent of other parts of the simulator, i.e. it's part of SimGear and can be used in a standalone fashion if needed, ala Canvas.
Line 80: Line 67:
 
* Its functionality overlaps Rembrandt: what can be done with Rembrandt can be done with the Compositor, but not vice versa.
 
* Its functionality overlaps Rembrandt: what can be done with Rembrandt can be done with the Compositor, but not vice versa.
 
* Fully configurable via a XML interface without compromising performance (ala Effects, using PropertyList files).
 
* Fully configurable via a XML interface without compromising performance (ala Effects, using PropertyList files).
* Optional at compile time to aid merge request efforts.
 
 
* Flexible, expandable and compatible with modern graphics.
 
* Flexible, expandable and compatible with modern graphics.
 
* It doesn't increase the hardware requirements, it expands the hardware range FG can run on. People with integrated GPUs (Intel HD etc) can run a Compositor with a single pass that renders directly to the screen like before, while people with more powerful cards can run a Compositor that implements deferred rendering, for example.
 
* It doesn't increase the hardware requirements, it expands the hardware range FG can run on. People with integrated GPUs (Intel HD etc) can run a Compositor with a single pass that renders directly to the screen like before, while people with more powerful cards can run a Compositor that implements deferred rendering, for example.
  
Unlike Rembrandt, the Compositor makes use of scene graph cameras
+
The Compositor is in an usable state right now: it works but there are no effects or pipelines developed for it. There are also some bugs and features that don't work as expected because of some hardcoded assumptions in the FlightGear Viewer code.
instead of viewer level cameras.
 
 
 
This allows CameraGroup to manage windows, near/far cameras and other
 
slaves without interfering with what is being rendered
 
(post-processing, shadows...).
 
 
 
The Compositor is in an usable state right now: it works but there are
 
no effects or pipelines developed for it. There are also some bugs and
 
features that don't work as expected because of some hardcoded
 
assumptions in the FlightGear Viewer code. Still, I think it's time to
 
announce it so people much more knowledgeable than me can point me in
 
the right direction to get this upstream and warn me about possible
 
issues and worries. :)
 
 
 
[[File:Compositor-08-2018.png|thumb|A screenshot showing post-processed gaussian blur and a blue filter:[https://i.imgur.com/Zzxre8a.png]]]
 
 
 
The next major (and hopefully final) iteration of the Compositor is
 
mostly done now, just some more cleanup is needed. I've tried to keep
 
the changes contained in CameraGroup.cxx/hxx and in new files in
 
SimGear to ease the merging process.
 
 
 
== Use Cases ==
 
* tail cameras
 
* mirrors
 
* [[Procedural Texturing]]
 
* [[Shuttle ADI ball]]
 
 
 
== FAQs ==
 
{{See also|Unifying the 2D rendering backend via canvas}}
 
 
 
I started the Compositor to hopefully unify rendering in FG in the most
 
non-radical way. Instead of changing a large part of the codebase like
 
Rembrandt, the Compositor aims to bring the same funcionality and slowly
 
deprecate parts of the FG Viewer code.
 
 
 
Indeed, this should help bring more interesting effects to the table.
 
I'm specially looking forward to porting ALS to a multi-pass
 
environment through PBR.
 
 
 
=== Shadows ===
 
It includes shadows indeed. The Compositor stacks several passes of
 
different types on top of each other, and exchanges buffers between
 
them. Nothing prevents someone from creating a new pass type that
 
implements a shadow map pre-pass. I've already tried myself
 
successfully, but the problem is telling the pass which light should be
 
used, which brings me to spotlights.
 
 
 
=== Light Sources ===
 
Allowing aircraft/scenery developers to include "real" lights is a whole
 
different aspect of the scene graph that shouldn't be managed by the
 
Compositor. Some kind of system that exposes OpenGL lights should be
 
implemented and linked to the Compositor. Some decisions have to be made
 
though. In a forward renderer lights aren't as cheap as in a deferred
 
renderer, so if the aircraft developer says there has to be a light
 
there, someone using a forward renderer might see a performance drop or
 
not see them at all.
 
 
 
 
 
=== Show me the code ===
 
I've submitted all I have to my FG fork:
 
 
 
 
 
https://sourceforge.net/u/fgarlin/simgear/ci/new-compositor/tree/
 
 
 
https://sourceforge.net/u/fgarlin/flightgear-src/ci/new-compositor/tree/
 
 
 
https://sourceforge.net/u/fgarlin/flightgear/ci/new-compositor/tree/
 
 
 
I haven't had the time to update the repos to the latest changes in the  official repos, but it shouldn't matter for now.
 
 
 
=== Merging ===
 
I'd love to have this merged as soon as possible, but maybe next month is a
 
bit too rushed. I don't even know if the approach I'm using will be the
 
definitive one or the small implications the Compositor might have on the
 
rest of the viewer related code. I'm not in a rush though, I can wait for
 
the next release if I can't make a decent enough merge request in June/July.
 
  
 
== Elements ==
 
== Elements ==
Line 166: Line 76:
 
=== Buffers ===
 
=== Buffers ===
  
A buffer represents a texture or, more generically, a region of GPU memory. Textures can be of any type allowed by OpenGL: 1D, 2D, rectangle, 2Darray, 3D or cubemap.
+
A buffer represents a texture or, more generically, a region of GPU memory. It can have the following parameters:
 +
 
 +
; '''name'''
 +
: Passes will be able to address the buffer by this name.
 +
; '''type'''
 +
: Any texture type allowed by OpenGL: 1d, 2d, 2d-array, 2d-multisample, 3d, rect or cubemap.
 +
; '''width'''
 +
: Texture width. It's possible to write 'screen' to use the physical viewport width.
 +
; '''screen-width-scale'''
 +
: If 'screen' was used, this controls the width scaling factor.
 +
; '''height'''
 +
: Texture height. It's possible to write 'screen' to use the physical viewport height.
 +
; '''screen-height-scale'''
 +
: If 'screen' was used, this controls the height scaling factor.
 +
; '''depth'''
 +
: Texture depth.
 +
; '''internal-format''', '''source-format''' and '''source-type'''
 +
: They specify the texture format. They correspond to the arguments ''internalformat'', ''format'' and ''type'' respectively of the OpenGL function ''glTexImage2D''.
 +
; '''min-filter''' and '''mag-filter''' (Optional)
 +
: They change the minification and magnification filtering respectively. Possible values are: linear, linear-mipmap-linear, linear-mipmap-nearest, nearest, nearest-mipmap-linear and nearest-mipmap-nearest. The default value for both filters is linear.
 +
; '''wrap-s''', '''wrap-t''' and '''wrap-r''' (Optional)
 +
: They change the wrap mode for each coordinate. Possible values are: clamp, clamp-to-edge, clamp-to-border, repeat and mirror. The default value for every coordinate is clamp-to-border.
  
 
A typical [[PropertyList XML File|property tree structure]] describing a buffer may be as follows:
 
A typical [[PropertyList XML File|property tree structure]] describing a buffer may be as follows:
  
 
<syntaxhighlight lang="xml">
 
<syntaxhighlight lang="xml">
      <buffer>
+
<buffer>
            <name>buffer-name</name>
+
  <name>color</name>
    <type>2d</type>
+
  <type>2d</type>
    <width>512</width>
+
  <width>screen</width>
    <height>512</height>
+
  <height>screen</height>
    <scale-factor>1.0</scale-factor>
+
  <screen-width-scale>1.5</screen-width-scale>
    <internal-format>rgba8</internal-format>
+
  <screen-height-scale>1.5</screen-height-scale>
    <source-format>rgba</source-format>
+
  <internal-format>rgba8</internal-format>
            <source-type>ubyte</source-type>
+
  <source-format>rgba</source-format>
      </buffer>
+
  <source-type>ubyte</source-type>
 +
  <min-filter>nearest-mipmap-nearest</min-filter>
 +
  <mag-filter>nearest-mipmap-nearest</mag-filter>
 +
  <wrap-s>repeat</wrap-s>
 +
  <wrap-t>repeat</wrap-t>
 +
</buffer>
 
</syntaxhighlight>
 
</syntaxhighlight>
  
 
=== Passes ===
 
=== Passes ===
  
A pass wraps around an [http://public.vrac.iastate.edu/vancegroup/docs/OpenSceneGraphReferenceDocs-3.0/a00089.html osg::Camera]. As of February 2018, there are two types of passes supported:
+
A pass wraps around an [http://public.vrac.iastate.edu/vancegroup/docs/OpenSceneGraphReferenceDocs-3.0/a00089.html osg::Camera]. The following types of passes are supported:
  
* '''scene'''. Renders a scene. The scene can be an already loaded scene graph node (terrain, aircraft etc.) or a path to a custom 3D model.
+
; '''shadow-map'''
* '''quad'''. Renders a fullscreen quad with an optional [[Effects|effect]] applied. Useful for screen space shaders (like SSAO, Screen Space Reflections or bloom) and deferred rendering.
+
: Renders the scene from a light's point of view.
 +
; '''scene'''
 +
: Renders the scene from the point of view given by the CameraGroup.
 +
; '''quad'''
 +
: Renders a fullscreen quad with an optional [[Effects|effect]] applied. Useful for screen space shaders (like SSAO, Screen Space Reflections or bloom) and deferred rendering.
  
Passes can render to a buffer (Render to Texture), to several buffers (Multiple Render Targets) or directly to the OSG context. This allows chaining of multiple passes, sharing buffers between them. They can also receive other buffers as input, so effects can access them as normal textures.
+
Passes can render to a buffer (Render to Texture), to several buffers (Multiple Render Targets) or directly to the framebuffer. This allows chaining of multiple passes, sharing buffers between them. They can also receive other buffers as input, so effects can access them as textures.
  
Example XML for a ''quad'' type pass:
+
Example XML for a ''scene'' type pass:
  
 
<syntaxhighlight lang="xml">
 
<syntaxhighlight lang="xml">
      <pass-quad>
+
<pass>
          <name>pass-name</name>
+
  <name>forward-lighting</name>
 +
  <type>scene</type>
 +
  <clear-color type="vec4d">0 0 0 0</clear-color>
  
          <effect>Effects/test</effect>
+
  <clustered-forward/>
   
 
          <input-buffer>
 
              <buffer>some-buffer</buffer>
 
              <unit>0</unit>
 
          </input-buffer>
 
  
          <output-buffer>
+
  <attachment>
              <buffer>color-buffer</buffer>
+
    <buffer>color</buffer>
              <component>color</component>
+
    <component>color0</component>
          </output-buffer>
+
    <multisample-samples>4</multisample-samples>
          <output-buffer>
+
    <multisample-color-samples>4</multisample-color-samples>
              <buffer>depth-buffer</buffer>
+
  </attachment>
              <component>depth</component>
+
  <attachment>
          </output-buffer>
+
    <buffer>depth</buffer>
      </pass-quad>
+
    <component>depth</component>
 +
  </attachment>
 +
</pass>
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Line 238: Line 177:
 
== Known Issues ==
 
== Known Issues ==
  
* Canvas (sc::Image) doesn't update its texture unless .update() is invoked in Nasal. Is this a feature?
+
* Setting a buffer scale factor different from 1.0 and rendering to it might not scale the splash screen correctly.
* The scene graph and the viewer are still managed by FGRenderer. Maybe hijacking the init sequence is a good idea?
+
* No error handling when the Compositor wasn't found. FlightGear should quit gracefully but right now it segfaults.
* light sources ?
+
* Hardcoded Z handling. Better handling of depth is required to allow, for example, near and far cameras to prevent Z-fighting.
  
 
== Related ==
 
== Related ==
 
{{Appendix}}
 
{{Appendix}}

Revision as of 00:13, 2 January 2019

This article describes content/features that may not yet be available in the latest stable version of FlightGear (2020.3).
You may need to install some extra components, use the latest development (Git) version or even rebuild FlightGear from source, possibly from a custom topic branch using special build settings: .

This feature is scheduled for FlightGear 2019.3. 80}% completed

If you'd like to learn more about getting your own ideas into FlightGear, check out Implementing new features for FlightGear.

Compositor Subsystem
Canvas-view-element-prototype-by-icecode gl.png
Started in 01/2018
Description Dynamic rendering pipeline configured via the property tree
Maintainer(s) none
Contributor(s) Icecode
Status
  • experimental as of 02/2018
  • put up for review/discussion as of 08/2018
Topic branches:
$FG_SRC https://sourceforge.net/u/fgarlin/flightgear-src/ci/new-compositor/tree/
fgdata https://sourceforge.net/u/fgarlin/flightgear/ci/new-compositor/tree/

The Compositor aims to bring multi-pass rendering to FlightGear. It encapsulates a rendering pipeline and exposes its parameters to the Property Tree. The pipeline used on each physical viewport defined on the |CameraGroup settings can be changed via the <compositor/> tag containing a valid path to the Compositor XML file.

Background

First discussed in 03/2012 during the early Rembrandt days, Zan came up with patches demonstrating how to create an XML-configurable rendering pipeline.

Back then, this work was considered to look pretty promising [1] and at the time plans were discussed to unify this with the ongoing Rembrandt implementation (no longer maintained).

Adopting Zan's approach would have meant that efforts like Rembrandt could have been implemented without requiring C++ space modifications.

Rembrandt's developer (FredB) suggested to extend the format to avoid duplicating the stages when you have more than one viewport, i.e. specifying a pipeline as a template, with conditions like in effects, and have the current camera layout refer the pipeline that would be duplicated, resized and positioned for each declared viewport [2]


Zan's original patches can still be found in his newcameras branches which allow the user to define the rendering pipeline in preferences.xml:

At that point, it didn't have everything Rembrandt's pipeline needs, but most likely could be easily enhanced to support those things.

Basically the original version added support for multiple camera passes, texture targets, texture formats, passing textures from one pass to another etc, while preserving the standard rendering line if user wants that. [3]

The corresponding set of patches (topic branches) were put up for review in 08/2018 to discuss the underlying approach and hopefully get this merged in 2019.

Features

  • Completely independent of other parts of the simulator, i.e. it's part of SimGear and can be used in a standalone fashion if needed, ala Canvas.
  • Although independent, its aim is to be fully compatible with the current rendering framework in FG. This includes the Effects system, CameraGroup, Rembrandt and ALS (and obviously the Canvas).
  • Its functionality overlaps Rembrandt: what can be done with Rembrandt can be done with the Compositor, but not vice versa.
  • Fully configurable via a XML interface without compromising performance (ala Effects, using PropertyList files).
  • Flexible, expandable and compatible with modern graphics.
  • It doesn't increase the hardware requirements, it expands the hardware range FG can run on. People with integrated GPUs (Intel HD etc) can run a Compositor with a single pass that renders directly to the screen like before, while people with more powerful cards can run a Compositor that implements deferred rendering, for example.

The Compositor is in an usable state right now: it works but there are no effects or pipelines developed for it. There are also some bugs and features that don't work as expected because of some hardcoded assumptions in the FlightGear Viewer code.

Elements

Buffers

A buffer represents a texture or, more generically, a region of GPU memory. It can have the following parameters:

name
Passes will be able to address the buffer by this name.
type
Any texture type allowed by OpenGL: 1d, 2d, 2d-array, 2d-multisample, 3d, rect or cubemap.
width
Texture width. It's possible to write 'screen' to use the physical viewport width.
screen-width-scale
If 'screen' was used, this controls the width scaling factor.
height
Texture height. It's possible to write 'screen' to use the physical viewport height.
screen-height-scale
If 'screen' was used, this controls the height scaling factor.
depth
Texture depth.
internal-format, source-format and source-type
They specify the texture format. They correspond to the arguments internalformat, format and type respectively of the OpenGL function glTexImage2D.
min-filter and mag-filter (Optional)
They change the minification and magnification filtering respectively. Possible values are: linear, linear-mipmap-linear, linear-mipmap-nearest, nearest, nearest-mipmap-linear and nearest-mipmap-nearest. The default value for both filters is linear.
wrap-s, wrap-t and wrap-r (Optional)
They change the wrap mode for each coordinate. Possible values are: clamp, clamp-to-edge, clamp-to-border, repeat and mirror. The default value for every coordinate is clamp-to-border.

A typical property tree structure describing a buffer may be as follows:

<buffer>
  <name>color</name>
  <type>2d</type>
  <width>screen</width>
  <height>screen</height>
  <screen-width-scale>1.5</screen-width-scale>
  <screen-height-scale>1.5</screen-height-scale>
  <internal-format>rgba8</internal-format>
  <source-format>rgba</source-format>
  <source-type>ubyte</source-type>
  <min-filter>nearest-mipmap-nearest</min-filter>
  <mag-filter>nearest-mipmap-nearest</mag-filter> 
  <wrap-s>repeat</wrap-s>
  <wrap-t>repeat</wrap-t>
</buffer>

Passes

A pass wraps around an osg::Camera. The following types of passes are supported:

shadow-map
Renders the scene from a light's point of view.
scene
Renders the scene from the point of view given by the CameraGroup.
quad
Renders a fullscreen quad with an optional effect applied. Useful for screen space shaders (like SSAO, Screen Space Reflections or bloom) and deferred rendering.

Passes can render to a buffer (Render to Texture), to several buffers (Multiple Render Targets) or directly to the framebuffer. This allows chaining of multiple passes, sharing buffers between them. They can also receive other buffers as input, so effects can access them as textures.

Example XML for a scene type pass:

<pass>
  <name>forward-lighting</name>
  <type>scene</type>
  <clear-color type="vec4d">0 0 0 0</clear-color>

  <clustered-forward/>

  <attachment>
    <buffer>color</buffer>
    <component>color0</component>
    <multisample-samples>4</multisample-samples>
    <multisample-color-samples>4</multisample-color-samples>
  </attachment>
  <attachment>
    <buffer>depth</buffer>
    <component>depth</component>
  </attachment>
</pass>

Canvas integration

Post processed image (right) after applying gaussian blur and a blue filter to a scene pass (left).

Apart from serving as a debugging tool for visualizing the contents of a buffer, integrating the Compositor with Canvas allows aircraft developers to access RTT capabilities. Compositor buffers can be accessed within Canvas via a new custom Canvas Image protocol buffer://. For example, the path buffer://test-compositor/test-buffer displays the buffer test-buffer declared in test-compositor.

var (width,height) = (612,612);
var title = 'Compositor&Canvas test';
var window = canvas.Window.new([width,height],"dialog")
 .set('title',title);
var myCanvas = window.createCanvas().set("background", canvas.style.getColor("bg_color"));
var root = myCanvas.createGroup();
var path = "buffer://test-compositor/test-buffer";
var child = root.createChild("image")
    .setTranslation(50, 50)
    .setSize(512, 512)
    .setFile(path);

Known Issues

  • Setting a buffer scale factor different from 1.0 and rendering to it might not scale the splash screen correctly.
  • No error handling when the Compositor wasn't found. FlightGear should quit gracefully but right now it segfaults.
  • Hardcoded Z handling. Better handling of depth is required to allow, for example, near and far cameras to prevent Z-fighting.

Related

References
  1. Mathias Fröhlich  (Mar 7th, 2012).  Re: [Flightgear-devel] [Rembrandt] the plan .
  2. Frederic Bouvier  (Mar 7th, 2012).  Re: [Flightgear-devel] [Rembrandt] the plan .
  3. Lauri Peltonen  (Mar 7th, 2012).  [Flightgear-devel] [Rembrandt] the plan .