Difference between revisions of "Canvas View Camera Element"

From FlightGear wiki
Jump to navigation Jump to search
m (→‎Create a Canvas Element subclass: https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg26193.html)
 
(45 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
{{FGCquote
 +
  |It would make sense to integrate all efforts ([[CompositeViewer Support|CompositeViewer]], [[Canvas]] and [[Compositor]]) to create a [[Canvas_Development#Elements|Canvas Element]] that can render an [[Canvas_Development#Supporting_Cameras|arbitrary view]].
 +
 +
Some time ago I wrote what I think are the required steps to get something like cockpit mirrors working: [[#Roadmap]]
 +
 +
The main showstopper was having [[CompositeViewer Support]], but since that effort is underway, all the parts are there for someone willing to dive in.
 +
  |{{cite web |url=https://sourceforge.net/p/flightgear/mailman/message/37089780/
 +
    |title=<nowiki>Independent view windows</nowiki>
 +
    |author=<nowiki>Fernando García Liñán </nowiki>
 +
    |date=<nowiki>2020-08-20 14:27:58</nowiki>
 +
  }}
 +
}}
 +
 
{{infobox subsystem
 
{{infobox subsystem
 +
|image      = cvcanvas-777.jpeg|Demonstration of multiple view windows in flightgear
 +
|started    = 11/2020
 +
 
|name        = Canvas Camera Views
 
|name        = Canvas Camera Views
|description = Rendering arbitrary views to a texture/Canvas
+
|description = Rendering arbitrary scene views to a texture/Canvas
|status      = planned/prototype
+
|status      = experimental prototype
 
|maintainers = none
 
|maintainers = none
|developers  = none
+
|developers  = Cgdae,  Icecode
 
}}
 
}}
  
Line 11: Line 27:
 
Several aircraft developers have manifested their interest in being able to render scene views to a texture (RTT) {{Wikipedia|Render target}} and use it inside cockpits as mirrors, external cameras (so called tail cams) and other uses.
 
Several aircraft developers have manifested their interest in being able to render scene views to a texture (RTT) {{Wikipedia|Render target}} and use it inside cockpits as mirrors, external cameras (so called tail cams) and other uses.
  
With the [[Compositor]] framework it's possible to configure custom render pipelines in XML space, which together with Effect schemes allow complete control over how a rendered scene looks. A synthetic terrain can be rendered by assigning an Effect scheme to a <tt>scene</tt> type pass, which textures all terrain with a brown checkerboard pattern and paints the sky in a solid blue color. All other objects are ignored and aren't rendered, saving a lot of computation time. Custom render distances independent from the main LOD settings can also be used to further optimize the performance of the display.
+
With the [[Compositor]] framework it's possible to configure custom render pipelines in XML space, which together with Effect schemes allow complete control over how a rendered scene looks. A synthetic terrain can be rendered by assigning an Effect scheme to a <tt>scene</tt> type pass, which textures all terrain with a brown checkerboard pattern and paints the sky in a solid blue color. All other objects are ignored and aren't rendered, saving a lot of computation time. Custom render distances independent from the main LOD settings can also be used to further optimize the performance of the display.<ref>https://forum.flightgear.org/viewtopic.php?f=71&t=23929&p=318025&#p317946</ref>
  
 
== Use Cases ==
 
== Use Cases ==
 +
[[File:AFS-Panel-11-2020.png|thumb|MD11 Autopilot dialog implemented in Canvas to replace the existing PUI dialog.<ref>https://forum.flightgear.org/viewtopic.php?f=4&t=32822&p=377189#p377189</ref>]]
 
* Tail Cams (gear view)
 
* Tail Cams (gear view)
 
* Mirrors
 
* Mirrors
Line 20: Line 37:
 
* Missile/payload views
 
* Missile/payload views
 
* Prototyping/testing HUDs or PFDs requiring synthetic terrain to work properly
 
* Prototyping/testing HUDs or PFDs requiring synthetic terrain to work properly
 +
* rendering an orthographic view to implement [[Canvas_Tile_Element|moving maps]] that render actual FlightGear terrain/DEM
 +
* automated creation/rendering of cockpit GUI dialogs by rendering a perspective-corrected front-view of cockpit elements (MCP, AP, CDU etc) <ref>https://forum.flightgear.org/viewtopic.php?f=4&t=32822&p=377189&#p377189</ref>
  
 
== Proof of Concept ==
 
== Proof of Concept ==
 +
{{Main article|Hackathon Proposal: CompositeViewer and Canvas}}
 +
 +
All the work we did is on branch topics/cvcanvas of the main flightgear, simgear and fgdata repositories:
 +
 +
* {{repo link
 +
| site  = sf
 +
| repo  = flightgear
 +
| branch = topics/cvcanvas/~
 +
}}
 +
* {{repo link
 +
| site  = sf
 +
| repo  = simgear
 +
| branch = topics/cvcanvas/~
 +
}}
 +
* {{repo link
 +
| site  = sf
 +
| repo  = fgdata
 +
| branch = topics/cvcanvas/~
 +
}}
 +
 +
 +
<!--
 
{{Note|This is inspired by the long standing idea to render camera views to a texture for use as tailcams, gear views etc <ref>https://forum.flightgear.org/viewtopic.php?f=71&t=23929</ref>, it's based on code originally provided by F-JJTH<ref>https://forum.flightgear.org/viewtopic.php?f=71&t=23929#p317448</ref>, and subsequently reworked by Icecode GL and Hooray to integrate it with the Canvas system back in 2017<ref>https://forum.flightgear.org/viewtopic.php?f=47&t=32846&p=318046</ref>
 
{{Note|This is inspired by the long standing idea to render camera views to a texture for use as tailcams, gear views etc <ref>https://forum.flightgear.org/viewtopic.php?f=71&t=23929</ref>, it's based on code originally provided by F-JJTH<ref>https://forum.flightgear.org/viewtopic.php?f=71&t=23929#p317448</ref>, and subsequently reworked by Icecode GL and Hooray to integrate it with the Canvas system back in 2017<ref>https://forum.flightgear.org/viewtopic.php?f=47&t=32846&p=318046</ref>
 
At the time, the main issue was that the new view element needed to be explicitly updated, but that should no longer be the case thanks to James' work on the setPixel() API <ref>https://sourceforge.net/p/flightgear/mailman/message/36926486/</ref> which also means that sc::Image now has a <code>dirtyPixels()</code> API which we can reuse <ref>https://sourceforge.net/p/flightgear/simgear/ci/ad3621e23b4056e1be7e8d9eb8dd4d513455add8/</ref>
 
At the time, the main issue was that the new view element needed to be explicitly updated, but that should no longer be the case thanks to James' work on the setPixel() API <ref>https://sourceforge.net/p/flightgear/mailman/message/36926486/</ref> which also means that sc::Image now has a <code>dirtyPixels()</code> API which we can reuse <ref>https://sourceforge.net/p/flightgear/simgear/ci/ad3621e23b4056e1be7e8d9eb8dd4d513455add8/</ref>
Due to the adoption of the [[Compositor]] framework, this will probably need to reworked to be properly integrated.}}
+
Due to the adoption of the [[Compositor]] framework, this will probably need to reworked to be properly integrated.
 +
Furthermore, beginning in mid 2020 Julian Smith has started working on optional [[CompositeViewer Support]] so that independent scene views can be rendered.  }}
  
 
SimGear:
 
SimGear:
Line 415: Line 457:
 
  # Canvas
 
  # Canvas
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
-->
  
 
== Roadmap ==
 
== Roadmap ==
  
=== Use CompositeViewer {{Pending}} ===
+
=== Use CompositeViewer ===
{{See also|CompositeViewer Support}}
+
{{Progressbar|80}} {{See also|CompositeViewer Support}}  
 
+
<small>
Currently FlightGear uses only one instance of <tt>osg::Viewer</tt>, which is used by CameraGroup to manage the slave cameras. Supporting CompositeViewer would require modifying {{flightgear file|src/Viewer/fg_os_osgviewer.cxx}} and creating some kind of wrapper class that manages the CompositeViewer instance and assigns a CameraGroup to each <tt>osg::View</tt>. It's also important to note that currently all FG subsystems assume there is a single instance of [[Howto:CameraGroup talks|CameraGroup]].
+
Originally, FlightGear only used one instance of <tt>osg::Viewer</tt>, which is used by CameraGroup to manage the slave cameras. Supporting CompositeViewer did require modifying {{flightgear file|src/Viewer/fg_os_osgviewer.cxx}} to create some kind of wrapper class that manages the CompositeViewer instance and assign a CameraGroup to each <tt>osg::View</tt>. It's also important to note that currently all FG subsystems assume there is a single instance of [[Howto:CameraGroup talks|CameraGroup]].</small>
  
=== Standardize the [[View manager]] {{Pending}} ===
+
=== Standardize the View manager  ===
 +
{{Note|As of 11/2020, Jules has already created a custom view manager implementation [[StepView]] that supports multiple independent instances, so this step might be obsolete by now.}}
  
The first step would be to port the view code to SimGear so it can be used and known by the Compositor and Canvas. The view manager ({{flightgear file|src/Viewer/viewmgr.cxx}}) currently has some hardcoded assumptions, so it would either need to be rewritten to remove them or a new interface for the Views could be created specifically for the Canvas Camera View.
+
<small>The first step would be to port the view code to SimGear so it can be used and known by the Compositor and Canvas. The view manager ({{flightgear file|src/Viewer/viewmgr.cxx}}) currently has some hardcoded assumptions, so it would either need to be rewritten to remove them or a new interface for the Views could be created specifically for the Canvas Camera View.
  
 
In FlightGear, the Canvas system (which resides in SimGear) is integrated using the equivalent of a FGCanvasSystemAdapter, which provides all FG APIs to SimGear and makes the Canvas system available inside FG: {{flightgear file|src/Canvas/FGCanvasSystemAdapter.cxx}}
 
In FlightGear, the Canvas system (which resides in SimGear) is integrated using the equivalent of a FGCanvasSystemAdapter, which provides all FG APIs to SimGear and makes the Canvas system available inside FG: {{flightgear file|src/Canvas/FGCanvasSystemAdapter.cxx}}
  
 
Instead of using tied properties, the adapted view code would either used propertyObject<> or the propertyBasedMgr abstraction, so that the corresponding canvas element can continue to use well-known conventions to manipulate views: ({{flightgear file|src/Viewer/view.cxx}})
 
Instead of using tied properties, the adapted view code would either used propertyObject<> or the propertyBasedMgr abstraction, so that the corresponding canvas element can continue to use well-known conventions to manipulate views: ({{flightgear file|src/Viewer/view.cxx}})
 +
</small>
  
=== Create a Canvas Element subclass {{Pending}} ===
+
=== Create a Canvas Element subclass ===
{{Main article|Canvas_Development#Adding_a_new_Element}}
+
[[File:CanvasView-Prototyping-UI.png|thumb|[[Canvas]] GUI dialog (code at [http://wiki.flightgear.org/File_talk:CanvasView-Prototyping-UI.png]) to provide a UI <ref>https://sourceforge.net/p/flightgear/mailman/message/37163870/</ref> for the experimental [[Canvas View Camera Element]] prototyped during the [[Virtual FSweekend Hackathon 2020]], for more details see: [[Hackathon Proposal: CompositeViewer and Canvas]]]]
 +
 
 +
[[File:CompositeViewer-prototype-UI.png|thumb|[[Canvas]] based GUI dialog to create [[CompositeViewer Support|CompositeViewer views]] dynamically using a configurable [[Compositor]] pipeline.]]
 +
 
 +
{{Progressbar|30}}
 +
{{See also|Talk:Hackathon Proposal: CompositeViewer and Canvas}}
  
 
This derived class would require the following configuration:
 
This derived class would require the following configuration:
 
* [[Compositor]] to use.
 
* [[Compositor]] to use.
 +
* Scene graph to use. By default the main scene graph would be used, but an arbitrary XML file can be loaded to [[Howto:Extending Canvas to support rendering 3D models|render a custom model]]. A typical use-case would be instruments that need to [[Shuttle ADI ball|manipulate/render a 3D object]] (for which we have working code, too)
 
* [[View manager|View]] to use. This could be one of the "main" ones (i.e. the ones on the main property tree), or a locally-defined one that is only known to this Canvas Element (think FLIR).
 
* [[View manager|View]] to use. This could be one of the "main" ones (i.e. the ones on the main property tree), or a locally-defined one that is only known to this Canvas Element (think FLIR).
* Scene graph to use. By default the main scene graph would be used, but an arbitrary XML file can be loaded to [[Howto:Extending Canvas to support rendering 3D models|render a custom model]]. A typical use-case would be instruments that need to [[Shuttle ADI ball|manipulate/render a 3D object]] (for which we have working code, too)
 
* Several optimization/miscellaneous parameters like framerate cap, etc.
 
  
Simplifying a lot, this Canvas Element would be an aggregation of a Compositor instance, a View and a pointer to a <tt>osg::Group</tt> representing the scene graph to render. This setup could be replicated in {{flightgear file|src/Viewer/fg_os_osgviewer.cxx}}, with the difference of ignoring Canvas and using native windowing features from OSG.
+
Several optimization/miscellaneous parameters exposed in the form of properties per element, like:
 +
* texture size/resolution (see Fernando's comments here: [[Talk:Hackathon_Proposal:_CompositeViewer_and_Canvas#SceneGraph_Cameras_vs._Prerender_Cams]] )
 +
* OSG rendering mode (continuous vs. lazy/on demand): <code>setRunFrameScheme( osgViewer::ViewerBase::ON_DEMAND );</code> <ref>https://forum.flightgear.org/viewtopic.php?f=71&t=17650&hilit=knuq+taxiways&start=15#p166990</ref> <ref>https://stackoverflow.com/a/12259344</ref>
 +
* framerate cap via something like <code>view.setRunMaxFrameRate()</code> <ref>https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg26193.html</ref> [https://www.programmersought.com/article/23264806182/] [https://www.mail-archive.com/osg-users@openscenegraph.net/msg14033.html] (background: https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg26178.html )
 +
* [[Draw masks|node masks]]
 +
* [[Level Of Detail (LOD) Ranges|LOD]] ranges etc. <ref>https://forum.flightgear.org/viewtopic.php?f=71&t=32845&p=318027&hilit=canvas+draw+masks#p318028</ref>
 +
* PagedLOD settings for <code>PIXEL_SIZE_ON_SCREEN</code> mode [https://sourceforge.net/p/flightgear/mailman/message/36438416/] [http://alphapixel.com/wp-content/uploads/2015/04/LOD-Level-of-detail-in-OpenSceneGraph-OSG.pdf] [https://osg-users.openscenegraph.narkive.com/DNuWtUiR/pagedlods-and-pixel-to-screen-size]
 +
* whether or not to enable OSG StatsHandler per view, i.e. for troubleshooting per view
 +
* whether or not to enable event handling: "Extra view windows don't yet handle events so one cannot change the view angle/zoom after creation." <ref>https://sourceforge.net/p/flightgear/mailman/message/37161269/</ref>
 +
 
 +
Note, we can use the view's Camera's LODScale to adjust the which level of LOD child is selected for each view.<ref>https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg66861.html</ref>
 +
Another common suggestion is to control the update rate of slave cameras by inserting them into the scene graph as needed, and then use a PostDrawCallback for the camera that removes the
 +
camera again when it is no longer needed.<ref>https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg66868.html</ref>
 +
Robert (OSG) states that most apps should manage the frame for their own applications and data.  The OSG is a general purpose scene graph rather than a domain specific IG.  It gives you the tools to do your
 +
job, but it doesn't do it all for you.  The run() method stuff available in OSG/Viewer is really only for entry level app development.<ref>https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg26257.html</ref>
 +
More sophisticated types of frame rate management are well beyond what  the viewer convenience methods like run are supposed to handle.  People  should expect to roll your own frame management for this type of app.<ref>https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg26184.html</ref>
 +
 
 +
 
 +
Simplifying a lot, this Canvas Element would be an aggregation of:
 +
* a Compositor instance / effect scheme
 +
* a View  
 +
* a pointer to a <tt>osg::Group</tt> representing the scene graph to render (e.g. alias or filename based).
 +
 
 +
This setup could be replicated in {{flightgear file|src/Viewer/fg_os_osgviewer.cxx}}, with the difference of ignoring Canvas and using native windowing features from OSG (see Jules' fgcommands to clone a view and open a dedicated GraphicsContext/window).
  
 
== Related ==
 
== Related ==
Line 447: Line 519:
 
* {{forum link|t=32846|text=Canvas::View Development}}
 
* {{forum link|t=32846|text=Canvas::View Development}}
 
* {{forum link|t=23929|text=Gear View in Cockpit}}
 
* {{forum link|t=23929|text=Gear View in Cockpit}}
 +
* {{forum link|t=32845|text=Canvas::View element: performance/optimizations}}
 
* {{forum link|t=20057|text=Rear-View mirror}}
 
* {{forum link|t=20057|text=Rear-View mirror}}
 
* {{forum link|t=18905|text=Progress on synthetic terrain}}
 
* {{forum link|t=18905|text=Progress on synthetic terrain}}
 
* {{forum link|t=17184|text=Instruments with heightmaps}}
 
* {{forum link|t=17184|text=Instruments with heightmaps}}
 +
 +
== References ==
 +
{{Appendix}}
  
 
[[Category:Canvas Element Proposals]]
 
[[Category:Canvas Element Proposals]]

Latest revision as of 11:23, 3 December 2020

Cquote1.png It would make sense to integrate all efforts (CompositeViewer, Canvas and Compositor) to create a Canvas Element that can render an arbitrary view.

Some time ago I wrote what I think are the required steps to get something like cockpit mirrors working: #Roadmap

The main showstopper was having CompositeViewer Support, but since that effort is underway, all the parts are there for someone willing to dive in.


— Fernando García Liñán  (2020-08-20 14:27:58). Independent view windows.
(powered by Instant-Cquotes)
Cquote2.png
Canvas Camera Views
Cvcanvas-777.jpeg
Started in 11/2020
Description Rendering arbitrary scene views to a texture/Canvas
Maintainer(s) none
Contributor(s) Cgdae, Icecode
Status experimental prototype

Background

Several aircraft developers have manifested their interest in being able to render scene views to a texture (RTT) Render target This is a link to a Wikipedia article and use it inside cockpits as mirrors, external cameras (so called tail cams) and other uses.

With the Compositor framework it's possible to configure custom render pipelines in XML space, which together with Effect schemes allow complete control over how a rendered scene looks. A synthetic terrain can be rendered by assigning an Effect scheme to a scene type pass, which textures all terrain with a brown checkerboard pattern and paints the sky in a solid blue color. All other objects are ignored and aren't rendered, saving a lot of computation time. Custom render distances independent from the main LOD settings can also be used to further optimize the performance of the display.[1]

Use Cases

MD11 Autopilot dialog implemented in Canvas to replace the existing PUI dialog.[2]
  • Tail Cams (gear view)
  • Mirrors
  • Dynamic in-sim view configuration
  • On demand creation of views and windows (e.g. FGCamera previews)
  • Missile/payload views
  • Prototyping/testing HUDs or PFDs requiring synthetic terrain to work properly
  • rendering an orthographic view to implement moving maps that render actual FlightGear terrain/DEM
  • automated creation/rendering of cockpit GUI dialogs by rendering a perspective-corrected front-view of cockpit elements (MCP, AP, CDU etc) [3]

Proof of Concept

1rightarrow.png See Hackathon Proposal: CompositeViewer and Canvas for the main article about this subject.

All the work we did is on branch topics/cvcanvas of the main flightgear, simgear and fgdata repositories:


Roadmap

Use CompositeViewer

80}% completed

Originally, FlightGear only used one instance of osg::Viewer, which is used by CameraGroup to manage the slave cameras. Supporting CompositeViewer did require modifying flightgear/src/Viewer/fg_os_osgviewer.cxx to create some kind of wrapper class that manages the CompositeViewer instance and assign a CameraGroup to each osg::View. It's also important to note that currently all FG subsystems assume there is a single instance of CameraGroup.

Standardize the View manager

Note  As of 11/2020, Jules has already created a custom view manager implementation StepView that supports multiple independent instances, so this step might be obsolete by now.

The first step would be to port the view code to SimGear so it can be used and known by the Compositor and Canvas. The view manager (flightgear/src/Viewer/viewmgr.cxx) currently has some hardcoded assumptions, so it would either need to be rewritten to remove them or a new interface for the Views could be created specifically for the Canvas Camera View.

In FlightGear, the Canvas system (which resides in SimGear) is integrated using the equivalent of a FGCanvasSystemAdapter, which provides all FG APIs to SimGear and makes the Canvas system available inside FG: flightgear/src/Canvas/FGCanvasSystemAdapter.cxx

Instead of using tied properties, the adapted view code would either used propertyObject<> or the propertyBasedMgr abstraction, so that the corresponding canvas element can continue to use well-known conventions to manipulate views: (flightgear/src/Viewer/view.cxx)

Create a Canvas Element subclass

Canvas GUI dialog (code at [1]) to provide a UI [4] for the experimental Canvas View Camera Element prototyped during the Virtual FSweekend Hackathon 2020, for more details see: Hackathon Proposal: CompositeViewer and Canvas
Canvas based GUI dialog to create CompositeViewer views dynamically using a configurable Compositor pipeline.

30}% completed

This derived class would require the following configuration:

  • Compositor to use.
  • Scene graph to use. By default the main scene graph would be used, but an arbitrary XML file can be loaded to render a custom model. A typical use-case would be instruments that need to manipulate/render a 3D object (for which we have working code, too)
  • View to use. This could be one of the "main" ones (i.e. the ones on the main property tree), or a locally-defined one that is only known to this Canvas Element (think FLIR).

Several optimization/miscellaneous parameters exposed in the form of properties per element, like:

Note, we can use the view's Camera's LODScale to adjust the which level of LOD child is selected for each view.[10] Another common suggestion is to control the update rate of slave cameras by inserting them into the scene graph as needed, and then use a PostDrawCallback for the camera that removes the camera again when it is no longer needed.[11] Robert (OSG) states that most apps should manage the frame for their own applications and data. The OSG is a general purpose scene graph rather than a domain specific IG. It gives you the tools to do your job, but it doesn't do it all for you. The run() method stuff available in OSG/Viewer is really only for entry level app development.[12] More sophisticated types of frame rate management are well beyond what the viewer convenience methods like run are supposed to handle. People should expect to roll your own frame management for this type of app.[13]


Simplifying a lot, this Canvas Element would be an aggregation of:

  • a Compositor instance / effect scheme
  • a View
  • a pointer to a osg::Group representing the scene graph to render (e.g. alias or filename based).

This setup could be replicated in flightgear/src/Viewer/fg_os_osgviewer.cxx, with the difference of ignoring Canvas and using native windowing features from OSG (see Jules' fgcommands to clone a view and open a dedicated GraphicsContext/window).

Related

Forum topics

References

References