Hackathon Proposal: CompositeViewer and Canvas

From FlightGear wiki
Jump to navigation Jump to search


Title: CompositeViewer and Canvas

Cvcanvas-777.jpeg
Potential mentors: Cgdae, Fernando,
Intro: (short intro)
Interested Parties:
Status: Post-hackathon write-up (11-2020)

Summary

We came across some tricky issues, but made lots of progress and ended up with a couple of working demos.

The code changes we made are very hacky and will need a fair amount of cleaning up before we think of pushing to next.

All the work we did is on branch topics/cvcanvas of the main flightgear, simgear and fgdata repositories:

Note: Whatever we end up with on next will not be based on these branches, but will instead be written independently, albeit using the knowledge gained in the hackathon. So people are better off waiting for Canvas Views to be implemented properly on next.[1]

Demo videos

  • cvcanvas-777-tailcam.webm Canvas View used for cockpit display of tail camera on a 777.
  • cvcanvas-demo.ogv Canvas View on the panel of a Harrier-GR3 showing a view from the camera pod under the aircraft, plus a flying billboard and some extra view windows.

Things we learnt or talked about

  • We ran into some minor problems making an object in the aircraft model serve as a placement for a Canvas View:
    • The object surface should uses a material with emis 1 1 1, otherwise it seems to only shows up when lit up by sunlight. This can be confusing if one is testing while one's aircraft is facing into the sun.
    • When the surface(s) of the object reference vertices, they need to specify differing texture coordinates (not just 0 0 for each vertice reference), otherwise nothing shows up.
  • The textures that Canvas uses are tied to graphics contexts, so aren't shared between windows. This means that extra view windows don't show in-cockpit displays. It was suggested that sharing textures was actually possible, as long as the graphics contexts are on the same GPU. Done Done (08/2021)
  • We can't yet change a Canvas View to show a different view, e.g. swap between tail-camera and gear-camera. The underlying CompositeViewer has full support for destroying and creating new views, and Canvas supports adding/removing elements, so hopefully this will be relatively easy to address.
  • Frame rates might be significantly affected even if a Canvas View is only showing a small part of the scene.
  • Apparently it should be possible to make a Canvas View show different effects, such as infra-red vision or artificial terrain. The implementation internally uses a Compositor instance to manage the view's rendering pipeline, so it is possible to completely change the Effects and shaders that the view uses.
  • Canvas cameras being in the scene graph is a problem in the long run and should be revisited. The scene graph cameras are being rendered for each slave camera present in the rendering pipeline (i.e. each Compositor render pass), so there are definitely other motivations to make this happen separate from this particular proposal: Post FlightGear 2020.2 LTS changes#Canvas Tick icon Fixed since FlightGear commit 83b0a3 Tick icon Fixed since SimGear commit a97e14
Summary: CompositeViewer Support already allows us to create multiple top-level windows showing different views of the same scenery.

We want to extend the CompositeViewer topic branch to render different views of the scenery to a custom Canvas element subclass.

Then by adding the new canvas element to a canvas shown in a cockpit we will be able to implement things like:

  • Modern avionics:
    • Exterior views (tail cam, gear cam etc).
    • Synthetic terrain.
    • moving map displays displaying actual FlightGear terrain data by rendering an orthographic view of the scene using custom LOD and node masks (for the time being FlightGear avionics tend to render pre-downloaded images/PDF files or data external web services that do not match FlightGear scenery) .
  • Rear-view mirrors.
  • Shuttle RMS arm/END EFF mode [2] [3] etc) [4] [5]
Background:
Details:
Ideas:
Required skills: C++
Learning Opportunities:

  • OpenSceneGraph
  • Canvas
  • CompositeViewer
  • Compositor
Notes: Information about Canvas from Hooray

It would make sense to reach out to Jules (CompositeViewer) and Fernando (Compositor) to learn how to best approach this today.

The original set of patches (touching SimGear and fgdata) implements a new Canvas::Element by creating a sub-class named Canvas::View. The meat of it is in the constructor, i.e. Canvas::View::View(), where an off-screen camera (RTT/FBO) is set up, the FGCanvasSystemAdapter file has been extended to provide access to the FlightGear view manager to compute/obtain the view-specific view matrix, which is then used by this new canvas view element to update the offscreen camera in Canvas::View::update() accordingly.

BTW: This is also a good way to stress-test the renderer, as new cameras can be easily added to the scene at runtime, so that the impact of doing so can be easily measured.

the patch is experimental, it will basically look up a view and dynamically add a slave camera to the renderer that renders the whole thing to a Canvas, a Canvas is a fancy word for a RTT/FBO context in FlightGear that can be updated by using a property-based API built on top of the property tree in the form of events/signals that are represented via listeners.Which is to say each Canvas has a handful of well-defined property names (and types) that it is watching to handle "events" - think stuff like changing the sie/view port etc. And then there is a single top-level root group, which serves as the top-level element to keep other Canvas elements.A Canvas element is nothing more than a rendering primitive that the Canvas system can handle - e.g. stuff like a raster image can be added to a Canvas group, a text string/font, and 2D drawing primitives in the form of OpenVG instrutions mapped to ShivaVG. And that's basically about it (with a few exceptions that handle use-case specific stuff like 2D mapping/charts).Apart from that, the main thing to keep in mind is that a Canvas is really just a FBO - i.e. an invisible RTT context - to become actually visible, you need to add a so called "placement" - this tells the rendering engine to look up a certain canvas and add it to the scene/cockpit or the GUI (dialogs/windows).

Canvas-view-element-prototype-by-icecode gl.png

The basic boilerplate needed to add/register a new Canvas element can be seen below:


You will want to add a new Canvas::Element subclass whenever you want to add support for features which cannot be currently expressed easily (or efficiently) using existing means/canvas drawing primitives (i.e. via existing elements and scripting space frameworks).

For example, this may involve projects requiring camera support, i.e. rendering scenery views to a texture, rendering 3D models to a texture or doing a complete moving map with terrain elevations/height maps (even though the latter could be implemented by sub-classing Canvas::Image to some degree).

Another good example for implementing new elements is rendering file formats like PDF, 3d models or ESRI shape files.

To create a new element, you need to create a new child class which inherits from Canvas::Element base class (or any of its child-classes, e.g. Canvas::Image) and implement the interface of the parent class by providing/overriding the correspond virtual methods.

To add a new element, these are the main steps:

  • Set up a working build environment (including simgear): Building FlightGear
  • update/pull simgear,flightgear and fgdata
  • check out a new set of topic branches for each repo: git checkout -b topic/canvas-Model
  • Navigate to $SG_SRC/canvas/elements
  • Create a new set of files Model.cxx/.hxx (as per Adding a new Canvas element)
  • add them to $SG_SRC/canvas/elements/CMakeLists.txt (as per Developing using CMake)
  • edit $SG_SRC/canvas/elements/CanvasGroup.cxx to register your new element (header and staticInit)
  • begin replacing the stubs with your own C++ code
  • map the corresponding OSG/library APIs to properties/events understood by the Canvas element (see the valueChanged() and update() methods)
  • alternatively, consider using dedicated Nasal/CppBind bindings

Below, you can find patches illustrating how to approach each of these steps using boilerplate code, which you will need to customize/replace accordingly:



Model: Canvas Model (required cmake changes)


Model: Canvas Model (minimal C++ changes)


Model: Canvas Model (registering the new element)


Model: Canvas Model $FG_ROOT/Nasal/canvas/api.nas changes


Model: Canvas Model (testing the new element using the Nasal Console)
References
Retrieved from "https://wiki.flightgear.org/w/index.php?title=Hackathon_Proposal:_CompositeViewer_and_Canvas&oldid=132974"