CompositeViewer support

From FlightGear wiki
Jump to navigation Jump to search


CompositeViewer
Fg-cv-textures2.jpeg
Started in 2020 July
Description Support for multiple independent scene views
Contributor(s) Julian Smith
Status Merged into next 2020 Nov [1]



CompositeViewer

CompositeViewer allows multiple independent views of the Flightgear scene, e.g. in independent top-level windows.

Status updates

  • 2020-10-5: Issues with window resize/close appear to be bugs in OpenSceneGraph-3.4, and are fixed by building with OpenSceneGraph-3.6.
  • 2020-9-27: Extra view windows now show textures and clouds etc, and rendering appears to be identical to the main view. This works by creating a new Compositor instance for each extra view window, and calling its update() method each frame.

General information

  • Use of CompositeViewer is enabled at runtime with: --composite-viewer=1
  • Video showing extra view windows and initial implementation of Canvas View: cvcanvas-demo.ogv
  • When enabled, CompositeViewer requires OpenSceneGraph-3.6 to work well.
  • Extra view windows:
    • The UI for creating new view windows uses new View menu items to clone the current view or take eye/target points from two earlier views. This seems to be convenient and avoids the need for a separate dialogue box or similar.
    • Can be resized.
    • Can clone Pilot View, Helicopter View, Chase View or Tower View.
    • One can create an extra view window that keeps two aircraft in view with one in the foreground. E.g. see this video from 2020-9-6 (prior to getting textures working): http://op59.net/fg-cv-demo3.mpeg
    • One can create an extra view window that uses the eye points from two recent views as eye and target. For example this allows a window to show a view from one aircraft to another.
    • There doesn't seem to be any noticeable speed penalty if no extra view windows are opened.
    • Use a new view system called Sview (step view), which allows multiple instances and dynamic specification of eye and target points.
  • CompositeViewer support will allow us to render a view to a canvas and implement things like rear-view mirrors etc - see: Canvas_View_Camera_Element.

Problems

  • for the time being, use of reset/re-init and aggressive OSG threading options seems to cause stability problems even without having cloned any views [3]
  • Using OpenSceneGraph-3.6 causes problems elsewhere: OSGText Issues (under investigation as of 11/2020) [4]

Current limitations

  • Extra view windows don't have event handling, so for example one cannot change the angle of a cloned Helicopter view.
  • Sview does not support some views, e.g. Tower View AGL, Fly-past view.
  • Sview does not support aircraft-specific views.

Issues when using OpenSceneGraph-3.4

OpenSceneGraph-3.4 seems to get event handling wrong which causes problems if we open extra view windows - resize/close events get sent to our main window event handler which confuses things. There is currently no workaround for this.

These problems appear to be fixed in OpenSceneGraph-3.6.

Symptoms with OpenSceneGraph-3.4 include:

  • Creating extra view windows can cause the main window to seemingly think it has same size as the new window, so menubar and dialogues are shown partially or not at all. A manual resize of the main window seems to sort things out.
  • Attempting to closing an extra view windows sometimes closes a different extra view window.
  • Attempting to closing an extra view windows sometimes closes Flightgear down because the code thinks the main window has been closed.

Code

Latest code can be found on the 'next' branch of flightgear, simgear and fgdata.


Background

  • osgViewer::CompositeViewer is designed for applications that have multiple Views and your usage model fits this perfectly. The only thing to be careful of is when you are adding and removing View's from the CompositeViewer you should do is calling stopThreading() on the viewer prior to adding or removing views, then call startThreading() afterwards. If you are running SingleThreaded or CullDrawThreadPerContext you won't need to worry about stop and starting threads.[5]
  • As a general approach, if you want multiple View's which have their own or share Scene's then the appropriate class to use is CompositeViewer as it's written specifically for this purpose.[6]
  • The usual way to manage multiple window views of a single scene graph is to use a CompositeViewer with multiple View's each view using its own or sharing a graphics window. [7]
  • What CompositeViewer provides is not so much performance improvement across the board, but rather far better granularity of design.[8]
  • The CompositeViewer and Viewer should have exactly the same performance characteristics w.r.t managing multiple cameras - as it's exactly the same ViewerBase code underneath that is managing all the threading and graphics rendering.[9]
  • CompositeViewer and Viewer share much of their implementation, the only key difference is that Viewer "is a" View, while CompositeViewer has a list of Views. All the event handling, camera manipulator and scene graph setting is done a the View level so has identical API to access. [10]

Views

For example, you could have a working set of View's that share the scene graph, and share the same graphics context. All these Views would be added to the CompositeViewer. At start up these View's would be disabled by setting their View Camera's NodeMask to 0x0.

When you need to render a View for a client you'd enable a View of one is attached and not enabled, enable by setting the Camera's NodeMask to 0xffffffff. If you run out of View then simply create a new one for the purpose. Potentially you could do this on demand - so have none at start up.

When you no longer need a View you could remove it, but it's lighter to just disable it via the NodeMask trick.

With this approach you aren't creating/deleting graphics contexts, will lower memory usage and you'll get better performance.[11]

Note: the slave osg::Camera aren't direct children of the View's master osg::Camera, but they have their view and projection matrices updated from the master camera.

Note II: osgViewer only threads Camera that are in the viewer, not ones enclosed in the scene graph, so your Camera in Camera won't thread.[12]

  • The right way to remove a view is outside of frame(). Not from an event handler from within the view, this will crash as you'll be deleting the object you are doing the work from.[13]

DatabasePager

  • The OSG has an database paging class call osgDB::DatabasePager that is built into the osgViewer::Viewer/CompositeViewer that will automatically load databases and merge them with the main scene graph during the update traversal. This is built around the osg::PagedLOD class, but you can also use the osg::ProxyNode.[14]
  • Normally one should ever need to worry about constructing or destructing the DatabasePager, it should happen behinds the scenes managed by osgViewer. osgViewer::Scene is used internally by osgViewer to manage one DatabasePager per scene graph. The Scene object will be shared automatically between Views if you assign the same Node pointer then you call View::setSceneData. This sharing is done automatically for you. When a Scene object is destructed it's DatabasePager will be destructed if no other references to it exist. If you have multiple Views that you should be using CompositeViewer, not multiple Viewer. If you are creating and destroying views regularly then you are probably best to enable/disable them by setting the View's master Camera's NodeMask to 0x0 and back to 0xffffffff, as this will switch off rendering but keep the backend around ready to be re-enabled.[15]
  • osgViewer automatically shares a single DatabasePager between Views when the Views share the same scene graph. This is required to avoid inconsistencies in the scene graph causing errors. Sharing a single DatabasePager doesn't prevent that pager from handling multiple viewpoints at the same time, if fact it knows nothing about viewpoints, it only handles database requests for tiles, so it totally agnostic to how you manage your viewpoints. Everything should basically just work out of the box without any need for specific settings from yourself.[16]

Threading

  • osgViewer::CompositeViewer runs all of the views synchronously - one frame() call dispatches update, event, cull and draw traversals for all the views. [17]
  • osgViewer::Viewer/CompositeViewer all have the DatabasePager built into them, and will automatically run the database pager thread on demand and take care of all the operations required to manage a paged database.[18]
  • OpenGL doesn't support multi-threading within a single graphics context, so you are constrained to doing the rendering for each context in a single thread. The threading models that the OSG provides reflect this, enabling threading of the update, event and cull traversals in parallel with the draw thread. [19]
  • if any code executed by the cull or draw threads (such as your own callbacks or custom nodes) isn't thread safe, then you must use SingleThreaded. [20]
  • as long as you have two GPU's the most efficient way to drive them should be multi-threaded - there is a caveat though, hardware and drivers aren't always up to scratch, and even then they should be able to manage the multi-threads and multi-gpus seemless they fail too.[21]

Sharing scenes

  • Each View has one scene graph, and can share its scene graph between other instances of View. The View can also share the same GraphicsWindow, or have its own GraphicsWindow. The View also has a master Camera, and an optional list of slave Camera so you can scale from simple views up to complete distortion correction or multiple display output setups. Each View has its own event handlers and cameras handlers. Its extremely flexible and configurable. [22]
  • sharing a scene between View's is OK within one CompositeViewer as they will Views on the same scene will share the same FrameStamp i.e. there will be all at the same point in time. Sharing one scene between multiple Viewers will hit up against the problem that in one set of traversals the scene graph is one time and then the traversals from the other viewer will try to change the time back - and likely to cause a mess. This timing issue isn't likely to cause problems with high level rendering though - it should just mess up things like particle systems and sequences.[23]
  • The best thing we could do would be to create a single GraphicsWindow and then share this between all our Views, we then won't have any problems with rendering order and sharing of textures or FBOs as it'll all be on one graphics context. See the sogcompositeviewer example for how to set up the Views/Camera & GraphicsWindow.[24]
  • Sharing a single window between multiple views is demonstated in the osgcompositeviewer example - you simply assign the same GraphicsWindow to the Camera's in each of the Views. You change views you can stop the viewer threads and then add/remove views you need then restart the threading, this will drop a few frames though due to stopping/start of threads. The other way is to switch off the rendering of the view by setting its Camera's NodeMask to 0x0 to disable it.[25]

Context sharing

  • It's possible to share contexts in the OSG [...] As for general desirability of share GL objects between contexts, yes it can reduce memory usage, but it forces you to use the OSG single threaded otherwise two contexts will be contended for the same resources that deliberately aren't mutex locked for performance reasons. There is also on a limited set of cases where drivers/hardware will actually share OpenGL contexts. [26]
  • Neither the OSG or OpenGL can provide thread safe sharing of GL objects when sharing contexts. If you want to run multiple context with multiple threads you will have to keep these contexts independent. [27]
  • If all your views share the same graphics context then it's only possible to single thread the draw dispatch. With this usage you'll be able to use DrawThreadPerContext which will allow the update and cull traversals to overlap the previous frames draw traversal, but overlap will only extend from the dispatch of the last dynamic object in the draw traversal being dispatched. If you have a large static scene then the overlap can be the whole frame, if you have lots of StateSet and Geometry with a DataVariance of DYNAMIC then the scope for threading is reduced, and at worst case will essentially be serialized and equivilant to SingleThreaded. Things that affect the draw traversals sometimes need draw threads to be stopped completely. Things like adding views to a CompositeViewer, or changing the graphics context on a camera, or things like that. It's pretty rare you need to do this. It's also pretty costly, because stopThreading() will only return once the draw threads have been stopped and deleted, and startThreading() only returns once new draw threads have been created and started.[28]
  • an OpenGL context is tied to a single window or pixel buffer. [29]
  • Sharing contexts is also something the forces a few limits on how you use the graphics contexts, such as it's only really safe to use them single threaded. [30]
  • If you are creating new graphics contexts and applying and old scene graph to it then you can't use the Texture::setUnRefImageDataAfterApply(true) feature of osg::Texture as this will discard the imagery once it's applied to all the graphics contexts that it knows about. [31] The typical problem is that the scene graph has been set up to unref texture images after apply so when it comes to reloading the texture images there aren't the to download. [32]

Image Sharing

  • Sharing of images is possible by using a frame buffer copy to osg::Image, or just having multiple FBO's all within one graphics context. If you can have the frames all done synchronously then perhaps you could have one frame loop and just disable the cameras via camera->setNodeMask(0x0); that you don't need updating on each frame, i.e. main viewer runs at 60Hz, and the other RTT cameras run at 20Hrz so get update on frame in 3.[33]
  • ImageStream is updated by a background thread. The xine-lib and QuickTime plugins both subclass from ImageStream and OpenThreads::Thread to provide a class thatautomatically runs updates on itself - this thread updates the image data on the ImageStream and then calls dirty to tell the rendering thread that it needs to download the data to any associated texture.[34]
  • It's better to update the data stored in the Image directly and call dirty, or to allocate the image memory separately and disable the deletion of the data on the image so the osg::Image(Stream) never calls delete on the data.[35]

Motivation

Experimental CompositeViewer Support showing 3 cloned views, with Draw masks set (only skydome shown) + OSG stats (~ 120 fps/window) while using CullThreadPerCameraDrawThreadPerContext [36]

Until mid-2020, FlightGear only supported one view position at a time. Multiple independent view positions (e.g. one screen for the tower view and a second screen for the plane) would complicate a "locked to cache" flag quite a lot...[37].

Aircraft can define their own views and so on. But only one view can be active at a time. So no matter how many windows and cameras you define in Defaults.xml, they all are relative to the current view in FG (i.e. cockpit, tower...). [38]

Back in 2008, Tim Moore provided a patch (mailing lists search for CameraGroup FlightGear Mailing Lists) to use the osgViewer class to set up windows, manage the main camera, etc. [39]

However, these windows have to use the same camera group as the main window so can only show the view from the same eye position, though typically at a different angle/offset so that one can emulate things like side windows of a cockpit displayed in a different window or monitor.[40]


Mathias Fröhlich used the slave camera feature of osgViewer to provide a "video wall" style of multiple displays that was demonstrated at LinuxTag for years. Later on, Tim generalized this to support general monitor arrangements (like a panoramic arc) and general combinations of screens and graphics cards. [41]

The default OSG model is that slave cameras are different views offset from a common viewpoint. This is easy to understand when considering a camera's view matrix, but not necessarily intuitive when thinking about the projection matrix. Because FG has its own view system we mostly treat the slaves as independent. It seems that most other uses of cameras during rendering -- for example, render to texture cameras for effects -- are best handled by slave cameras with independent views as well.[42]

People requiring multiple independent views on the same scenery, e.g. cockpit and tower view [...] these each need their own camera groups and so require OSG's CompositeViewer.[43]

And that's not really supported by the current architecture, neither by the tile cache nor by osgViewer::Viewer. We would need to move to a CompositeViewer model, which supports several scene graphs, and rely completely on the osg database paging machinery.[44]

That would require a change in current fg architecture to use a CompositeViewer instead of a single Viewer, but we're contemplating that anyway.[45]

The cameras in a camera group don't need to render directly to the screen. They can render to a texture which can be used either in the scene, like in a video screen in the instrument panel, or for distortion correction in a projected or dome environment. [46]

Open Scene Graph supports a CompositeViewer object that supports rendering from several widely separated viewpoints, complete with support for multiple terrain pager threads. We could move to CompositeViewer and support simultaneous views from e.g., the tower, AI models, drones, etc.[47]

Neither of these are supported at the present time, but it would be a good project. We would have to start using a different OSG class, CompositeViewer, to support multiple views from independent view points. Our terrain pager would need a complete overhaul to use the PagedLOD scheme of OSG, and the Flightgear View manager would need to be aware multiple active views.[48]

Related

Docs

Discussions

Search

References

References
  1. https://sourceforge.net/p/flightgear/mailman/message/37158652/
  2. https://sourceforge.net/p/flightgear/mailman/message/37158652/
  3. https://forum.flightgear.org/viewtopic.php?f=6&t=38334
  4. https://sourceforge.net/p/flightgear/mailman/message/37157550/
  5. https://groups.google.com/d/msg/osg-users/7OojxLpBGdw/mzorZe3rKwEJ
  6. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg70958.html
  7. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg61172.html
  8. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg01391.html
  9. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg22273.html
  10. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg17372.html
  11. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg63061.html
  12. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg11636.html
  13. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg35639.html
  14. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg68474.html
  15. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg63057.html
  16. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg34846.html
  17. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg18308.html
  18. https://groups.google.com/d/msg/osg-users/eMh8DfsKZhQ/F1V5BZS3hLEJ
  19. https://groups.google.com/d/msg/osg-users/iMJF1vp0U48/9RSD3DlPAAAJ<
  20. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg09620.html
  21. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg18332.html
  22. https://groups.google.com/d/msg/osg-users/kmSNMm6w008/zBBunL-VRJsJ
  23. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg07305.html
  24. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg16029.html
  25. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg18291.html
  26. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg32676.html
  27. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg75584.html
  28. https://groups.google.com/d/msg/osg-users/chUK1HzAfLA/QW4JPCgXfoQJ
  29. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg75586.html
  30. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg16791.html
  31. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg32658.html
  32. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg34284.html
  33. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg07360.html
  34. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg11405.html
  35. https://www.mail-archive.com/osg-users@lists.openscenegraph.org/msg11440.html
  36. https://forum.flightgear.org/viewtopic.php?f=6&t=38334
  37. http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg28864.html
  38. http://forum.flightgear.org/viewtopic.php?p=146136#p146136
  39. https://sourceforge.net/p/flightgear/mailman/message/19718339/
  40. https://sourceforge.net/p/flightgear/mailman/message/37059117/
  41. https://sourceforge.net/p/flightgear/mailman/message/24811861/
  42. https://sourceforge.net/p/flightgear/mailman/message/36295606/
  43. https://sourceforge.net/p/flightgear/mailman/message/37059117/
  44. http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg28869.html
  45. http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg17263.html
  46. http://sourceforge.net/p/flightgear/mailman/message/19718339/
  47. http://sourceforge.net/p/flightgear/mailman/message/19718339/
  48. =http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg27134.html