FlightGear wiki - User contributions [en]

Virtual Reality

2023-10-29T16:18:36Z

Gastcan:

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 22nd August 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
*SteamVR, Linux, AMD graphics:
**Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
**try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
*Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
***There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
*SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
**Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
**May need --disable-sentry
**Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
***SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
***SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{not done}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|60}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**Geometry shader multi viewport {{not done}}
**OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
**Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{not done}}
***Sound reference frame (user's head or room)? {{not done}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
**Windows/OpenGL graphics bindings {{done}}
**Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2023-10-29T16:17:34Z

Gastcan: Added note for monado basalt

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 22nd August 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provide 6DoF).

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
*SteamVR, Linux, AMD graphics:
**Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
**try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
*Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
***There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
*SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
**Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
**May need --disable-sentry
**Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
***SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
***SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{not done}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|60}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**Geometry shader multi viewport {{not done}}
**OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
**Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{not done}}
***Sound reference frame (user's head or room)? {{not done}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
**Windows/OpenGL graphics bindings {{done}}
**Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2023-08-14T23:32:13Z

Gastcan: Strike-out comment

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 22nd August 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s>

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
*SteamVR, Linux, AMD graphics:
**Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
**try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
*Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
***There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
*SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
**Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
**May need --disable-sentry
**Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
***SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
***SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{not done}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|60}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**Geometry shader multi viewport {{not done}}
**OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
**Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{not done}}
***Sound reference frame (user's head or room)? {{not done}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
**Windows/OpenGL graphics bindings {{done}}
**Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2023-08-14T16:30:38Z

Gastcan: Added working configuration

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 22nd August 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
** The viewpoint/perspective drifts clockwise, to the right.

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
* If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make launcher accessible to VR user {{not done}}
** Make splash screen VR friendly {{done}}
** Make GUI accessible to VR user {{not done}}
** Make aircraft center accessible to VR user {{not done}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
** Show splash screen as quad layer for runtime to composite {{done}}
** Cube map/quads background provided by aircraft {{progressbar|20}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Launcher {{not done}}
** Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
** Transition to flightgear {{not done}}
** Transition to aircraft center with VR already started {{not done}}
** Transition back to flightgear from aircraft center {{not done}}
** More VR friendly environment
*** A simple background cubemap, shared with splash {{not done}}
*** A hanger (like xplane11)? {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
** Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
*** Grasp surfaces {{{progressbar|90}}
*** Force feedback for haptic gloves {{not done}}
** Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
*** Inverse kinematics of translate and rotate animations {{done}}
*** Ability to specify frictions {{not done}}
*** Ability to specify ranges (where can't be deduced) {{not done}}
*** Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

=== Aircraft compatibility ===

This section is largely about how usable controls are in VR with direct manipulation.

==== [[Cessna 172P]] ====
* Controls
** Flaps {{progressbar|90}}
*** kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
*** correct relative friction {{not done}}
** Opening of doors {{not done}}
*** reversing of gain expression {{not done}}

==== [[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
** External power on (GUI required) {{not done}}
** External power off (GUI required) {{not done}}
** Throttle roll (middle mouse drag required) {{not done}}
* Take off & flight
** Arms & hands get a bit in the way {{not done}}
* Controls
** Engine cutoff lever {{not done}}
*** Uses nasal, needs pick animation binding of some sort
** ELT {{progressbar|50}}
*** Both /ELT/test and /ELT/armed can be set to bool
*** Both can get set though
** Doors & hatches {{not done}}
*** Add limits

==== [[McDonnell Douglas F-15 Eagle]] ====
* Model
** Floating switches are very noticeable {{not done}}
** HUD should be at infinity {{not done}}

==== [[Robinson R22]] ====
* Controls
** Tilting T-bar {{progressbar|90}}
*** limit range? {{not done}}
** Collective awkward to get at {{not done}}
** Throttle roll & governor switch? {{not done}}

==== [[Robinson R44]] ====
* Controls
** Tilting T-bar {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2023-07-11T16:31:04Z

Gastcan: Corrected the date

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 22nd August 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, SteamVR, Windows 10 -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
* If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make launcher accessible to VR user {{not done}}
** Make splash screen VR friendly {{done}}
** Make GUI accessible to VR user {{not done}}
** Make aircraft center accessible to VR user {{not done}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
** Show splash screen as quad layer for runtime to composite {{done}}
** Cube map/quads background provided by aircraft {{progressbar|20}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Launcher {{not done}}
** Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
** Transition to flightgear {{not done}}
** Transition to aircraft center with VR already started {{not done}}
** Transition back to flightgear from aircraft center {{not done}}
** More VR friendly environment
*** A simple background cubemap, shared with splash {{not done}}
*** A hanger (like xplane11)? {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
** Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
*** Grasp surfaces {{{progressbar|90}}
*** Force feedback for haptic gloves {{not done}}
** Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
*** Inverse kinematics of translate and rotate animations {{done}}
*** Ability to specify frictions {{not done}}
*** Ability to specify ranges (where can't be deduced) {{not done}}
*** Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

=== Aircraft compatibility ===

This section is largely about how usable controls are in VR with direct manipulation.

==== [[Cessna 172P]] ====
* Controls
** Flaps {{progressbar|90}}
*** kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
*** correct relative friction {{not done}}
** Opening of doors {{not done}}
*** reversing of gain expression {{not done}}

==== [[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
** External power on (GUI required) {{not done}}
** External power off (GUI required) {{not done}}
** Throttle roll (middle mouse drag required) {{not done}}
* Take off & flight
** Arms & hands get a bit in the way {{not done}}
* Controls
** Engine cutoff lever {{not done}}
*** Uses nasal, needs pick animation binding of some sort
** ELT {{progressbar|50}}
*** Both /ELT/test and /ELT/armed can be set to bool
*** Both can get set though
** Doors & hatches {{not done}}
*** Add limits

==== [[McDonnell Douglas F-15 Eagle]] ====
* Model
** Floating switches are very noticeable {{not done}}
** HUD should be at infinity {{not done}}

==== [[Robinson R22]] ====
* Controls
** Tilting T-bar {{progressbar|90}}
*** limit range? {{not done}}
** Collective awkward to get at {{not done}}
** Throttle roll & governor switch? {{not done}}

==== [[Robinson R44]] ====
* Controls
** Tilting T-bar {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2023-07-10T15:36:24Z

Gastcan:

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 22nd August 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, SteamVR, Windows 10 -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 17 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
* If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make launcher accessible to VR user {{not done}}
** Make splash screen VR friendly {{done}}
** Make GUI accessible to VR user {{not done}}
** Make aircraft center accessible to VR user {{not done}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
** Show splash screen as quad layer for runtime to composite {{done}}
** Cube map/quads background provided by aircraft {{progressbar|20}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Launcher {{not done}}
** Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
** Transition to flightgear {{not done}}
** Transition to aircraft center with VR already started {{not done}}
** Transition back to flightgear from aircraft center {{not done}}
** More VR friendly environment
*** A simple background cubemap, shared with splash {{not done}}
*** A hanger (like xplane11)? {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
** Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
*** Grasp surfaces {{{progressbar|90}}
*** Force feedback for haptic gloves {{not done}}
** Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
*** Inverse kinematics of translate and rotate animations {{done}}
*** Ability to specify frictions {{not done}}
*** Ability to specify ranges (where can't be deduced) {{not done}}
*** Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

=== Aircraft compatibility ===

This section is largely about how usable controls are in VR with direct manipulation.

==== [[Cessna 172P]] ====
* Controls
** Flaps {{progressbar|90}}
*** kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
*** correct relative friction {{not done}}
** Opening of doors {{not done}}
*** reversing of gain expression {{not done}}

==== [[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
** External power on (GUI required) {{not done}}
** External power off (GUI required) {{not done}}
** Throttle roll (middle mouse drag required) {{not done}}
* Take off & flight
** Arms & hands get a bit in the way {{not done}}
* Controls
** Engine cutoff lever {{not done}}
*** Uses nasal, needs pick animation binding of some sort
** ELT {{progressbar|50}}
*** Both /ELT/test and /ELT/armed can be set to bool
*** Both can get set though
** Doors & hatches {{not done}}
*** Add limits

==== [[McDonnell Douglas F-15 Eagle]] ====
* Model
** Floating switches are very noticeable {{not done}}
** HUD should be at infinity {{not done}}

==== [[Robinson R22]] ====
* Controls
** Tilting T-bar {{progressbar|90}}
*** limit range? {{not done}}
** Collective awkward to get at {{not done}}
** Throttle roll & governor switch? {{not done}}

==== [[Robinson R44]] ====
* Controls
** Tilting T-bar {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2023-07-10T15:06:55Z

Gastcan: Update to my comments

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 22nd August 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, SteamVR, Windows 10 -- [[User:Gastcan|Gastcan]] 22:36, 17 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
* If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make launcher accessible to VR user {{not done}}
** Make splash screen VR friendly {{done}}
** Make GUI accessible to VR user {{not done}}
** Make aircraft center accessible to VR user {{not done}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
** Show splash screen as quad layer for runtime to composite {{done}}
** Cube map/quads background provided by aircraft {{progressbar|20}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Launcher {{not done}}
** Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
** Transition to flightgear {{not done}}
** Transition to aircraft center with VR already started {{not done}}
** Transition back to flightgear from aircraft center {{not done}}
** More VR friendly environment
*** A simple background cubemap, shared with splash {{not done}}
*** A hanger (like xplane11)? {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
** Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
*** Grasp surfaces {{{progressbar|90}}
*** Force feedback for haptic gloves {{not done}}
** Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
*** Inverse kinematics of translate and rotate animations {{done}}
*** Ability to specify frictions {{not done}}
*** Ability to specify ranges (where can't be deduced) {{not done}}
*** Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

=== Aircraft compatibility ===

This section is largely about how usable controls are in VR with direct manipulation.

==== [[Cessna 172P]] ====
* Controls
** Flaps {{progressbar|90}}
*** kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
*** correct relative friction {{not done}}
** Opening of doors {{not done}}
*** reversing of gain expression {{not done}}

==== [[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
** External power on (GUI required) {{not done}}
** External power off (GUI required) {{not done}}
** Throttle roll (middle mouse drag required) {{not done}}
* Take off & flight
** Arms & hands get a bit in the way {{not done}}
* Controls
** Engine cutoff lever {{not done}}
*** Uses nasal, needs pick animation binding of some sort
** ELT {{progressbar|50}}
*** Both /ELT/test and /ELT/armed can be set to bool
*** Both can get set though
** Doors & hatches {{not done}}
*** Add limits

==== [[McDonnell Douglas F-15 Eagle]] ====
* Model
** Floating switches are very noticeable {{not done}}
** HUD should be at infinity {{not done}}

==== [[Robinson R22]] ====
* Controls
** Tilting T-bar {{progressbar|90}}
*** limit range? {{not done}}
** Collective awkward to get at {{not done}}
** Throttle roll & governor switch? {{not done}}

==== [[Robinson R44]] ====
* Controls
** Tilting T-bar {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2023-07-10T03:37:14Z

Gastcan: Added working configuration

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 22nd August 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, SteamVR, Windows 10 -- [[User:Gastcan|Gastcan]] 22:36, 17 July 2023 (UTC)
** Surrounding scenery stutters a bit
** (Additional details to follow)

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
* If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make launcher accessible to VR user {{not done}}
** Make splash screen VR friendly {{done}}
** Make GUI accessible to VR user {{not done}}
** Make aircraft center accessible to VR user {{not done}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
** Show splash screen as quad layer for runtime to composite {{done}}
** Cube map/quads background provided by aircraft {{progressbar|20}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Launcher {{not done}}
** Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
** Transition to flightgear {{not done}}
** Transition to aircraft center with VR already started {{not done}}
** Transition back to flightgear from aircraft center {{not done}}
** More VR friendly environment
*** A simple background cubemap, shared with splash {{not done}}
*** A hanger (like xplane11)? {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
** Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
*** Grasp surfaces {{{progressbar|90}}
*** Force feedback for haptic gloves {{not done}}
** Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
*** Inverse kinematics of translate and rotate animations {{done}}
*** Ability to specify frictions {{not done}}
*** Ability to specify ranges (where can't be deduced) {{not done}}
*** Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

=== Aircraft compatibility ===

This section is largely about how usable controls are in VR with direct manipulation.

==== [[Cessna 172P]] ====
* Controls
** Flaps {{progressbar|90}}
*** kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
*** correct relative friction {{not done}}
** Opening of doors {{not done}}
*** reversing of gain expression {{not done}}

==== [[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
** External power on (GUI required) {{not done}}
** External power off (GUI required) {{not done}}
** Throttle roll (middle mouse drag required) {{not done}}
* Take off & flight
** Arms & hands get a bit in the way {{not done}}
* Controls
** Engine cutoff lever {{not done}}
*** Uses nasal, needs pick animation binding of some sort
** ELT {{progressbar|50}}
*** Both /ELT/test and /ELT/armed can be set to bool
*** Both can get set though
** Doors & hatches {{not done}}
*** Add limits

==== [[McDonnell Douglas F-15 Eagle]] ====
* Model
** Floating switches are very noticeable {{not done}}
** HUD should be at infinity {{not done}}

==== [[Robinson R22]] ====
* Controls
** Tilting T-bar {{progressbar|90}}
*** limit range? {{not done}}
** Collective awkward to get at {{not done}}
** Throttle roll & governor switch? {{not done}}

==== [[Robinson R44]] ====
* Controls
** Tilting T-bar {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]