FlightGear wiki - User contributions [en]

Virtual Reality

2026-03-28T19:03:56Z

Amalon: /* Enabling VR */ reword

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the [https://aur.archlinux.org/packages/osgxr osgxr] AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/flightgear-openscenegraph-git flightgear-openscenegraph-git], [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Usage & Controls ==

=== Headphones Setup ===

If you use headphones, or have speakers attached to your headset, enable "Head tracked sound (for headphones)" in the VR Options dialog (in the View menu). This will make surround sound relative to your head orientation. You can also enable binaural audio using HRTF filters in OpenAL to get better stereo surround sound.

=== Enabling VR ===

You can enable / disable VR at startup via the <code>--enable-vr</code> / <code>--disable-vr</code> command line options, or at runtime using the VR Options dialog (in the View menu).

=== VR Specific Controls ===

To use a mouse in VR, use the 360 mouse mode which captures the mouse and allows you to move it all around you. If you are using an OpenXR runtime which supports user presence detection then it will be enabled automatically while you have your headset on (if not you can toggle it manually using a keyboard shortcut).

You can use the following shortcuts on the next branch:

{| class="vatop"
|width="500" |
{| class="wikitable sortable"
|-
! Key combination
! Mouse combination
! VR input
! Action
|-
| {{key press|Ctrl|Tab}}
| {{key press|Shift|Right-click}}
| Put HMD on or take off (see "Enable 360 mouse when VR is in use" setting)
| Toggle VR/360 mouse mode
|-
| {{key press|Ctrl|Space}}
|
|
| Recenter VR view position and direction
|}
|}

== Status ==
'''Last updated: 29th January 2026'''

Stable in release/2024.1 branch, working in next. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in next branch:
* [[HDR_Pipeline]], including shadows etc.
* Head tracked sound
* 3D mouse cursors, 360 mouse mode

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities / work in progress:
* Frame pipelining improvements.
* Accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.
* Hand tracking.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* PSVR2 Monado Driver, Linux/Wayland (Debian Forky), AMDGPU -- [[User:DMJC|DMJC]] 03:12, 17 December 2025 (UTC)
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make GUI accessible to VR user {{progressbar|90}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{progressbar|80}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|80}}
** Ensure surround sound is relative to user's head {{done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|80}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}} {{paused}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|80}}
* Motion tracking {{done}}
*Settings {{progressbar|20}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
**Windows/OpenGL graphics bindings {{done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|80}} {{priority|2}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{progressbar|90}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Mouse support {{done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2026-03-28T19:01:27Z

Amalon: /* Usage & Controls */ subsections and audio

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the [https://aur.archlinux.org/packages/osgxr osgxr] AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/flightgear-openscenegraph-git flightgear-openscenegraph-git], [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Usage & Controls ==

=== Headphones Setup ===

If you use headphones, or have speakers attached to your headset, enable "Head tracked sound (for headphones)" in the VR Options dialog (in the View menu). This will make surround sound relative to your head orientation. You can also enable binaural audio using HRTF filters in OpenAL to get better stereo surround sound.

=== Enabling VR ===

You can enable / disable VR at startup via the <code>--enable-vr</code> / <code>--disable-vr</code> command line options, or using the GUI from the VR Options dialog (in the View menu).

=== VR Specific Controls ===

To use a mouse in VR, use the 360 mouse mode which captures the mouse and allows you to move it all around you. If you are using an OpenXR runtime which supports user presence detection then it will be enabled automatically while you have your headset on (if not you can toggle it manually using a keyboard shortcut).

You can use the following shortcuts on the next branch:

{| class="vatop"
|width="500" |
{| class="wikitable sortable"
|-
! Key combination
! Mouse combination
! VR input
! Action
|-
| {{key press|Ctrl|Tab}}
| {{key press|Shift|Right-click}}
| Put HMD on or take off (see "Enable 360 mouse when VR is in use" setting)
| Toggle VR/360 mouse mode
|-
| {{key press|Ctrl|Space}}
|
|
| Recenter VR view position and direction
|}
|}

== Status ==
'''Last updated: 29th January 2026'''

Stable in release/2024.1 branch, working in next. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in next branch:
* [[HDR_Pipeline]], including shadows etc.
* Head tracked sound
* 3D mouse cursors, 360 mouse mode

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities / work in progress:
* Frame pipelining improvements.
* Accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.
* Hand tracking.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* PSVR2 Monado Driver, Linux/Wayland (Debian Forky), AMDGPU -- [[User:DMJC|DMJC]] 03:12, 17 December 2025 (UTC)
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make GUI accessible to VR user {{progressbar|90}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{progressbar|80}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|80}}
** Ensure surround sound is relative to user's head {{done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|80}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}} {{paused}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|80}}
* Motion tracking {{done}}
*Settings {{progressbar|20}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
**Windows/OpenGL graphics bindings {{done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|80}} {{priority|2}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{progressbar|90}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Mouse support {{done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2026-03-28T18:50:09Z

Amalon: /* Usage & Controls */ reword

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the [https://aur.archlinux.org/packages/osgxr osgxr] AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/flightgear-openscenegraph-git flightgear-openscenegraph-git], [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Usage & Controls ==

You can enable / disable VR at startup via the <code>--enable-vr</code> / <code>--disable-vr</code> command line options, or using the GUI from the VR Options dialog (in the View menu).

You can use the following shortcuts on the next branch:

{| class="vatop"
|width="500" |
{| class="wikitable sortable"
|-
! Key combination
! Mouse combination
! VR input
! Action
|-
| {{key press|Ctrl|Tab}}
| {{key press|Shift|Right-click}}
| Put HMD on or take off (see "Enable 360 mouse when VR is in use" setting)
| Toggle VR/360 mouse mode
|-
| {{key press|Ctrl|Space}}
|
|
| Recenter VR view position and direction
|}
|}

== Status ==
'''Last updated: 29th January 2026'''

Stable in release/2024.1 branch, working in next. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in next branch:
* [[HDR_Pipeline]], including shadows etc.
* Head tracked sound
* 3D mouse cursors, 360 mouse mode

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities / work in progress:
* Frame pipelining improvements.
* Accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.
* Hand tracking.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* PSVR2 Monado Driver, Linux/Wayland (Debian Forky), AMDGPU -- [[User:DMJC|DMJC]] 03:12, 17 December 2025 (UTC)
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make GUI accessible to VR user {{progressbar|90}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{progressbar|80}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|80}}
** Ensure surround sound is relative to user's head {{done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|80}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}} {{paused}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|80}}
* Motion tracking {{done}}
*Settings {{progressbar|20}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
**Windows/OpenGL graphics bindings {{done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|80}} {{priority|2}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{progressbar|90}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Mouse support {{done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2026-03-28T18:49:26Z

Amalon: /* Usage & Controls */ describe how to start VR.

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the [https://aur.archlinux.org/packages/osgxr osgxr] AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/flightgear-openscenegraph-git flightgear-openscenegraph-git], [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Usage & Controls ==

You can enable / disable VR at startup via the <code>--enable-vr</code> / <code>--disable-vr</code> command line options, or using the GUI from the View->VR Options dialog.

You can use the following shortcuts on the next branch:

{| class="vatop"
|width="500" |
{| class="wikitable sortable"
|-
! Key combination
! Mouse combination
! VR input
! Action
|-
| {{key press|Ctrl|Tab}}
| {{key press|Shift|Right-click}}
| Put HMD on or take off (see "Enable 360 mouse when VR is in use" setting)
| Toggle VR/360 mouse mode
|-
| {{key press|Ctrl|Space}}
|
|
| Recenter VR view position and direction
|}
|}

== Status ==
'''Last updated: 29th January 2026'''

Stable in release/2024.1 branch, working in next. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in next branch:
* [[HDR_Pipeline]], including shadows etc.
* Head tracked sound
* 3D mouse cursors, 360 mouse mode

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities / work in progress:
* Frame pipelining improvements.
* Accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.
* Hand tracking.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* PSVR2 Monado Driver, Linux/Wayland (Debian Forky), AMDGPU -- [[User:DMJC|DMJC]] 03:12, 17 December 2025 (UTC)
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make GUI accessible to VR user {{progressbar|90}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{progressbar|80}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|80}}
** Ensure surround sound is relative to user's head {{done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|80}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}} {{paused}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|80}}
* Motion tracking {{done}}
*Settings {{progressbar|20}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
**Windows/OpenGL graphics bindings {{done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|80}} {{priority|2}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{progressbar|90}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Mouse support {{done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2026-03-28T14:01:16Z

Amalon: add some usage help

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the [https://aur.archlinux.org/packages/osgxr osgxr] AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/flightgear-openscenegraph-git flightgear-openscenegraph-git], [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Usage & Controls ==

You can use the following shortcuts on the next branch:

{| class="vatop"
|width="500" |
{| class="wikitable sortable"
|-
! Key combination
! Mouse combination
! VR input
! Action
|-
| {{key press|Ctrl|Tab}}
| {{key press|Shift|Right-click}}
| Put HMD on or take off (see "Enable 360 mouse when VR is in use" setting)
| Toggle VR/360 mouse mode
|-
| {{key press|Ctrl|Space}}
|
|
| Recenter VR view position and direction
|}
|}

== Status ==
'''Last updated: 29th January 2026'''

Stable in release/2024.1 branch, working in next. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in next branch:
* [[HDR_Pipeline]], including shadows etc.
* Head tracked sound
* 3D mouse cursors, 360 mouse mode

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities / work in progress:
* Frame pipelining improvements.
* Accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.
* Hand tracking.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* PSVR2 Monado Driver, Linux/Wayland (Debian Forky), AMDGPU -- [[User:DMJC|DMJC]] 03:12, 17 December 2025 (UTC)
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make GUI accessible to VR user {{progressbar|90}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{progressbar|80}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|80}}
** Ensure surround sound is relative to user's head {{done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|80}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}} {{paused}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|80}}
* Motion tracking {{done}}
*Settings {{progressbar|20}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
**Windows/OpenGL graphics bindings {{done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|80}} {{priority|2}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{progressbar|90}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Mouse support {{done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2026-03-28T11:24:37Z

Amalon: /* User facing features */ updates and drop done/old items

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the [https://aur.archlinux.org/packages/osgxr osgxr] AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/flightgear-openscenegraph-git flightgear-openscenegraph-git], [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 29th January 2026'''

Stable in release/2024.1 branch, working in next. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in next branch:
* [[HDR_Pipeline]], including shadows etc.
* Head tracked sound
* 3D mouse cursors, 360 mouse mode

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities / work in progress:
* Frame pipelining improvements.
* Accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.
* Hand tracking.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* PSVR2 Monado Driver, Linux/Wayland (Debian Forky), AMDGPU -- [[User:DMJC|DMJC]] 03:12, 17 December 2025 (UTC)
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make GUI accessible to VR user {{progressbar|90}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{progressbar|80}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|80}}
** Ensure surround sound is relative to user's head {{done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|80}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}} {{paused}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|80}}
* Motion tracking {{done}}
*Settings {{progressbar|20}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
**Windows/OpenGL graphics bindings {{done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|80}} {{priority|2}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{progressbar|90}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Mouse support {{done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2026-03-28T11:21:18Z

Amalon: /* General tasks */ drop old items (stability)

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the [https://aur.archlinux.org/packages/osgxr osgxr] AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/flightgear-openscenegraph-git flightgear-openscenegraph-git], [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 29th January 2026'''

Stable in release/2024.1 branch, working in next. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in next branch:
* [[HDR_Pipeline]], including shadows etc.
* Head tracked sound
* 3D mouse cursors, 360 mouse mode

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities / work in progress:
* Frame pipelining improvements.
* Accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.
* Hand tracking.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* PSVR2 Monado Driver, Linux/Wayland (Debian Forky), AMDGPU -- [[User:DMJC|DMJC]] 03:12, 17 December 2025 (UTC)
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make GUI accessible to VR user {{progressbar|90}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{progressbar|80}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|80}}
** Ensure surround sound is relative to user's head {{done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|80}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{done}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}} {{paused}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|80}}
* Motion tracking {{done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{progressbar|90}}
***Sound reference frame (user's head or room)? {{progressbar|90}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
**Windows/OpenGL graphics bindings {{done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Mouse support {{progressbar|90}}
**3D mouse cursors {{done}}
***mouse leave events on Windows {{progressbar|50}}
**360 mouse mode {{done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2026-03-28T11:20:39Z

Amalon: /* Development targets */ drop old items, and set gui to 90% done

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the [https://aur.archlinux.org/packages/osgxr osgxr] AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/flightgear-openscenegraph-git flightgear-openscenegraph-git], [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 29th January 2026'''

Stable in release/2024.1 branch, working in next. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in next branch:
* [[HDR_Pipeline]], including shadows etc.
* Head tracked sound
* 3D mouse cursors, 360 mouse mode

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities / work in progress:
* Frame pipelining improvements.
* Accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.
* Hand tracking.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* PSVR2 Monado Driver, Linux/Wayland (Debian Forky), AMDGPU -- [[User:DMJC|DMJC]] 03:12, 17 December 2025 (UTC)
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make GUI accessible to VR user {{progressbar|90}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
** Handle OpenXR stopping {{done}}
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{progressbar|80}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|80}}
** Ensure surround sound is relative to user's head {{done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|80}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{done}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}} {{paused}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|80}}
* Motion tracking {{done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{progressbar|90}}
***Sound reference frame (user's head or room)? {{progressbar|90}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
**Windows/OpenGL graphics bindings {{done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Mouse support {{progressbar|90}}
**3D mouse cursors {{done}}
***mouse leave events on Windows {{progressbar|50}}
**360 mouse mode {{done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2026-02-07T16:42:29Z

Amalon: /* General tasks */ head tracked sound is done

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the [https://aur.archlinux.org/packages/osgxr osgxr] AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/flightgear-openscenegraph-git flightgear-openscenegraph-git], [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 29th January 2026'''

Stable in release/2024.1 branch, working in next. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in next branch:
* [[HDR_Pipeline]], including shadows etc.
* Head tracked sound
* 3D mouse cursors, 360 mouse mode

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities / work in progress:
* Frame pipelining improvements.
* Accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.
* Hand tracking.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* PSVR2 Monado Driver, Linux/Wayland (Debian Forky), AMDGPU -- [[User:DMJC|DMJC]] 03:12, 17 December 2025 (UTC)
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
** Handle OpenXR stopping {{done}}
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{progressbar|80}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|80}}
** Ensure surround sound is relative to user's head {{done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|80}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{done}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}} {{paused}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|80}}
* Motion tracking {{done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{progressbar|90}}
***Sound reference frame (user's head or room)? {{progressbar|90}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
**Windows/OpenGL graphics bindings {{done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Mouse support {{progressbar|90}}
**3D mouse cursors {{done}}
***mouse leave events on Windows {{progressbar|50}}
**360 mouse mode {{done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2026-01-29T12:18:46Z

Amalon: /* User facing features */ mouse update

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the [https://aur.archlinux.org/packages/osgxr osgxr] AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/flightgear-openscenegraph-git flightgear-openscenegraph-git], [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 29th January 2026'''

Stable in release/2024.1 branch, working in next. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in next branch:
* [[HDR_Pipeline]], including shadows etc.
* Head tracked sound
* 3D mouse cursors, 360 mouse mode

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities / work in progress:
* Frame pipelining improvements.
* Accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.
* Hand tracking.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* PSVR2 Monado Driver, Linux/Wayland (Debian Forky), AMDGPU -- [[User:DMJC|DMJC]] 03:12, 17 December 2025 (UTC)
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
** Handle OpenXR stopping {{done}}
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{progressbar|80}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|80}}
** Ensure surround sound is relative to user's head {{progressbar|90}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|80}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{done}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}} {{paused}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|80}}
* Motion tracking {{done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{progressbar|90}}
***Sound reference frame (user's head or room)? {{progressbar|90}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
**Windows/OpenGL graphics bindings {{done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Mouse support {{progressbar|90}}
**3D mouse cursors {{done}}
***mouse leave events on Windows {{progressbar|50}}
**360 mouse mode {{done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2026-01-29T12:17:14Z

Amalon: /* Status */ update, vr mouse merged

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the [https://aur.archlinux.org/packages/osgxr osgxr] AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/flightgear-openscenegraph-git flightgear-openscenegraph-git], [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 29th January 2026'''

Stable in release/2024.1 branch, working in next. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in next branch:
* [[HDR_Pipeline]], including shadows etc.
* Head tracked sound
* 3D mouse cursors, 360 mouse mode

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities / work in progress:
* Frame pipelining improvements.
* Accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.
* Hand tracking.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* PSVR2 Monado Driver, Linux/Wayland (Debian Forky), AMDGPU -- [[User:DMJC|DMJC]] 03:12, 17 December 2025 (UTC)
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
** Handle OpenXR stopping {{done}}
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{progressbar|80}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|80}}
** Ensure surround sound is relative to user's head {{progressbar|90}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|80}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{done}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}} {{paused}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|80}}
* Motion tracking {{done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{progressbar|90}}
***Sound reference frame (user's head or room)? {{progressbar|90}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
**Windows/OpenGL graphics bindings {{done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Mouse support {{progressbar|90}}
**3D mouse cursors {{progressbar|90}}
***mouse leave events on Windows {{not_done}}
**360 mouse mode {{progressbar|90}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2026-01-07T08:41:42Z

Amalon: /* User facing features */ deduplicate

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the [https://aur.archlinux.org/packages/osgxr osgxr] AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/flightgear-openscenegraph-git flightgear-openscenegraph-git], [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 24th December 2025'''

Stable in release/2024.1 branch, working in next. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in next branch:
* [[HDR_Pipeline]], including shadows etc.

Pending work for next branch:
* Head tracked sound, 3D mouse cursors, 360 mouse mode ([https://gitlab.com/flightgear/openscenegraph/-/merge_requests/14 OSG], [https://gitlab.com/flightgear/simgear/-/merge_requests/201 SG], [https://gitlab.com/flightgear/flightgear/-/merge_requests/368 FG], [https://gitlab.com/flightgear/fgdata/-/merge_requests/207 FGData])

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities / work in progress:
* Frame pipelining improvements.
* Accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.
* Hand tracking.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* PSVR2 Monado Driver, Linux/Wayland (Debian Forky), AMDGPU -- [[User:DMJC|DMJC]] 03:12, 17 December 2025 (UTC)
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
** Handle OpenXR stopping {{done}}
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{progressbar|80}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|80}}
** Ensure surround sound is relative to user's head {{progressbar|90}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|80}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{done}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}} {{paused}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|80}}
* Motion tracking {{done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{progressbar|90}}
***Sound reference frame (user's head or room)? {{progressbar|90}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
**Windows/OpenGL graphics bindings {{done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Mouse support {{progressbar|90}}
**3D mouse cursors {{progressbar|90}}
***mouse leave events on Windows {{not_done}}
**360 mouse mode {{progressbar|90}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2026-01-07T08:37:28Z

Amalon: /* Source code */ AUR osg link

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the [https://aur.archlinux.org/packages/osgxr osgxr] AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/flightgear-openscenegraph-git flightgear-openscenegraph-git], [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 24th December 2025'''

Stable in release/2024.1 branch, working in next. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in next branch:
* [[HDR_Pipeline]], including shadows etc.

Pending work for next branch:
* Head tracked sound, 3D mouse cursors, 360 mouse mode ([https://gitlab.com/flightgear/openscenegraph/-/merge_requests/14 OSG], [https://gitlab.com/flightgear/simgear/-/merge_requests/201 SG], [https://gitlab.com/flightgear/flightgear/-/merge_requests/368 FG], [https://gitlab.com/flightgear/fgdata/-/merge_requests/207 FGData])

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities / work in progress:
* Frame pipelining improvements.
* Accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.
* Hand tracking.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* PSVR2 Monado Driver, Linux/Wayland (Debian Forky), AMDGPU -- [[User:DMJC|DMJC]] 03:12, 17 December 2025 (UTC)
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
** Handle OpenXR stopping {{done}}
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{progressbar|80}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|80}}
** Ensure surround sound is relative to user's head {{progressbar|90}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|80}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{done}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}} {{paused}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|80}}
* Motion tracking {{done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{progressbar|90}}
***Sound reference frame (user's head or room)? {{progressbar|90}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
**Windows/OpenGL graphics bindings {{done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Mouse support {{progressbar|90}}
**3D mouse cursors {{progressbar|90}}
***mouse leave events on Windows {{not_done}}
**360 mouse mode {{progressbar|90}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Hand tracking {{progressbar|50}}
** Show hand (basic) {{progressbar|90}}
** Show hand (skinned) {{not_done}}
** Direct and intuitive interaction {{progressbar|50}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2026-01-07T08:33:02Z

Amalon: /* User facing features */ more updates

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the [https://aur.archlinux.org/packages/osgxr osgxr] AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 24th December 2025'''

Stable in release/2024.1 branch, working in next. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in next branch:
* [[HDR_Pipeline]], including shadows etc.

Pending work for next branch:
* Head tracked sound, 3D mouse cursors, 360 mouse mode ([https://gitlab.com/flightgear/openscenegraph/-/merge_requests/14 OSG], [https://gitlab.com/flightgear/simgear/-/merge_requests/201 SG], [https://gitlab.com/flightgear/flightgear/-/merge_requests/368 FG], [https://gitlab.com/flightgear/fgdata/-/merge_requests/207 FGData])

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities / work in progress:
* Frame pipelining improvements.
* Accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.
* Hand tracking.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* PSVR2 Monado Driver, Linux/Wayland (Debian Forky), AMDGPU -- [[User:DMJC|DMJC]] 03:12, 17 December 2025 (UTC)
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
** Handle OpenXR stopping {{done}}
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{progressbar|80}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|80}}
** Ensure surround sound is relative to user's head {{progressbar|90}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|80}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{done}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}} {{paused}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|80}}
* Motion tracking {{done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{progressbar|90}}
***Sound reference frame (user's head or room)? {{progressbar|90}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
**Windows/OpenGL graphics bindings {{done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Mouse support {{progressbar|90}}
**3D mouse cursors {{progressbar|90}}
***mouse leave events on Windows {{not_done}}
**360 mouse mode {{progressbar|90}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Hand tracking {{progressbar|50}}
** Show hand (basic) {{progressbar|90}}
** Show hand (skinned) {{not_done}}
** Direct and intuitive interaction {{progressbar|50}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2026-01-07T08:31:09Z

Amalon: /* User facing features */ misc updates

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the [https://aur.archlinux.org/packages/osgxr osgxr] AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 24th December 2025'''

Stable in release/2024.1 branch, working in next. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in next branch:
* [[HDR_Pipeline]], including shadows etc.

Pending work for next branch:
* Head tracked sound, 3D mouse cursors, 360 mouse mode ([https://gitlab.com/flightgear/openscenegraph/-/merge_requests/14 OSG], [https://gitlab.com/flightgear/simgear/-/merge_requests/201 SG], [https://gitlab.com/flightgear/flightgear/-/merge_requests/368 FG], [https://gitlab.com/flightgear/fgdata/-/merge_requests/207 FGData])

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities / work in progress:
* Frame pipelining improvements.
* Accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.
* Hand tracking.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* PSVR2 Monado Driver, Linux/Wayland (Debian Forky), AMDGPU -- [[User:DMJC|DMJC]] 03:12, 17 December 2025 (UTC)
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
** Handle OpenXR stopping {{done}}
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{progressbar|80}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|80}}
** Ensure surround sound is relative to user's head {{progressbar|90}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|80}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{done}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}} {{paused}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|80}}
* Motion tracking {{done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{progressbar|90}}
***Sound reference frame (user's head or room)? {{progressbar|90}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
**Windows/OpenGL graphics bindings {{done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Mouse support {{progressbar|90}}
**3D mouse cursors {{progressbar|90}}
***mouse leave events on Windows {{not_done}}
**360 mouse mode {{progressbar|90}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2026-01-07T08:20:59Z

Amalon: /* General tasks */ general progress, some steamvr on linux issues resolved

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the [https://aur.archlinux.org/packages/osgxr osgxr] AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 24th December 2025'''

Stable in release/2024.1 branch, working in next. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in next branch:
* [[HDR_Pipeline]], including shadows etc.

Pending work for next branch:
* Head tracked sound, 3D mouse cursors, 360 mouse mode ([https://gitlab.com/flightgear/openscenegraph/-/merge_requests/14 OSG], [https://gitlab.com/flightgear/simgear/-/merge_requests/201 SG], [https://gitlab.com/flightgear/flightgear/-/merge_requests/368 FG], [https://gitlab.com/flightgear/fgdata/-/merge_requests/207 FGData])

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities / work in progress:
* Frame pipelining improvements.
* Accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.
* Hand tracking.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* PSVR2 Monado Driver, Linux/Wayland (Debian Forky), AMDGPU -- [[User:DMJC|DMJC]] 03:12, 17 December 2025 (UTC)
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
** Handle OpenXR stopping {{done}}
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{progressbar|80}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|80}}
** Ensure surround sound is relative to user's head {{progressbar|90}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|60}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{progressbar|50}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|20}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
**Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{not done}}
***Sound reference frame (user's head or room)? {{not done}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
**Windows/OpenGL graphics bindings {{done}}
**Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2025-12-24T09:31:26Z

Amalon: /* Status */

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the [https://aur.archlinux.org/packages/osgxr osgxr] AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 24th December 2025'''

Stable in release/2024.1 branch, working in next. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in next branch:
* [[HDR_Pipeline]], including shadows etc.

Pending work for next branch:
* Head tracked sound, 3D mouse cursors, 360 mouse mode ([https://gitlab.com/flightgear/openscenegraph/-/merge_requests/14 OSG], [https://gitlab.com/flightgear/simgear/-/merge_requests/201 SG], [https://gitlab.com/flightgear/flightgear/-/merge_requests/368 FG], [https://gitlab.com/flightgear/fgdata/-/merge_requests/207 FGData])

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities / work in progress:
* Frame pipelining improvements.
* Accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.
* Hand tracking.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* PSVR2 Monado Driver, Linux/Wayland (Debian Forky), AMDGPU -- [[User:DMJC|DMJC]] 03:12, 17 December 2025 (UTC)
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
***SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
***SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{not done}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|60}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{progressbar|50}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|20}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
**Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{not done}}
***Sound reference frame (user's head or room)? {{not done}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
**Windows/OpenGL graphics bindings {{done}}
**Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2025-12-24T09:30:45Z

Amalon: /* Status */ Update status a bit

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the [https://aur.archlinux.org/packages/osgxr osgxr] AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 24th December 2025'''

Stable in release/2024.1 branch, working in next. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in next branch:
* [[HDR_Pipeline]].

Pending work for next branch:
* Head tracked sound, 3D mouse cursors, 360 mouse mode ([https://gitlab.com/flightgear/openscenegraph/-/merge_requests/14 OSG], [https://gitlab.com/flightgear/simgear/-/merge_requests/201 SG], [https://gitlab.com/flightgear/flightgear/-/merge_requests/368 FG], [https://gitlab.com/flightgear/fgdata/-/merge_requests/207 FGData])

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities / work in progress:
* Frame pipelining improvements.
* Accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.
* Hand tracking.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* PSVR2 Monado Driver, Linux/Wayland (Debian Forky), AMDGPU -- [[User:DMJC|DMJC]] 03:12, 17 December 2025 (UTC)
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
***SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
***SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{not done}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|60}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{progressbar|50}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|20}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
**Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{not done}}
***Sound reference frame (user's head or room)? {{not done}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
**Windows/OpenGL graphics bindings {{done}}
**Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2025-05-21T09:50:20Z

Amalon: /* Source code */ link to osgxr AUR

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the [https://aur.archlinux.org/packages/osgxr osgxr] AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 13th January 2025'''

Stable in release/2024.1 branch, broken in next pending HDR changes, partly out of tree. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Broken in next branch:
* Pending HDR fixes.

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Support [[HDR Pipeline]] and accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* [[HDR Pipeline]] is unsupported, but work to support it is in progress
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
*SteamVR, Linux, AMD graphics:
**Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
**try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
*Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
***There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
*SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
**Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
**May need --disable-sentry
**Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
***SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
***SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{not done}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|60}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{progressbar|50}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|20}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
**Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{not done}}
***Sound reference frame (user's head or room)? {{not done}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
**Windows/OpenGL graphics bindings {{done}}
**Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2025-05-21T09:49:51Z

Amalon: /* Source code */ strike out osgxr branches, they're long stale

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* <s>More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.</s> (stale)
* <s>More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)
* <s>More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).</s> (stale)

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 13th January 2025'''

Stable in release/2024.1 branch, broken in next pending HDR changes, partly out of tree. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Broken in next branch:
* Pending HDR fixes.

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Support [[HDR Pipeline]] and accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* [[HDR Pipeline]] is unsupported, but work to support it is in progress
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
*SteamVR, Linux, AMD graphics:
**Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
**try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
*Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
***There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
*SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
**Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
**May need --disable-sentry
**Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
***SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
***SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{not done}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|60}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{progressbar|50}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|20}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
**Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{not done}}
***Sound reference frame (user's head or room)? {{not done}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
**Windows/OpenGL graphics bindings {{done}}
**Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2025-05-21T09:48:42Z

Amalon: /* Source code */ link to nightly builds and AUR packages

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. [https://aur.archlinux.org/packages/simgear-git simgear-git], [https://aur.archlinux.org/packages/flightgear-git flightgear-git], [https://aur.archlinux.org/packages/flightgear-data-git flightgear-data-git].
** On Windows & Linux, you can use the [https://www.flightgear.org/download/nightly/ nightly builds].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 13th January 2025'''

Stable in release/2024.1 branch, broken in next pending HDR changes, partly out of tree. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Broken in next branch:
* Pending HDR fixes.

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Support [[HDR Pipeline]] and accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* [[HDR Pipeline]] is unsupported, but work to support it is in progress
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
*SteamVR, Linux, AMD graphics:
**Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
**try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
*Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
***There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
*SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
**Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
**May need --disable-sentry
**Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
***SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
***SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{not done}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|60}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{progressbar|50}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|20}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
**Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{not done}}
***Sound reference frame (user's head or room)? {{not done}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
**Windows/OpenGL graphics bindings {{done}}
**Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2025-04-07T12:03:53Z

Amalon: Drop SF changelog link, development is now on gitlab

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 13th January 2025'''

Stable in release/2024.1 branch, broken in next pending HDR changes, partly out of tree. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Broken in next branch:
* Pending HDR fixes.

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Support [[HDR Pipeline]] and accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* [[HDR Pipeline]] is unsupported, but work to support it is in progress
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
*SteamVR, Linux, AMD graphics:
**Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
**try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
*Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
***There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
*SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
**Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
**May need --disable-sentry
**Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
***SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
***SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{not done}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|60}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{progressbar|50}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|20}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
**Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{not done}}
***Sound reference frame (user's head or room)? {{not done}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
**Windows/OpenGL graphics bindings {{done}}
**Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2025-01-13T16:20:54Z

Amalon: /* Status */ typo

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 13th January 2025'''

Stable in release/2024.1 branch, broken in next pending HDR changes, partly out of tree. Development in progress.

VR support is working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Broken in next branch:
* Pending HDR fixes.

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Support [[HDR Pipeline]] and accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* [[HDR Pipeline]] is unsupported, but work to support it is in progress
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
*SteamVR, Linux, AMD graphics:
**Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
**try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
*Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
***There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
*SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
**Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
**May need --disable-sentry
**Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
***SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
***SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{not done}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|60}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{progressbar|50}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|20}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
**Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{not done}}
***Sound reference frame (user's head or room)? {{not done}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
**Windows/OpenGL graphics bindings {{done}}
**Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2025-01-13T16:10:22Z

Amalon: /* Status */ Update status a bit

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 13th January 2025'''

Stable in release/2024.1 branch, broken in next pending HDR changes, partly out of tree. Development in progress.

VR support is Working in release/2024.1 branch, and enabled in Linux and Windows GitLab CI builds:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Broken in next branch:
* Pending HDR fixes.

Working in osgxr/osgxr_clean branch (stale, needs rebasing):
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Support [[HDR Pipeline]] and accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken on release/2024.1 branch:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
** Will be fixed with HDR changes.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
** Will be fixed with HDR changes.
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* [[HDR Pipeline]] is unsupported, but work to support it is in progress
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
*SteamVR, Linux, AMD graphics:
**Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
**try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
*Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
***There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
*SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
**Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
**May need --disable-sentry
**Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
***SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
***SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{not done}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|60}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{progressbar|50}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|20}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
**Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{not done}}
***Sound reference frame (user's head or room)? {{not done}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
**Windows/OpenGL graphics bindings {{done}}
**Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2025-01-13T15:58:31Z

Amalon: Though VR support is still quite basic, its not really experimental any longer, and will be in the next release, so drop experimental header

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 31st March 2024'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Support [[HDR Pipeline]] and accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* [[HDR Pipeline]] is unsupported, but work to support it is in progress
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:nia|Nia]] ([[User talk:nia|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
*SteamVR, Linux, AMD graphics:
**Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
**try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
*Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
***There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
*SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
**Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
**May need --disable-sentry
**Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
***SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
***SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{not done}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|60}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{progressbar|50}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|20}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
**Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{not done}}
***Sound reference frame (user's head or room)? {{not done}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
**Windows/OpenGL graphics bindings {{done}}
**Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2024-03-31T11:06:27Z

Amalon: /* User facing features */ General HDR Pipeline support in progress

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 31st March 2024'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Support [[HDR Pipeline]] and accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* [[HDR Pipeline]] is unsupported, but work to support it is in progress
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
*SteamVR, Linux, AMD graphics:
**Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
**try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
*Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
***There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
*SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
**Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
**May need --disable-sentry
**Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
***SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
***SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{not done}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|60}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**General [[HDR Pipeline]] Support {{progressbar|50}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|20}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
**Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{not done}}
***Sound reference frame (user's head or room)? {{not done}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
**Windows/OpenGL graphics bindings {{done}}
**Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2024-03-31T11:05:31Z

Amalon: /* User facing features */ geometry shader multiviewport and OVR_multivew2 are now in progress

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 31st March 2024'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Support [[HDR Pipeline]] and accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* [[HDR Pipeline]] is unsupported, but work to support it is in progress
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
*SteamVR, Linux, AMD graphics:
**Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
**try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
*Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
***There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
*SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
**Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
**May need --disable-sentry
**Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
***SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
***SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{not done}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|60}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**Geometry shader multi viewport ([[HDR Pipeline]]) {{progressbar|50}}
**OVR_multiview2 with OSG branch ([[HDR Pipeline]]) {{progressbar|20}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
**Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{not done}}
***Sound reference frame (user's head or room)? {{not done}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
**Windows/OpenGL graphics bindings {{done}}
**Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2024-03-31T11:00:44Z

Amalon: /* Status */ update date

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 31st March 2024'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Support [[HDR Pipeline]] and accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* [[HDR Pipeline]] is unsupported, but work to support it is in progress
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
*SteamVR, Linux, AMD graphics:
**Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
**try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
*Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
***There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
*SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
**Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
**May need --disable-sentry
**Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
***SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
***SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{not done}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|60}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**Geometry shader multi viewport {{not done}}
**OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
**Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{not done}}
***Sound reference frame (user's head or room)? {{not done}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
**Windows/OpenGL graphics bindings {{done}}
**Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2024-03-31T11:00:22Z

Amalon: /* Known issues */ HDR support

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 22nd August 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Support [[HDR Pipeline]] and accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* [[HDR Pipeline]] is unsupported, but work to support it is in progress
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
*SteamVR, Linux, AMD graphics:
**Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
**try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
*Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
***There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
*SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
**Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
**May need --disable-sentry
**Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
***SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
***SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{not done}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|60}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**Geometry shader multi viewport {{not done}}
**OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
**Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{not done}}
***Sound reference frame (user's head or room)? {{not done}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
**Windows/OpenGL graphics bindings {{done}}
**Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2024-03-31T10:59:08Z

Amalon: /* Status */ currently working on HDR / multiview rendering, monado pretty stable

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 22nd August 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Support [[HDR Pipeline]] and accelerated multiview rendering.
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* Oculus Quest 2, Oculus App, Windows 10 --[[User:Necolatis|Necolatis]] ([[User talk:Necolatis|talk]]) 13:22, 7 June 2023 (UTC)
** Works fine, shadows gets rendered, but move when turning the head.
** Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
* Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --[[User:Skyclmbr]] 18:17, 11 October 2023 (UTC)
** Head tracking works fine, but right eye screen flickers constantly throughout flight
** Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 02:15, 09 July 2023 (UTC)
** (Using the Windows nightly build 08-Jul-23)
** The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit

* HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- [[User:Gastcan|Gastcan]] ([[User talk:Gastcan|talk]]) 16:15, 14 August 2023 (UTC)
** Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
** The view follows the direction of the headset with minor stuttering.
**<s>The viewpoint/perspective drifts clockwise, to the right.</s> <- Drifting is virtually eliminated by adding '[https://gitlab.freedesktop.org/mateosss/basalt Basalt for Monado]' (provides 6DoF).

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
*SteamVR, Linux, AMD graphics:
**Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
**try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
*Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
***There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
*SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
**Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
**May need --disable-sentry
**Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

==Performance==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
**On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
*If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
**On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
*If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
*Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
*If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

==Implementation Details==

===Concepts===

====Interaction Modes====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
*When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
*When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
*When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
*When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

==Roadmap==

===Development targets===
*Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
**Bare functionality {{done}}
**Stability {{done}}
*Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
*Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
**Make launcher accessible to VR user {{not done}}
**Make splash screen VR friendly {{done}}
**Make GUI accessible to VR user {{not done}}
**Make aircraft center accessible to VR user {{not done}}
*Functionality {{progressbar|40}}
**Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
**Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

===General tasks ===

*Stability {{done}}
**Clean shutdown {{done}}
***SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
***SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
*Resolve questions around VR mirror switching {{done}}
*Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
**SceneView mode: Combine culling traversal {{not done}}
**Check why update isn't proceeding in parallel with multithreading {{not done}}
**Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
*Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
**Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
**Specify separate sound device for VR so it switches automatically? {{not done}}

===User facing features ===

*Display {{progressbar|60}}
**Show views on HMD {{done}}
**Configurable mirror on desktop window {{done}}
***Left, Right, Left & Right {{done}}
***Composited mirror mode (combine & blend views and allow zooming) {{paused}}
**Geometry shader multi viewport {{not done}}
**OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
**Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
*Settings {{progressbar|20}}
** Mirror settings {{done}}
**Dynamic toggling of VR {{done}}
**Sound settings? {{not done}}
***Sound reference frame (user's head or room)? {{not done}}
**Advanced VR settings {{not done}}
***validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
***osgXR VR mode {{not done}}
****Fix slave cameras mode clearing {{not done}}
***osgXR swapchain mode {{not done}}
*Platform support {{progressbar|70}}
**Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
**Windows/OpenGL graphics bindings {{done}}
**Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
**Show splash screen as quad layer for runtime to composite {{done}}
**Cube map/quads background provided by aircraft {{progressbar|20}}
*GUI {{progressbar|10}} {{priority|2}}
**Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
**Interact with GUI buttons etc using controllers as lasers {{not done}}
*Launcher {{not done}}
**Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
**Transition to flightgear {{not done}}
**Transition to aircraft center with VR already started {{not done}}
**Transition back to flightgear from aircraft center {{not done}}
**More VR friendly environment
***A simple background cubemap, shared with splash {{not done}}
***A hanger (like xplane11)? {{not done}}
*Controller support {{progressbar|50}}
**Basic action support in osgXR API {{done}}
**Tie osgXR actions into FG's input system {{done}}
**Helicopter flight controls (can use for planes too) {{done}}
**Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
**Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
**Basic interaction with controls {{progressbar|80}}
***Grasp surfaces {{{progressbar|90}}
***Force feedback for haptic gloves {{not done}}
**Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
***Inverse kinematics of translate and rotate animations {{done}}
***Ability to specify frictions {{not done}}
***Ability to specify ranges (where can't be deduced) {{not done}}
***Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
**Parabolic arc teleport {{not done}}
**Snap into seats {{not done}}
**Easy turning around {{not done}}

===Aircraft compatibility===

This section is largely about how usable controls are in VR with direct manipulation.

====[[Cessna 172P]] ====
*Controls
**Flaps {{progressbar|90}}
***kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
***correct relative friction {{not done}}
**Opening of doors {{not done}}
***reversing of gain expression {{not done}}

====[[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
**External power on (GUI required) {{not done}}
**External power off (GUI required) {{not done}}
**Throttle roll (middle mouse drag required) {{not done}}
*Take off & flight
** Arms & hands get a bit in the way {{not done}}
*Controls
** Engine cutoff lever {{not done}}
***Uses nasal, needs pick animation binding of some sort
**ELT {{progressbar|50}}
***Both /ELT/test and /ELT/armed can be set to bool
***Both can get set though
**Doors & hatches {{not done}}
***Add limits

====[[McDonnell Douglas F-15 Eagle]] ====
*Model
**Floating switches are very noticeable {{not done}}
**HUD should be at infinity {{not done}}

====[[Robinson R22]] ====
*Controls
**Tilting T-bar {{progressbar|90}}
***limit range? {{not done}}
**Collective awkward to get at {{not done}}
**Throttle roll & governor switch? {{not done}}

====[[Robinson R44]] ====
*Controls
**Tilting T-bar {{not done}}

==References==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2023-04-30T14:28:48Z

Amalon: /* Aircraft compatibility */ note

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 22nd August 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
* If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make launcher accessible to VR user {{not done}}
** Make splash screen VR friendly {{done}}
** Make GUI accessible to VR user {{not done}}
** Make aircraft center accessible to VR user {{not done}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
** Show splash screen as quad layer for runtime to composite {{done}}
** Cube map/quads background provided by aircraft {{progressbar|20}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Launcher {{not done}}
** Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
** Transition to flightgear {{not done}}
** Transition to aircraft center with VR already started {{not done}}
** Transition back to flightgear from aircraft center {{not done}}
** More VR friendly environment
*** A simple background cubemap, shared with splash {{not done}}
*** A hanger (like xplane11)? {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
** Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
*** Grasp surfaces {{{progressbar|90}}
*** Force feedback for haptic gloves {{not done}}
** Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
*** Inverse kinematics of translate and rotate animations {{done}}
*** Ability to specify frictions {{not done}}
*** Ability to specify ranges (where can't be deduced) {{not done}}
*** Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

=== Aircraft compatibility ===

This section is largely about how usable controls are in VR with direct manipulation.

==== [[Cessna 172P]] ====
* Controls
** Flaps {{progressbar|90}}
*** kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
*** correct relative friction {{not done}}
** Opening of doors {{not done}}
*** reversing of gain expression {{not done}}

==== [[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
** External power on (GUI required) {{not done}}
** External power off (GUI required) {{not done}}
** Throttle roll (middle mouse drag required) {{not done}}
* Take off & flight
** Arms & hands get a bit in the way {{not done}}
* Controls
** Engine cutoff lever {{not done}}
*** Uses nasal, needs pick animation binding of some sort
** ELT {{progressbar|50}}
*** Both /ELT/test and /ELT/armed can be set to bool
*** Both can get set though
** Doors & hatches {{not done}}
*** Add limits

==== [[McDonnell Douglas F-15 Eagle]] ====
* Model
** Floating switches are very noticeable {{not done}}
** HUD should be at infinity {{not done}}

==== [[Robinson R22]] ====
* Controls
** Tilting T-bar {{progressbar|90}}
*** limit range? {{not done}}
** Collective awkward to get at {{not done}}
** Throttle roll & governor switch? {{not done}}

==== [[Robinson R44]] ====
* Controls
** Tilting T-bar {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2023-04-30T14:27:12Z

Amalon: /* Roadmap */ Add aircraft compatibility section

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 22nd August 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
* If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make launcher accessible to VR user {{not done}}
** Make splash screen VR friendly {{done}}
** Make GUI accessible to VR user {{not done}}
** Make aircraft center accessible to VR user {{not done}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
** Show splash screen as quad layer for runtime to composite {{done}}
** Cube map/quads background provided by aircraft {{progressbar|20}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Launcher {{not done}}
** Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
** Transition to flightgear {{not done}}
** Transition to aircraft center with VR already started {{not done}}
** Transition back to flightgear from aircraft center {{not done}}
** More VR friendly environment
*** A simple background cubemap, shared with splash {{not done}}
*** A hanger (like xplane11)? {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
** Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
*** Grasp surfaces {{{progressbar|90}}
*** Force feedback for haptic gloves {{not done}}
** Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
*** Inverse kinematics of translate and rotate animations {{done}}
*** Ability to specify frictions {{not done}}
*** Ability to specify ranges (where can't be deduced) {{not done}}
*** Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

=== Aircraft compatibility ===

==== [[Cessna 172P]] ====
* Controls
** Flaps {{progressbar|90}}
*** kinematic needs to be reversible {{progressbar|90}}
** Visors {{progressbar|70}}
*** correct relative friction {{not done}}
** Opening of doors {{not done}}
*** reversing of gain expression {{not done}}

==== [[Eurocopter EC130 B4]] ====
* Engine start
** Engine cutoff lever reversing (Alt+C required) {{not done}}
** External power on (GUI required) {{not done}}
** External power off (GUI required) {{not done}}
** Throttle roll (middle mouse drag required) {{not done}}
* Take off & flight
** Arms & hands get a bit in the way {{not done}}
* Controls
** Engine cutoff lever {{not done}}
*** Uses nasal, needs pick animation binding of some sort
** ELT {{progressbar|50}}
*** Both /ELT/test and /ELT/armed can be set to bool
*** Both can get set though
** Doors & hatches {{not done}}
*** Add limits

==== [[McDonnell Douglas F-15 Eagle]] ====
* Model
** Floating switches are very noticeable {{not done}}
** HUD should be at infinity {{not done}}

==== [[Robinson R22]] ====
* Controls
** Tilting T-bar {{progressbar|90}}
*** limit range? {{not done}}
** Collective awkward to get at {{not done}}
** Throttle roll & governor switch? {{not done}}

==== [[Robinson R44]] ====
* Controls
** Tilting T-bar {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2023-04-30T14:11:45Z

Amalon: /* User facing features */ expand on hand tracking work

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 22nd August 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
* If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make launcher accessible to VR user {{not done}}
** Make splash screen VR friendly {{done}}
** Make GUI accessible to VR user {{not done}}
** Make aircraft center accessible to VR user {{not done}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
** Show splash screen as quad layer for runtime to composite {{done}}
** Cube map/quads background provided by aircraft {{progressbar|20}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Launcher {{not done}}
** Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
** Transition to flightgear {{not done}}
** Transition to aircraft center with VR already started {{not done}}
** Transition back to flightgear from aircraft center {{not done}}
** More VR friendly environment
*** A simple background cubemap, shared with splash {{not done}}
*** A hanger (like xplane11)? {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
** Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
*** Grasp surfaces {{{progressbar|90}}
*** Force feedback for haptic gloves {{not done}}
** Advanced interaction with animated controls {{progressbar|90}}
*** Direct manipulation mouse mode for testing without VR {{progressbar|90}}
*** Inverse kinematics of translate and rotate animations {{done}}
*** Ability to specify frictions {{not done}}
*** Ability to specify ranges (where can't be deduced) {{not done}}
*** Ability to hint what aircraft processes/filters/kinematics should be reversible {{progressbar|10}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2023-04-30T14:05:16Z

Amalon: /* User facing features */ A bunch of hand tracking and direct manipulation work is ongoing and progressing well

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 22nd August 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
* If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make launcher accessible to VR user {{not done}}
** Make splash screen VR friendly {{done}}
** Make GUI accessible to VR user {{not done}}
** Make aircraft center accessible to VR user {{not done}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
** Show splash screen as quad layer for runtime to composite {{done}}
** Cube map/quads background provided by aircraft {{progressbar|20}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Launcher {{not done}}
** Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
** Transition to flightgear {{not done}}
** Transition to aircraft center with VR already started {{not done}}
** Transition back to flightgear from aircraft center {{not done}}
** More VR friendly environment
*** A simple background cubemap, shared with splash {{not done}}
*** A hanger (like xplane11)? {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
** Hand tracking (robot hand) {{progressbar|90}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
** Advanced interaction with animated controls {{progressbar|90}}
** Holding axis controls (joystick, yoke, helicopter collective) {{progressbar|90}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Knob and Slider Duplication

2023-02-11T18:58:02Z

Amalon: example unknown location

Since 2013, knob & slider animations are effectively duplicated if they already take part in certain animations resulting in exaggerated motion.
This incorrect behaviour will be removed in a future version, which will reduce the motion of these objects.

Warnings have been added to next when this happens. Aircraft developers can work around the issue and silence the warnings by rearranging the XML so that the knob or slider animation node comes before the other animation node. You may need to multiply the animation factor or property-adjust ranges to match the current range of motion.

== Example warning ==
<pre>
[ALRT]:general Deprecation Alert: The knob/slider animations listed below are getting duplicated.
[ALRT]:general This will be fixed in a future version. Please apply workarounds to ensure compatibility.
[ALRT]:general For details see: https://wiki.flightgear.org/Knob_and_Slider_Duplication
...
[ALRT]:general Warning: 2x duplication of knob animation on object "AdjustKnob" is deprecated behaviour (see above)
[ALRT]:general Animation node at: .../fgdata/Aircraft/c172p/Models/Interior/Panel/Instruments/AI/AI-fg1000.xml:142
[ALRT]:general Duplicated due to: .../fgdata/Aircraft/c172p/Models/Interior/Panel/Instruments/AI/AI-fg1000.xml:38
</pre>

If you see mention of unknown locations such as the log output below, please raise it on the mailing list so that the code can be added to report which node is to blame:
<pre>
[ALRT]:general Duplicated scene node '<name>' at unknown location
[ALRT]:general parent 0 named '<parent0>'
[ALRT]:general parent 1 named '<parent1>'
</pre>

== Conditions under which this happens ==

Animation nodes of type "knob" and "slider" are affected (in the example above, the "Animation node" starting at AI-fg1000.xml line 142), for any objects which as a result of another animation (in the example above, the "Duplicated due to" node starting at AI-fg1000.xml line 38) causes the object to appear multiple times in the scene graph. Animations which can cause this duplication in the scene graph include:
* "material" animations
* "pick", "knob" and "slider" animations
* "range" animations

Note that some knob and slider animations are used purely for pick bindings, relying on rotate or translate animations for the actual motion. These are unaffected, and won't trigger warnings.
In these cases the knob / slider animation either won't have an axis set, or won't have a property set, in which case they already have no effect and the duplication is already fixed without compatibility issues.

== The incorrect behaviour ==

In versions prior to 2013, the animation will have been applied once.
In versions since 2013, the animation will be applied multiple times (2 times in the example above), so the effect of the animation will be multiplied.
When the issue is fixed in some future version, the animation will again only be applied once.

Therefore any affected animations created between 2013 and 2023 will have compatibility issues.

== The workaround ==

The XML can be rearranged such that the knob / slider animation (in the example above, the "Animation node" starting at AI-fg1000.xml line 142) is before any causing nodes (in the example above, the "Duplicated due to" node starting at AI-fg1000.xml line 38).

This causes the knob / slider animation to be applied first, avoiding the issue and resulting in consistent behaviour both now and in future versions.

If the animation has been configured by trial and error since 2013, you will need to multiply the animation factor or property-adjust ranges to match the current range of motion (in the example above, by 2x).

Knob and Slider Duplication

2023-02-10T19:13:44Z

Amalon: Some more details

Since 2013, knob & slider animations are effectively duplicated if they already take part in certain animations resulting in exaggerated motion.
This incorrect behaviour will be removed in a future version, which will reduce the motion of these objects.

Warnings have been added to next when this happens. Aircraft developers can work around the issue and silence the warnings by rearranging the XML so that the knob or slider animation node comes before the other animation node. You may need to multiply the animation factor or property-adjust ranges to match the current range of motion.

== Example warning ==
<pre>
[ALRT]:general Deprecation Alert: The knob/slider animations listed below are getting duplicated.
[ALRT]:general This will be fixed in a future version. Please apply workarounds to ensure compatibility.
[ALRT]:general For details see: https://wiki.flightgear.org/Knob_and_Slider_Duplication
...
[ALRT]:general Warning: 2x duplication of knob animation on object "AdjustKnob" is deprecated behaviour (see above)
[ALRT]:general Animation node at: .../fgdata/Aircraft/c172p/Models/Interior/Panel/Instruments/AI/AI-fg1000.xml:142
[ALRT]:general Duplicated due to: .../fgdata/Aircraft/c172p/Models/Interior/Panel/Instruments/AI/AI-fg1000.xml:38
</pre>

== Conditions under which this happens ==

Animation nodes of type "knob" and "slider" are affected (in the example above, the "Animation node" starting at AI-fg1000.xml line 142), for any objects which as a result of another animation (in the example above, the "Duplicated due to" node starting at AI-fg1000.xml line 38) causes the object to appear multiple times in the scene graph. Animations which can cause this duplication in the scene graph include:
* "material" animations
* "pick", "knob" and "slider" animations
* "range" animations

Note that some knob and slider animations are used purely for pick bindings, relying on rotate or translate animations for the actual motion. These are unaffected, and won't trigger warnings.
In these cases the knob / slider animation either won't have an axis set, or won't have a property set, in which case they already have no effect and the duplication is already fixed without compatibility issues.

== The incorrect behaviour ==

In versions prior to 2013, the animation will have been applied once.
In versions since 2013, the animation will be applied multiple times (2 times in the example above), so the effect of the animation will be multiplied.
When the issue is fixed in some future version, the animation will again only be applied once.

Therefore any affected animations created between 2013 and 2023 will have compatibility issues.

== The workaround ==

The XML can be rearranged such that the knob / slider animation (in the example above, the "Animation node" starting at AI-fg1000.xml line 142) is before any causing nodes (in the example above, the "Duplicated due to" node starting at AI-fg1000.xml line 38).

This causes the knob / slider animation to be applied first, avoiding the issue and resulting in consistent behaviour both now and in future versions.

If the animation has been configured by trial and error since 2013, you will need to multiply the animation factor or property-adjust ranges to match the current range of motion (in the example above, by 2x).

Knob and Slider Duplication

2023-02-10T18:09:32Z

Amalon: Initial text about this issue

Virtual Reality

2022-09-11T17:10:31Z

Amalon: aur packages now depend on osgxr

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 22nd August 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
* If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make launcher accessible to VR user {{not done}}
** Make splash screen VR friendly {{done}}
** Make GUI accessible to VR user {{not done}}
** Make aircraft center accessible to VR user {{not done}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
** Show splash screen as quad layer for runtime to composite {{done}}
** Cube map/quads background provided by aircraft {{progressbar|20}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Launcher {{not done}}
** Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
** Transition to flightgear {{not done}}
** Transition to aircraft center with VR already started {{not done}}
** Transition back to flightgear from aircraft center {{not done}}
** More VR friendly environment
*** A simple background cubemap, shared with splash {{not done}}
*** A hanger (like xplane11)? {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
** Hand tracking (robot hand) {{not done}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
** Advanced interaction with animated controls {{not done}}
** Holding axis controls (joystick, yoke, helicopter collective) {{not done}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2022-08-31T22:12:58Z

Amalon: /* Roadmap */ thoughts for vr launcher

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, if you're using the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git, as long as flightgear-git was built from scratch after installing osgxr it should use it.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 22nd August 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
* If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make launcher accessible to VR user {{not done}}
** Make splash screen VR friendly {{done}}
** Make GUI accessible to VR user {{not done}}
** Make aircraft center accessible to VR user {{not done}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
** Show splash screen as quad layer for runtime to composite {{done}}
** Cube map/quads background provided by aircraft {{progressbar|20}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Launcher {{not done}}
** Show launcher in VR similar to GUI when enabled via command line or launcher option {{not done}}
** Transition to flightgear {{not done}}
** Transition to aircraft center with VR already started {{not done}}
** Transition back to flightgear from aircraft center {{not done}}
** More VR friendly environment
*** A simple background cubemap, shared with splash {{not done}}
*** A hanger (like xplane11)? {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
** Hand tracking (robot hand) {{not done}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
** Advanced interaction with animated controls {{not done}}
** Holding axis controls (joystick, yoke, helicopter collective) {{not done}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2022-08-22T17:14:58Z

Amalon: /* Source code */ link to windows nightlies

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, if you're using the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git, as long as flightgear-git was built from scratch after installing osgxr it should use it.
** On Windows, you can use the [[Nightly Build#Windows|Windows nightly builds]].
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 22nd August 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
* If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make GUI accessible to VR user {{not done}}
** Make splash screen VR friendly {{done}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
** Show splash screen as quad layer for runtime to composite {{done}}
** Cube map/quads background provided by aircraft {{progressbar|20}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
** Hand tracking (robot hand) {{not done}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
** Advanced interaction with animated controls {{not done}}
** Holding axis controls (joystick, yoke, helicopter collective) {{not done}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2022-08-22T17:10:07Z

Amalon: /* Status */ link to windows nightlies, since they now work

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, if you're using the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git, as long as flightgear-git was built from scratch after installing osgxr it should use it.
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 22nd August 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. [[Nightly Build#Windows|Windows nightly builds]]):
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
* If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make GUI accessible to VR user {{not done}}
** Make splash screen VR friendly {{done}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
** Show splash screen as quad layer for runtime to composite {{done}}
** Cube map/quads background provided by aircraft {{progressbar|20}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
** Hand tracking (robot hand) {{not done}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
** Advanced interaction with animated controls {{not done}}
** Holding axis controls (joystick, yoke, helicopter collective) {{not done}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2022-07-30T23:18:37Z

Amalon: /* User facing features */ some equirect splash background experimentation works well, but monado has issues and no fallback for steamvr

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, if you're using the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git, as long as flightgear-git was built from scratch after installing osgxr it should use it.
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 31st July 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
* If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make GUI accessible to VR user {{not done}}
** Make splash screen VR friendly {{done}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
** Show splash screen as quad layer for runtime to composite {{done}}
** Cube map/quads background provided by aircraft {{progressbar|20}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
** Hand tracking (robot hand) {{not done}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
** Advanced interaction with animated controls {{not done}}
** Holding axis controls (joystick, yoke, helicopter collective) {{not done}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2022-07-30T23:16:32Z

Amalon: /* User facing features */ basic rendering of controllers (cuboids)

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, if you're using the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git, as long as flightgear-git was built from scratch after installing osgxr it should use it.
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 31st July 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
* If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make GUI accessible to VR user {{not done}}
** Make splash screen VR friendly {{done}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
** Show splash screen as quad layer for runtime to composite {{done}}
** Cube map/quads background provided by aircraft {{not done}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{progressbar|20}}
** Hand tracking (robot hand) {{not done}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
** Advanced interaction with animated controls {{not done}}
** Holding axis controls (joystick, yoke, helicopter collective) {{not done}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2022-07-30T23:15:11Z

Amalon: /* Status */ rendering of cuboid VR controllers

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, if you're using the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git, as long as flightgear-git was built from scratch after installing osgxr it should use it.
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 31st July 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.
* Rendering of controllers (as basic cuboids).

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
* If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make GUI accessible to VR user {{not done}}
** Make splash screen VR friendly {{done}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
** Show splash screen as quad layer for runtime to composite {{done}}
** Cube map/quads background provided by aircraft {{not done}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{not done}}
** Hand tracking (robot hand) {{not done}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
** Advanced interaction with animated controls {{not done}}
** Holding axis controls (joystick, yoke, helicopter collective) {{not done}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2022-07-12T22:09:33Z

Amalon: /* User facing features */ splash screen using quad

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, if you're using the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git, as long as flightgear-git was built from scratch after installing osgxr it should use it.
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 12th July 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
* If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make GUI accessible to VR user {{not done}}
** Make splash screen VR friendly {{done}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|80}}
** Show splash screen as quad layer for runtime to composite {{done}}
** Cube map/quads background provided by aircraft {{not done}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{not done}}
** Hand tracking (robot hand) {{not done}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
** Advanced interaction with animated controls {{not done}}
** Holding axis controls (joystick, yoke, helicopter collective) {{not done}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2022-07-12T22:08:43Z

Amalon: /* Development targets */ splash screen is vr friendly now

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, if you're using the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git, as long as flightgear-git was built from scratch after installing osgxr it should use it.
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 12th July 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
* If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make GUI accessible to VR user {{not done}}
** Make splash screen VR friendly {{done}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|50}}
** Show splash screen as quad layer for runtime to composite {{progressbar|90}} {{priority|2}}
** Cube map/quads background provided by aircraft {{not done}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{not done}}
** Hand tracking (robot hand) {{not done}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
** Advanced interaction with animated controls {{not done}}
** Holding axis controls (joystick, yoke, helicopter collective) {{not done}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2022-07-12T22:06:46Z

Amalon: /* Performance */ point to corectrl for improved performance on AMD GPUs in particular

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, if you're using the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git, as long as flightgear-git was built from scratch after installing osgxr it should use it.
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 12th July 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>
* If using an AMD GPU, use [https://gitlab.com/corectrl/corectrl CoreCtrl] and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make GUI accessible to VR user {{not done}}
** Make splash screen VR friendly {{progressbar|90}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|50}}
** Show splash screen as quad layer for runtime to composite {{progressbar|90}} {{priority|2}}
** Cube map/quads background provided by aircraft {{not done}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{not done}}
** Hand tracking (robot hand) {{not done}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
** Advanced interaction with animated controls {{not done}}
** Holding axis controls (joystick, yoke, helicopter collective) {{not done}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2022-07-12T22:01:08Z

Amalon: /* Status */ splash screen is now vr friendly on next

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, if you're using the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git, as long as flightgear-git was built from scratch after installing osgxr it should use it.
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 12th July 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* VR friendly splash screen.
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* <s>Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.</s>
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make GUI accessible to VR user {{not done}}
** Make splash screen VR friendly {{progressbar|90}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|50}}
** Show splash screen as quad layer for runtime to composite {{progressbar|90}} {{priority|2}}
** Cube map/quads background provided by aircraft {{not done}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{not done}}
** Hand tracking (robot hand) {{not done}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
** Advanced interaction with animated controls {{not done}}
** Holding axis controls (joystick, yoke, helicopter collective) {{not done}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2022-02-09T00:00:27Z

Amalon: /* Status */ update status date

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, if you're using the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git, as long as flightgear-git was built from scratch after installing osgxr it should use it.
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 9th Feb 2022'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make GUI accessible to VR user {{not done}}
** Make splash screen VR friendly {{progressbar|90}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|50}}
** Show splash screen as quad layer for runtime to composite {{progressbar|90}} {{priority|2}}
** Cube map/quads background provided by aircraft {{not done}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{not done}}
** Hand tracking (robot hand) {{not done}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
** Advanced interaction with animated controls {{not done}}
** Holding axis controls (joystick, yoke, helicopter collective) {{not done}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]

Virtual Reality

2022-02-08T23:57:14Z

Amalon: /* Roadmap */ splash screen progress

{{Experimental|disclaimer=VR support in flightgear still comes with a number of caveats.}}

{{infobox subsystem
|image = VR screenshot c182s.png
|name = Virtual Reality
|started = 07/2021
|description = Virtual Reality support using osgXR / OpenXR
|status = Active
|developers = James Hogan <ref>https://sourceforge.net/u/amalon/profile/</ref>
|changelog = https://sourceforge.net/u/amalon/profile/feed.rss
}}

The '''Virtual Reality''' project aims to add immersive VR support to FlightGear. This is done using the OpenXR<ref>{{cite web |url=https://www.khronos.org/OpenXR/ |title=OpenXR Overview |publisher=The Khronos Group Inc}}</ref> API, via an external library called osgXR<ref name="github-osgXR">{{github source |user=amalon |repo=osgXR |full=1}}</ref> for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

== Background ==

A previous attempt to support some form of VR is described in [[OculusImplementation]].

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

== Source code ==

* osgXR source can be found on github<ref name="github-osgXR"/>, or the version included in the flightgear 3rdparty directory can be used.
** On ArchLinux, you can install the osgxr AUR package <ref>https://aur.archlinux.org/packages/osgxr/</ref> to get the version needed by FlightGear's next branches.
* Some VR code can now be found in FlightGear's next branches (see below for details).
** On ArchLinux, if you're using the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git, as long as flightgear-git was built from scratch after installing osgxr it should use it.
* More FlightGear changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=flightgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased). A mirror is available on github for issue tracking<ref>{{github source |user=amalon |repo=flightgear-vr}}</ref>.
* More SimGear changes can be found in James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=simgear |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).
* More FGData changes can be found on James Hogan's osgxr branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr |full=1}}</ref> (non-rebased branch), or osgxr_clean branch<ref>{{sourceforge source |user=amalon |repo=fgdata |branch=osgxr_clean |full=1}}</ref> (clean history, may be rebased).

== Building & Packaging Notes ==

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as <code>-DENABLE_VR=ON</code>). It will do the following:
# Look for the osgXR library, and if found it will link against that (same as <code>-DSYSTEM_OSGXR=ON</code>).
# If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
# If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of <code>-- VR support enabled/disabled...</code>.

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:
* preferably depend upon an osgXR package providing the osgXR dynamic library
* OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
* OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

== Status ==
'''Last updated: 21/01/2021'''

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch:
* Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
* Displaying stereo images on the HMD.
* Motion tracking updates HMD views (but doesn't move main camera).
* Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
* Visibility mask support.

Working in osgxr/osgxr_clean branch:
* Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
* Controller pointing support with some mouse emulation.

Current priorities:
* Improving platform support (Monado in particular).
* Continue to polish controller flight support, add haptics etc.
* Get started on other controller interaction and controller rendering.

=== Known issues ===

* Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.
* Performance could be better so crank down the graphics if frame rate is low.
* Shadows are broken:
** When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
** Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
* Spotlights (taxi/landing lights) are relative to camera in default mode
** Affects default / scene view mode (<code>--prop:/sim/vr/mode=AUTOMATIC</code>), as stereo callbacks position cameras
** Doesn't affect slave cameras mode (<code>--prop:/sim/vr/mode=SLAVE_CAMERAS</code>), as slave cameras are positioned correctly
* ALS filters appear to operate in screen space but don't take into account side by side rendering
* Still some Monado issues:
** <s>Saturated output due to not using SRGB textures</s> (use osgXR 0.3.8)
** <s>Suboptimal depth format used</s> (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
** Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
* Multiple SteamVR issues on Linux:
** If it doesn't close cleanly on Linux it may be due to a SteamVR bug<ref name="github-SteamVR-422"/>, but a ctrl+c / SIGINT on the command line should close it.
* osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

=== Testing ===

VR has been tested with the following hardware & software configurations:
* HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 21:43, 27 August 2021 (UTC)
* Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- [[User:vezza|vezza]] ([[User talk:vezza|talk]]) 16:30 27 August 2021 (UTC)
* HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
* Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- [[User:Amalon|Amalon]] ([[User talk:Amalon|talk]]) 22:36, 17 December 2021 (UTC)
** only basic display to window tested
** osgXR master
** /sim/vr/depth-info=false
* Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers [[User:merspieler|merspieler]] ([[User talk:merspieler|talk]]) 23:27, 17 December 2021 (UTC)
** Applying the following patch: <syntaxhighlight lang="diff">
diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
unsigned int thisDepthBits = 0;
switch (format)
{
- case GL_RGBA16:
+ case GL_RGB10_A2:
// FIXME This one will do for now...
if (!chosenSwapchainFormat)
chosenSwapchainFormat = format;
</syntaxhighlight>

If you've got VR working, feel free to add your configuration above.

The following testing is desired:
* SteamVR, Linux, AMD graphics:
** Test that motion smoothing works (its unsupported on NVidia)
** moving head side to side should reproject using depth buffer
** try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
* Monado, Linux:
** May be awkward to test as it crashes in latest NVidia proprietary drivers<ref>https://gitlab.freedesktop.org/monado/monado/-/issues/122</ref>.
*** There's now a workaround mentioned in that ticket that allows for basic testing
** Use at least osgXR 0.3.6 for monado workaround, or <code>--disable-vr</code> and enable VR with GUI
* SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
** Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
** May need --disable-sentry
** Test headset output and controllers

Issues can be reported:
* On the flightgear-devel [[Mailing lists|mailing list]]
* To the {{tickets|FlightGear bug tracker}} if the issue is reproducible on the next branch
* To the [{{github issues url |user=amalon |repo=flightgear-vr}} flightgear-vr issue tracker]
* To the [{{github issues url |user=amalon |repo=osgXR}} osgXR issue tracker] if it is known to be a non-flightgear specific bug in osgXR

But please be aware of the known caveats mentioned above, and the roadmap below.

== Performance ==

Here are some pointers to get the best VR experience.
* Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
** On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
* If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
** On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
* If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
* Enabling multithreading may get you better performance, e.g. <code>--prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext</code>

== Implementation Details ==

=== Concepts ===

==== Interaction Modes ====

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:
* When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
* When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
* When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
* When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

== Roadmap ==

=== Development targets ===
* Merge stable & minimally functional code into next ({{Merge-request |project=flightgear |id=253}}, {{Merge-request |project=fgdata |id=238}}) {{done}}
** Bare functionality {{done}}
** Stability {{done}}
* Merge some form of desktop mirroring of VR view ({{Merge-request |project=flightgear |id=260}}, {{Merge-request |project=fgdata |id=241}}) {{done}}
* Merge visibility mask support ({{Merge-request |project=simgear |id=98}}, {{Merge-request |project=flightgear |id=264}}, {{Merge-request |project=fgdata |id=244}}) {{done}}
* Polish {{not done}}
** Make GUI accessible to VR user {{not done}}
** Make splash screen VR friendly {{progressbar|90}}
* Functionality {{progressbar|40}}
** Controller flight support {{progressbar|80}}
*** Haptic feedback {{not done}}
** Controller interaction support {{progressbar|40}}
** Walker locomotion / room scale {{not done}}

=== General tasks ===

* Stability {{done}}
** Clean shutdown {{done}}
*** SteamVR for Linux hangs on OpenXR app shutdown<ref name="github-SteamVR-422">{{github issues url|ValveSoftware|SteamVR-for-Linux|number=422}}</ref>.
** Handle OpenXR stopping {{done}}
*** SteamVR for Linux kills running app even if it ends session<ref>{{github issues url|ValveSoftware|SteamVR-for-Linux|number=461}}</ref>.
* Resolve questions around VR mirror switching {{done}}
* Performance {{progressbar|10}}
** Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) {{not done}}
** SceneView mode: Combine culling traversal {{not done}}
** Check why update isn't proceeding in parallel with multithreading {{not done}}
** Use OpenXR visibility mask extension to avoid rendering to invisible fragments {{done}}
* Sound {{progressbar|0}}
** Ensure surround sound is relative to user's head {{not done}}
** Investigate a way to use appropriate OpenAL binaural mode automatically {{not done}}
** Specify separate sound device for VR so it switches automatically? {{not done}}

=== User facing features ===

* Display {{progressbar|60}}
** Show views on HMD {{done}}
** Configurable mirror on desktop window {{done}}
*** Left, Right, Left & Right {{done}}
*** Composited mirror mode (combine & blend views and allow zooming) {{paused}}
** Geometry shader multi viewport {{not done}}
** OVR_multiview2 with OSG branch {{not done}}
* Motion tracking {{progressbar|50}}
** Follow motion of HMD {{done}}
** Move master camera instead of scene slave cameras (presumably would fix surround sound) {{not done}}
* Settings {{progressbar|20}}
** Mirror settings {{done}}
** Dynamic toggling of VR {{done}}
** Sound settings? {{not done}}
*** Sound reference frame (user's head or room)? {{not done}}
** Advanced VR settings {{not done}}
*** validation layer {{not done}}
*** submit depth information {{done}} but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
*** osgXR VR mode {{not done}}
**** Fix slave cameras mode clearing {{not done}}
*** osgXR swapchain mode {{not done}}
* Platform support {{progressbar|70}}
** Linux/X11/OpenGL graphics bindings {{done}}
** Monado support {{progressbar|60}}
** Windows/OpenGL graphics bindings {{done}}
** Windows Mixed Reality render via DirectX bindings {{not done}}
* Splash screen {{progressbar|50}}
** Show splash screen as quad layer for runtime to composite {{progressbar|90}} {{priority|2}}
** Cube map/quads background provided by aircraft {{not done}}
* GUI {{progressbar|10}} {{priority|2}}
** Shown over mirror on desktop window {{done}}
** Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate {{not done}}
** Interact with GUI buttons etc using controllers as lasers {{not done}}
* Controller support {{progressbar|50}}
** Basic action support in osgXR API {{done}}
** Tie osgXR actions into FG's input system {{done}}
** Helicopter flight controls (can use for planes too) {{done}}
** Plane flight controls {{not done}}
** Render controllers {{not done}}
** Hand tracking (robot hand) {{not done}}
** Render skinned hand {{not done}}
** Basic interaction with controls {{progressbar|80}}
** Advanced interaction with animated controls {{not done}}
** Holding axis controls (joystick, yoke, helicopter collective) {{not done}}
** Haptic feedback {{not done}}
* Walker VR locomotion / Room scale {{progressbar|0}}
** Walker follows user {{not done}}
** Parabolic arc teleport {{not done}}
** Snap into seats {{not done}}
** Easy turning around {{not done}}

== References ==
{{Appendix}}

[[Category:Core development projects]]