Virtual Reality

From FlightGear wiki
Jump to navigation Jump to search
Caution  The feature discussed below is currently considered experimental, i.e. must be considered proof-of-concept for the time being. VR support in flightgear still comes with a number of caveats. It's just to showcase what's being worked on. If you'd like to learn more, please get in touch via the developers mailing list.
Virtual Reality
VR screenshot c182s.png
Started in 07/2021
Description Virtual Reality support using osgXR / OpenXR
Contributor(s) James Hogan [1]
Status Active
Changelog https://sourceforge.net/u/amalon/profile/feed.rss

The Virtual Reality project aims to add immersive VR support to FlightGear. This is done using the OpenXR[2] API, via an external library called osgXR[3] for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

Background

A previous attempt to support some form of VR is described in OculusImplementation.

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

Source code

  • osgXR source can be found on github[3], or the version included in the flightgear 3rdparty directory can be used.
    • On ArchLinux, you can install the osgxr AUR package [4] to get the version needed by FlightGear's next branches.
  • Some VR code can now be found in FlightGear's next branches (see below for details).
    • On ArchLinux, you can use the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git.
    • On Windows, you can use the Windows nightly builds.
  • More FlightGear changes can be found on James Hogan's osgxr branch[5] (non-rebased branch), or osgxr_clean branch[6] (clean history, may be rebased). A mirror is available on github for issue tracking[7].
  • More SimGear changes can be found in James Hogan's osgxr branch[8] (non-rebased branch), or osgxr_clean branch[9] (clean history, may be rebased).
  • More FGData changes can be found on James Hogan's osgxr branch[10] (non-rebased branch), or osgxr_clean branch[11] (clean history, may be rebased).

Building & Packaging Notes

When CMake runs for FlightGear (next or osgxr/osgxr_clean branches), by default it will attempt to enable VR on Windows and Linux (same as -DENABLE_VR=ON). It will do the following:

  1. Look for the osgXR library, and if found it will link against that (same as -DSYSTEM_OSGXR=ON).
  2. If no osgXR library is found it will check for OpenXR and if found it will build its own static built-in copy of osgXR to use.
  3. If no OpenXR is found it will disable VR support.

In any case it will print a message along the lines of -- VR support enabled/disabled....

To avoid implicit dependencies when packaging, packagers may need to do ONE of the following:

  • preferably depend upon an osgXR package providing the osgXR dynamic library
  • OR depend on OpenXR and GLX, and explicitly pass `-DSYSTEM_OSGXR=OFF` to CMake so the built-in osgXR is built statically
  • OR explicitly pass `-DENABLE_VR=OFF` to CMake to disable VR support altogether

Status

Last updated: 31st March 2024

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch (e.g. Windows nightly builds):

  • Enable/disable at startup via --enable-vr / --disable-vr command line options, or from GUI (View->VR Options).
  • VR friendly splash screen.
  • Displaying stereo images on the HMD.
  • Motion tracking updates HMD views (but doesn't move main camera).
  • Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
  • Visibility mask support.

Working in osgxr/osgxr_clean branch:

  • Controller support for flight (helicopter collective in left hand, helicopter cyclic in right).
  • Controller pointing support with some mouse emulation.
  • Rendering of controllers (as basic cuboids).

Current priorities:

  • Support HDR Pipeline and accelerated multiview rendering.
  • Continue to polish controller flight support, add haptics etc.
  • Get started on other controller interaction and controller rendering.

Known issues

  • Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.
  • Performance could be better so crank down the graphics if frame rate is low.
  • Shadows are broken:
    • When shadows are enabled, the FlightGear window doesn't appear to refresh even though buffers are swapped, but VR output works.
    • Sun position is fixed relative to head, presumably because shadow maps are rendered based on master camera, but osgXR only moves slave cams.
  • Spotlights (taxi/landing lights) are relative to camera in default mode
    • Affects default / scene view mode (--prop:/sim/vr/mode=AUTOMATIC), as stereo callbacks position cameras
    • Doesn't affect slave cameras mode (--prop:/sim/vr/mode=SLAVE_CAMERAS), as slave cameras are positioned correctly
  • ALS filters appear to operate in screen space but don't take into account side by side rendering
  • HDR Pipeline is unsupported, but work to support it is in progress
  • Still some Monado issues:
    • Saturated output due to not using SRGB textures (use osgXR 0.3.8)
    • Suboptimal depth format used (osgXR 0.3.8 should end up using GL_DEPTH32F_STENCIL8)
    • Warnings about suggesting bindings of attached action sets when restarting VR (looks like a monado issue at first glance)
  • Multiple SteamVR issues on Linux:
    • If it doesn't close cleanly on Linux it may be due to a SteamVR bug[12], but a ctrl+c / SIGINT on the command line should close it.
  • osgXR doesn't support Direct3D graphics bindings for WMR (it isn't a priority for me). Contributions welcome.

Testing

VR has been tested with the following hardware & software configurations:

  • HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- Amalon (talk) 21:43, 27 August 2021 (UTC)
  • Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- vezza (talk) 16:30 27 August 2021 (UTC)
  • Oculus Quest 2, Oculus App, Windows 10 --Necolatis (talk) 13:22, 7 June 2023 (UTC)
    • Works fine, shadows gets rendered, but move when turning the head.
    • Eye view point is about 0.2 meter above where the view is defined, so I have to duck my head to look out through the HUD and stuff.
  • Oculus Quest 2, SteamVR via Virtual Streamer App, Windows 10, Nightly build --User:Skyclmbr 18:17, 11 October 2023 (UTC)
    • Head tracking works fine, but right eye screen flickers constantly throughout flight
    • Starting head position is off, depending on how much I move around while it's loading on the SteamVR page.
  • HTC Vive, Monado (to window), Linux/X11 (ArchLinux), nVidia proprietary -- Amalon (talk) 22:36, 17 December 2021 (UTC)
    • only basic display to window tested
    • osgXR master
  • Monado dummy driver, Linux/X11 (ArchLinux), Intel mesa -- Amalon (talk) 22:36, 17 December 2021 (UTC)
    • only basic display to window tested
    • osgXR master
    • /sim/vr/depth-info=false
  • Monado Dummy HMD, Linux/X11 (Debian), AMD Mesa Drivers Nia (talk) 23:27, 17 December 2021 (UTC)
    • Applying the following patch: 
      diff --git a/src/XRState.cpp b/src/XRState.cpp
index ce09bcd..5bcbbca 100644
--- a/src/XRState.cpp
+++ b/src/XRState.cpp
@@ -1017,7 +1017,7 @@ XRState::UpResult XRState::upSession()
         unsigned int thisDepthBits = 0;
         switch (format)
         {
-            case GL_RGBA16:
+            case GL_RGB10_A2:
                 // FIXME This one will do for now...
                 if (!chosenSwapchainFormat)
                     chosenSwapchainFormat = format;
  • HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Windows 10, SteamVR -- Gastcan (talk) 02:15, 09 July 2023 (UTC)
    • (Using the Windows nightly build 08-Jul-23)
    • The view follows the direction of the headset, but the image stutters; outside scenery and within the cockpit
  • HP Reverb G2, Gigabyte (AMD) RX 6700 XT, Debian (Trixie), Monado -- Gastcan (talk) 16:15, 14 August 2023 (UTC)
    • Compiled from sources: OpenXR-SDK-Source, Monado, osgXR (amalon), SimGear (amalon), FlightGear (amalon)
    • The view follows the direction of the headset with minor stuttering.
    • The viewpoint/perspective drifts clockwise, to the right. <- Drifting is virtually eliminated by adding 'Basalt for Monado' (provides 6DoF).


If you've got VR working, feel free to add your configuration above.

The following testing is desired:

  • SteamVR, Linux, AMD graphics:
    • Test that motion smoothing works (its unsupported on NVidia)
    • moving head side to side should reproject using depth buffer
    • try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
  • Monado, Linux:
    • May be awkward to test as it crashes in latest NVidia proprietary drivers[13].
      • There's now a workaround mentioned in that ticket that allows for basic testing
    • Use at least osgXR 0.3.6 for monado workaround, or --disable-vr and enable VR with GUI
  • SteamVR, Windows, with a headset (I've only attempted to null driver, and it at least partly worked):
    • Needs building with at least osgXR 0.3.7 for Windows bindings, and my osgxr/osgxr_clean branch of flightgear/fgdata/simgear
    • May need --disable-sentry
    • Test headset output and controllers

Issues can be reported:

But please be aware of the known caveats mentioned above, and the roadmap below.

Performance

Here are some pointers to get the best VR experience.

  • Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
    • On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
  • If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
    • On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
  • If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
  • Enabling multithreading may get you better performance, e.g. --prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext
  • If using an AMD GPU, use CoreCtrl and set the GPU to the "Virtual Reality" power profile with GPU and memory clocked to max. You may also benefit from changing the CPU frequency governor to Performance.

Implementation Details

Concepts

Interaction Modes

Interaction modes are a way of describing (via XML) how a VR controller should look and behave in a particular context. It roughly corresponds to a virtual control you can hold (and let go of), for example:

  • When you're holding a helicopter collective control, the pitch angle of the controller should control the collective pitch (and look like a collective in your hand).
  • When you're pointing the controller at a GUI or cockpit controls it should pick along a line, rendered in a specific way, and emulate mouse events to those controls (and look like a small laser pointer control or a pointing hand/glove).
  • When you're teleporting around, it should pick along a parabolic arc, rendered a specific way, and teleport there when the button is released.
  • When you're using the flight recorder it should treat the controller trackpad as a rewind/fast forward (and look like a fancy TV remote or something).

The definition of interaction modes should be generic enough that custom ones can be provided with aircraft, or defined separately and work with any aircraft, but also powerful enough that a complex and intuitive control can be defined with a minimum of Nasal scripting.

Roadmap

Development targets

  • Merge stable & minimally functional code into next (MR #253 (flightgear), MR #238 (fgdata)) Done Done
    • Bare functionality Done Done
    • Stability Done Done
  • Merge some form of desktop mirroring of VR view (MR #260 (flightgear), MR #241 (fgdata)) Done Done
  • Merge visibility mask support (MR #98 (simgear), MR #264 (flightgear), MR #244 (fgdata)) Done Done
  • Polish Not done Not done
    • Make launcher accessible to VR user Not done Not done
    • Make splash screen VR friendly Done Done
    • Make GUI accessible to VR user Not done Not done
    • Make aircraft center accessible to VR user Not done Not done
  • Functionality 40}% completed
    • Controller flight support 80}% completed
      • Haptic feedback Not done Not done
    • Controller interaction support 40}% completed
    • Walker locomotion / room scale Not done Not done

General tasks

  • Stability Done Done
    • Clean shutdown Done Done
      • SteamVR for Linux hangs on OpenXR app shutdown[12].
    • Handle OpenXR stopping Done Done
      • SteamVR for Linux kills running app even if it ends session[14].
  • Resolve questions around VR mirror switching Done Done
  • Performance 10}% completed
    • Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) Not done Not done
    • SceneView mode: Combine culling traversal Not done Not done
    • Check why update isn't proceeding in parallel with multithreading Not done Not done
    • Use OpenXR visibility mask extension to avoid rendering to invisible fragments Done Done
  • Sound 0}% completed
    • Ensure surround sound is relative to user's head Not done Not done
    • Investigate a way to use appropriate OpenAL binaural mode automatically Not done Not done
    • Specify separate sound device for VR so it switches automatically? Not done Not done

User facing features

  • Display 60}% completed
    • Show views on HMD Done Done
    • Configurable mirror on desktop window Done Done
      • Left, Right, Left & Right Done Done
      • Composited mirror mode (combine & blend views and allow zooming) Paused Paused
    • General HDR Pipeline Support 50}% completed
    • Geometry shader multi viewport (HDR Pipeline) 50}% completed
    • OVR_multiview2 with OSG branch (HDR Pipeline) 20}% completed
  • Motion tracking 50}% completed
    • Follow motion of HMD Done Done
    • Move master camera instead of scene slave cameras (presumably would fix surround sound) Not done Not done
  • Settings 20}% completed
    • Mirror settings Done Done
    • Dynamic toggling of VR Done Done
    • Sound settings? Not done Not done
      • Sound reference frame (user's head or room)? Not done Not done
    • Advanced VR settings Not done Not done
      • validation layer Not done Not done
      • submit depth information Done Done but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
      • osgXR VR mode Not done Not done
        • Fix slave cameras mode clearing Not done Not done
      • osgXR swapchain mode Not done Not done
  • Platform support 70}% completed
    • Linux/X11/OpenGL graphics bindings Done Done
    • Monado support 60}% completed
    • Windows/OpenGL graphics bindings Done Done
    • Windows Mixed Reality render via DirectX bindings Not done Not done
  • Splash screen 80}% completed
    • Show splash screen as quad layer for runtime to composite Done Done
    • Cube map/quads background provided by aircraft 20}% completed
  • GUI 10}% completed Priority
    • Shown over mirror on desktop window Done Done
    • Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate Not done Not done
    • Interact with GUI buttons etc using controllers as lasers Not done Not done
  • Launcher Not done Not done
    • Show launcher in VR similar to GUI when enabled via command line or launcher option Not done Not done
    • Transition to flightgear Not done Not done
    • Transition to aircraft center with VR already started Not done Not done
    • Transition back to flightgear from aircraft center Not done Not done
    • More VR friendly environment
      • A simple background cubemap, shared with splash Not done Not done
      • A hanger (like xplane11)? Not done Not done
  • Controller support 50}% completed
    • Basic action support in osgXR API Done Done
    • Tie osgXR actions into FG's input system Done Done
    • Helicopter flight controls (can use for planes too) Done Done
    • Plane flight controls Not done Not done
    • Render controllers 20}% completed
    • Hand tracking (robot hand) 90}% completed
    • Render skinned hand Not done Not done
    • Basic interaction with controls 80}% completed
      • Grasp surfaces {90}% completed
      • Force feedback for haptic gloves Not done Not done
    • Advanced interaction with animated controls 90}% completed
      • Direct manipulation mouse mode for testing without VR 90}% completed
      • Inverse kinematics of translate and rotate animations Done Done
      • Ability to specify frictions Not done Not done
      • Ability to specify ranges (where can't be deduced) Not done Not done
      • Ability to hint what aircraft processes/filters/kinematics should be reversible 10}% completed
    • Haptic feedback Not done Not done
  • Walker VR locomotion / Room scale 0}% completed
    • Walker follows user Not done Not done
    • Parabolic arc teleport Not done Not done
    • Snap into seats Not done Not done
    • Easy turning around Not done Not done

Aircraft compatibility

This section is largely about how usable controls are in VR with direct manipulation.

Cessna 172P

  • Controls
    • Flaps 90}% completed
      • kinematic needs to be reversible 90}% completed
    • Visors 70}% completed
      • correct relative friction Not done Not done
    • Opening of doors Not done Not done
      • reversing of gain expression Not done Not done

Eurocopter EC130 B4

  • Engine start
    • Engine cutoff lever reversing (Alt+C required) Not done Not done
    • External power on (GUI required) Not done Not done
    • External power off (GUI required) Not done Not done
    • Throttle roll (middle mouse drag required) Not done Not done
  • Take off & flight
    • Arms & hands get a bit in the way Not done Not done
  • Controls
    • Engine cutoff lever Not done Not done
      • Uses nasal, needs pick animation binding of some sort
    • ELT 50}% completed
      • Both /ELT/test and /ELT/armed can be set to bool
      • Both can get set though
    • Doors & hatches Not done Not done
      • Add limits

McDonnell Douglas F-15 Eagle

  • Model
    • Floating switches are very noticeable Not done Not done
    • HUD should be at infinity Not done Not done

Robinson R22

  • Controls
    • Tilting T-bar 90}% completed
      • limit range? Not done Not done
    • Collective awkward to get at Not done Not done
    • Throttle roll & governor switch? Not done Not done

Robinson R44

  • Controls
    • Tilting T-bar Not done Not done

References

References
  1. ↑ https://sourceforge.net/u/amalon/profile/
  2. ↑ OpenXR Overview. Published by The Khronos Group Inc.
  3. ↑ 3.0 3.1 https://github.com/amalon/osgXR
  4. ↑ https://aur.archlinux.org/packages/osgxr/
  5. ↑ https://sourceforge.net/u/amalon/flightgear/ci/osgxr/tree/
  6. ↑ https://sourceforge.net/u/amalon/flightgear/ci/osgxr_clean/tree/
  7. ↑ github/amalon/flightgear-vr
  8. ↑ https://sourceforge.net/u/amalon/simgear/ci/osgxr/tree/
  9. ↑ https://sourceforge.net/u/amalon/simgear/ci/osgxr_clean/tree/
  10. ↑ https://sourceforge.net/u/amalon/fgdata/ci/osgxr/tree/
  11. ↑ https://sourceforge.net/u/amalon/fgdata/ci/osgxr_clean/tree/
  12. ↑ 12.0 12.1 https://github.com/ValveSoftware/SteamVR-for-Linux/issues/422
  13. ↑ https://gitlab.freedesktop.org/monado/monado/-/issues/122
  14. ↑ https://github.com/ValveSoftware/SteamVR-for-Linux/issues/461
Retrieved from "https://wiki.flightgear.org/w/index.php?title=Virtual_Reality&oldid=140808"