Virtual Reality

From FlightGear wiki
Jump to navigation Jump to search
Caution  The feature discussed below is currently considered experimental, i.e. must be considered proof-of-concept for the time being. VR support in flightgear still comes with a number of caveats. It's just to showcase what's being worked on. If you'd like to learn more, please get in touch via the developers mailing list.
Virtual Reality
VR screenshot c182s.png
Started in 07/2021
Description Virtual Reality support using osgXR / OpenXR
Contributor(s) James Hogan [1]
Status Active
Changelog https://sourceforge.net/u/amalon/profile/feed.rss

The Virtual Reality project aims to add immersive VR support to FlightGear. This is done using the OpenXR[2] API, via an external library called osgXR[3] for handling OpenXR and OpenSceneGraph specifics, which is being developed in parallel.

Background

A previous attempt to support some form of VR is described in OculusImplementation.

The portable OpenXR API now seems to be getting widespread adoption, which will allow FlightGear to support a variety of VR devices across multiple platforms.

Source code

  • osgXR source can be found on github.[3]
    • On ArchLinux, you can install the osgxr AUR package [4] to get the version needed by FlightGear's next branches.
  • Some VR code can now be found in FlightGear's next branches (see below for details).
    • On ArchLinux, if you're using the -git AUR packages, i.e. flightgear-git, simgear-git, flightgear-data-git, as long as flightgear-git was built from scratch after installing osgxr it should use it.
  • More FlightGear changes can be found on James Hogan's osgxr branch[5] (non-rebased branch), or osgxr_clean branch[6] (clean history, may be rebased). A mirror is available on github for issue tracking[7].
  • More SimGear changes can be found in James Hogan's osgxr branch[8] (non-rebased branch), or osgxr_clean branch[9] (clean history, may be rebased).
  • More FGData changes can be found on James Hogan's osgxr branch[10] (non-rebased branch), or osgxr_clean branch[11] (clean history, may be rebased).

Status

Last updated: 08/10/2021

Highly experimental, partially in next, partly out of tree. Development in progress.

Working in next branch:

  • Enable/disable at startup via --enable-vr / --disable-vr command line options or /sim/vr/enabled property.
  • Displaying stereo images on the HMD.
  • Motion tracking updates HMD views, but doesn't move main camera.
  • Configurable VR mirror (see View->VR Settings) shown in default camera view (blank, left, right, side-by-side left & right).
  • Visibility mask support.

Working in osgxr/osgxr_clean branch:

  • Improved osgXR detection and VR enable/disable with CMake.

Caveats:

  • osgXR currently only supports Linux/X11. Adding alternative graphics bindings should be fairly simple, but isn't a priority for me. Contributions welcome.
  • Splash screen is rendered directly onto the left eye viewport which isn't very pleasant in VR.
  • Performance could be better so crank down the graphics if frame rate is low.
  • If it doesn't close cleanly it may be due to a SteamVR bug[12], but a ctrl+c / SIGINT on the command line should close it.

Current priorities:

  • Get started on controller support

Testing

VR has been tested with the following hardware & software configurations:

  • HTC Vive, SteamVR, Linux/X11 (ArchLinux), nVidia proprietary -- Amalon (talk) 21:43, 27 August 2021 (UTC)
  • Oculus Quest 2, SteamVR (ALVR), Linux/X11 (Fedora 34) -- vezza (talk) 16:30 27 August 2021 (UTC)

If you've got VR working, feel free to add your configuration above.

The following testing is desired:

  • SteamVR, Linux, AMD graphics:
    • Test that motion smoothing works (its unsupported on NVidia)
    • moving head side to side should reproject using depth buffer
    • try suspending fgfs with Ctrl+Z in terminal (resume with fg) to force reprojection
  • Monado, Linux:
    • I can't test as it crashes in latest NVidia proprietary drivers[13].

Issues can be reported:

But please be aware of the known caveats mentioned above, and the roadmap below.

Performance

Here are some pointers to get the best VR experience.

  • Ensure your runtime supports asynchronous reprojection, which may alleviate the nausea induced by lower frame rates.
    • On Linux / SteamVR / NVidia proprietary driver, ensure you have at least driver version 270.
  • If your frame rate is being capped at the refresh rate of your desktop monitor, try disabling VSync in both flightgear settings and your driver settings
    • On Linux / NVidia proprietary driver, its found in OpenGL settings in nvidia-settings.
  • If FlightGear is getting throttled by SteamVR (to try and ensure a consistent, if slow, frame rate), consider switching to fixed throttling (in per-application video settings) and playing with the sliders. You may prefer no frame rate throttling but a higher predicted frame time.
  • Enabling multithreading may get you better performance, e.g. --prop:/sim/rendering/multithreading-mode=CullThreadPerCameraDrawThreadPerContext

Roadmap

Development targets

General tasks

  • Stability Done Done
    • Clean shutdown Done Done
      • SteamVR for Linux hangs on OpenXR app shutdown[12].
    • Handle OpenXR stopping Done Done
      • SteamVR for Linux kills running app even if it ends session[14].
  • Resolve questions around VR mirror switching Done Done
  • Performance 10}% completed
    • Handle shouldRender=false by somehow disabling VR cameras (maybe setting cull mask to 0) Not done Not done
    • SceneView mode: Combine culling traversal Not done Not done
    • Check why update isn't proceeding in parallel with multithreading Not done Not done
    • Use OpenXR visibility mask extension to avoid rendering to invisible fragments Done Done
  • Sound 0}% completed
    • Ensure surround sound is relative to user's head Not done Not done
    • Investigate a way to use appropriate OpenAL binaural mode automatically Not done Not done
    • Fix: It always seems to select speakers rather than headphones on HTC Vive Not done Not done

User facing features

  • Display 60}% completed
    • Show views on HMD Done Done
    • Configurable mirror on desktop window Done Done
      • Left, Right, Left & Right Done Done
      • Composited mirror mode (combine & blend views and allow zooming) Paused Paused
    • Geometry shader multi viewport Not done Not done
    • OVR_multiview2 with OSG branch Not done Not done
  • Motion tracking 50}% completed
    • Follow motion of HMD Done Done
    • Move master camera instead of scene slave cameras (presumably would fix surround sound) Not done Not done
  • Settings 20}% completed
    • Mirror settings Done Done
    • Dynamic toggling of VR Done Done
    • Sound settings? Not done Not done
      • Sound reference frame (user's head or room)? Not done Not done
    • Advanced VR settings Not done Not done
      • validation layer Not done Not done
      • submit depth information Done Done but needs testing (SteamVR+Linux+nVidia doesn't do motion smoothing yet)
      • osgXR VR mode Not done Not done
        • Fix slave cameras mode clearing Not done Not done
      • osgXR swapchain mode Not done Not done
  • Platform support 30}% completed
    • Linux/X11/OpenGL graphics bindings Done Done
    • Windows/OpenGL graphics bindings Not done Not done
    • Windows Mixed Reality render via DirectX bindings Not done Not done
  • Splash screen 0}% completed
    • Show splash screen as quad layer for runtime to composite Not done Not done Priority
    • Cube map/quads background provided by aircraft Not done Not done
  • GUI 10}% completed Priority
    • Shown over mirror on desktop window Done Done
    • Expose a menubar & GUI, rendered to a quad or cylinder in front of VR user when they activate Not done Not done
    • Interact with GUI buttons etc using controllers as lasers Not done Not done
  • Controller support 10}% completed
    • Basic action support in osgXR API Done Done
    • Tie osgXR actions into FG's input system Not done Not done
    • Render controllers Not done Not done
    • Hand tracking (robot hand) Not done Not done
    • Render skinned hand Not done Not done
    • Basic interaction with controls Not done Not done
    • Advanced interaction with animated controls Not done Not done
    • Holding axis controls (joystick, yoke, helicopter collective) Not done Not done
    • Haptic feedback Not done Not done
  • Walker VR locomotion / Room scale 0}% completed
    • Walker follows user Not done Not done
    • Parabolic arc teleport Not done Not done
    • Snap into seats Not done Not done
    • Easy turning around Not done Not done

References

References