APS For Oculus

APS now includes direct Oculus SDK support for Quest finger tracking, face and eye tracking and mixed reality passthrough for Quest 2, 3 and Quest Pro over Oculus-Link.

Oculus Branch

By default Steam installs the SteamVR branch of APS. However to take advantage of the Quest's features it is necessary to install the Oculus branch of APS.

To install the Oculus branch select the game from Steam and right click on the title of the Playtest (or the Full version) and select Properties > Betas and select the latest build for "Oculus" from the dropdown:

Make sure you have these before continuing:

Next, in order to use the Quest's tracking features it's necessary to enable developer options from the Oculus app.

Developer Features

Enable the developer features from the Oculus app's settings for features such as passthrough, eye tracking and expressions to work with Mocap Fusion.

Create An Organizatino

Create a new organization Or go to: https://developer.oculus.com/manage/organizations/create 

The Meta account should become a developer account by creating an organization. Restart your Oculus app and you will then be able to see the Developer Runtime Features in the Beta tab of the Oculus app. 

Finger Tracking

Finger tracking is supported for most avatars that include more than one phalanges.

When the user sets down the controllers and holds their hands directly in front of the HMD then finger tracking is automatically enabled on the avatar. 

By picking up the controllers again will disable hand tracking and switches back to UI laser pointers:

menu.mp4

Accessing the VR Menu

It is also possible to access the VR menu when using finger tracking!!
To open the VR menu simply pinch for 3 seconds with your hand out in front of the HMD.

To recap we have:

Mixed Reality Passthrough

One of the coolest features in the Oculus branch is the mixed reality motion capture review. This allows reviewing motion capture in a mixed reality environment and makes avatars appear to be in the same room! 

Mixed reality is a great way to see obstacles in the room when recording mocap.

To enable mixed reality click on the Passthrough toggle located at the bottom of the VR menu:

passthrough2.mp4

Install the Oculus Developer Hub if you want to record the Mixed Reality mocap directly from the headset:

mixedreality.mp4

Use the Animation Editor tools to live puppet avatars in mixed reality (with lighting and shadows):

If you are using only a HMD and controllers then the next steps are not required!

If you would like to include full body tracking using SteamVR Vive trackers then please continue reading.

Body trackers are supported in the Oculus branch but require a simple UDP server to stream data from SteamVR into Mocap Fusion.

Enter the IP to your PC or use: 127.0.0.1
Alternatively try the IP: 255.255.255.255

Body Tracking

APS OVR-Tracker Data Server 

The OVR Tracker server uses the openvr api from SteamVR to stream Vive tracker coordinates data to Mocap Fusion. 

You can download the Tracker Data Server from the download button. Be sure that SteamVR is running before starting the server!

SteamVR Lighthouse

Body Tracking

Debugging Window

When launching the server a window appears and prints status for each tracker as it streams data out of the UDP port.

Trackers are valid and report as "TRUE" when SteamVR is active and a VIve tracker has been connected. 

Trackers report "FALSE" if there is no device connected or while SteamVR's menu is open. If all trackers show as reporting "FALSE" open the VR view and close the SteamVR menu if it is open.

Note that SteamVR must to be installed for the Vive trackers to function.
You may also wish to enable the SteamVR null-driver to avoid HMD errors.

Trouble Shooting

Note: If you notice SteamVR status reports "Standing By" and trackers are not responding (all False), try closing the APS Tracker Data Server and start the server again.

If that does not work then likely the Steam dashboard might be opened in VR and can't be closed since there is no HMD headset connected to steamVR and no controllers..

Try clicking on "Toggle Dashboard" and the trackers should resume tracking and will be valid/true.

Tracker Setup

When using Steam trackers when Quest HMDs it is necessary to calibrate the tracking space offsets of the two playspaces.

In APS you can simply assign the left and right feet trackers then calibrate T-Pose and the playspace alignment is then calibrated and the two playspaces should agree.

Optional Space Tracker

A tracker can be attached directly to the Quest HMD to help keep both playspaces aligned at all times but is not required. Read more: Space Tracker setup tutorial

CalibrateTracker.mp4

Above video shows how to calibrate Trackers and Space offsets using just left and right feet tracker (center) as the alignment point.

Space Calibrator.mp4

Alternatively using the Built-In Space-Calibrator to align the playspace offsets when making a figure "∞" using the controller and tracker together.

feet_trackers.mp4

Optional Feet Trackers

Simultaneous controllers and hand tracking (multi-modal mode) are now supported! This allows you to use Touch controllers as feet trackers for full body tracking while finger tracking is enabled.

Multi-Modal Feet Tracking Toggle

Disabling the "FEET TRACKING" toggle will disable multi-modal mode and controllers will only work when held. This disabled or enables simultaneous controllers. When multi-modal mode is enable after setting down the controllers you will see the controllers shown as trackers and listed in the Tracker setup menu.

T-Pose Calibration

Full Body Tracking + Quest Finger Tracking 

Before calibrating tpose you should open a VR Mirror in front of the avatar and double check that the trackers are aligned with the avatar, also check they are moving correctly in VR!

Note: Calibrating the floor height before calibrating t-pose is often necessary!

Note: if the Space tracker has not been properly aligned on the HMD it might need slight adjusting. If the other trackers in the scene appear offset then try physically rotating the Space trackers on the HMD slightly and recalibrating.

Face Tracking

If you are using a Quest Pro you can include face tracking with when building custom avatars. Since APS_SDK (v7.8.1) - Meta Facecap Linker V2 update now includes support for all face tracking blendshapes for Quest Pro (v60 OpenXR update).

Thanks for reading ❤️