APS For Oculus

APS now includes direct Oculus SDK support for Quest finger tracking, face and eye tracking and mixed reality passthrough for Quest 2, 3 and Quest Pro over Oculus-Link.

Oculus Branch

By default Steam installs the SteamVR branch of APS. However to take advantage of the Quest's features it is necessary to install the Oculus branch of APS.

To install the Oculus branch select the game from Steam and right click on the title of the Playtest (or the Full version) and select Properties > Betas and select the latest build for "Oculus" from the dropdown:

Make sure you have these before continuing:

Next, in order to use the Quest's tracking features it's necessary to enable developer options from the Oculus app.

Developer Features

Enable the developer features from the Oculus app's settings for features such as passthrough, eye tracking and expressions to work with Mocap Fusion.

Create An Organizatino

Create a new organization Or go to: https://developer.oculus.com/manage/organizations/create 

The Meta account should become a developer account by creating an organization. Restart your Oculus app and you will then be able to see the Developer Runtime Features in the Beta tab of the Oculus app. 

Finger Tracking

Finger tracking is supported for most avatars that include more than one phalanges.

When the user sets down the controllers and holds their hands directly in front of the HMD then finger tracking is automatically enabled on the avatar. 

By picking up the controllers again will disable hand tracking and switches back to UI laser pointers:

menu.mp4

Accessing the VR Menu

It is also possible to access the VR menu when using finger tracking!!
To open the VR menu simply pinch for 3 seconds with your hand out in front of the HMD.

To recap we have:

Mixed Reality Passthrough

Using the Oculus Developer Hub you can use the HMD as a AR camera and record the VR stream directly from the headset.

mixedreality.mp4

A neat feature available in the Oculus branch is the mixed reality motion capture. This allows reviewing motion capture in a mixed reality environment and makes avatars appear to be in the same room.

Use the Animation Editor tools to live puppet avatars in mixed reality with lighting and shadows.

To enable mixed reality click on the Passthrough toggle located at the bottom of the VR menu:

passthrough2.mp4

Install the Oculus Developer Hub if you want to record the Mixed Reality mocap directly from the headset:

Adjust AR shadows and lighting to match the room lighting exactly!

If you are using only a HMD and controllers then the next steps are not required!

If you would like to include full body tracking using SteamVR Vive trackers then please read the next section:

The Oculus branch automatically detects trackers from SteamVR if SteamVR is running on the same PC. This makes it simple to add Vive trackers:

Body Trackers

Full Body Tracking Fusion

with Quest Finger Tracking

ovr trackers.mp4

Full support was just added for OVR Trackers

available in the latest APS update v8.2.0

SteamVR Lighthouse

Body Tracking

Note that SteamVR must to be installed for the Vive trackers to function.
You may also wish to enable the SteamVR null-driver to avoid HMD errors.

⚠️The legacy Tracker Data Server is no longer required, you can still enable the legacy tracker server inputs if needed read more.

Trouble Shooting

Note: If you notice SteamVR status reports "Standing By" and trackers are not responding (all False), try closing the APS Tracker Data Server and start the server again.

If that does not work then likely the Steam dashboard might be opened in VR and can't be closed since there is no HMD headset connected to steamVR and no controllers..

Try clicking on "Toggle Dashboard" and the trackers should resume tracking and will be valid/true.

Tracker Setup

When using Steam trackers when Quest HMDs it is necessary to calibrate the tracking space offsets of the two playspaces.

In APS you can simply assign the left and right feet trackers then calibrate T-Pose and the playspace alignment is then calibrated and the two playspaces should agree.

Optional Space Tracker

A tracker can be attached directly to the Quest HMD to help keep both playspaces aligned at all times but is not required. Read more: Space Tracker setup tutorial

CalibrateTracker.mp4

Above video shows how to calibrate Trackers and Space offsets using just left and right feet tracker (center) as the alignment point.

Space Calibrator.mp4

Alternatively using the Built-In Space-Calibrator to align the playspace offsets when making a figure "∞" using the controller and tracker together.

feet_trackers.mp4

Optional Feet Trackers

Simultaneous controllers and hand tracking (multi-modal mode) are now supported! This allows you to use Touch controllers as feet trackers for full body tracking while finger tracking is enabled.

Multi-Modal Feet Tracking Toggle

Disabling the "FEET TRACKING" toggle will disable multi-modal mode and controllers will only work when held. This disabled or enables simultaneous controllers. When multi-modal mode is enable after setting down the controllers you will see the controllers shown as trackers and listed in the Tracker setup menu.

T-Pose Calibration

Full Body Tracking + Quest Finger Tracking 

Before calibrating tpose you should open a VR Mirror in front of the avatar and double check that the trackers are aligned with the avatar, also check they are moving correctly in VR!

Note: Calibrating the floor height before calibrating t-pose is often necessary!

Note: if the Space tracker has not been properly aligned on the HMD it might need slight adjusting. If the other trackers in the scene appear offset then try physically rotating the Space trackers on the HMD slightly and recalibrating.

Face Tracking

If you are using a Quest Pro you can include face tracking with when building custom avatars. Since APS_SDK (v7.8.1) - Meta Facecap Linker V2 update now includes support for all face tracking blendshapes for Quest Pro (v60 OpenXR update).

Thanks for reading ❤️