APS For Oculus
APS now includes direct Oculus SDK support for Quest finger tracking, face and eye tracking and mixed reality passthrough for Quest 2, 3 and Quest Pro over Oculus-Link.
Enable mixed reality passthrough and see your room while recording.
Review mocap in a mixed reality environment!
Finger tracking!!
Oculus Branch
By default Steam installs the SteamVR branch of APS. However to take advantage of the Quest's features it is necessary to install the Oculus branch of APS.
To install the Oculus branch select the game from Steam and right click on the title of the Playtest (or the Full version) and select Properties > Betas and select the latest build for "Oculus" from the dropdown:
Make sure you have these before continuing:
For best results you should be using a high quality USB-C 3.1 cable (eg. like this one).
The Oculus app should be installed on your PC and Oculus-Link is setup and running.
From the Oculus app Oculus is set as the active OpenXR runtime - Set Oculus as active.
Next, in order to use the Quest's tracking features it's necessary to enable developer options from the Oculus app.
Developer Features
Enable the developer features from the Oculus app's settings for features such as passthrough, eye tracking and expressions to work with Mocap Fusion.
If you're using a Quest Pro be sure to also enabled and calibrate the eye and face tracking from in the HMD!
Create An Organizatino
Create a new organization Or go to: https://developer.oculus.com/manage/organizations/create
The Meta account should become a developer account by creating an organization. Restart your Oculus app and you will then be able to see the Developer Runtime Features in the Beta tab of the Oculus app.
Finger Tracking
Finger tracking is supported for most avatars that include more than one phalanges.
When the user sets down the controllers and holds their hands directly in front of the HMD then finger tracking is automatically enabled on the avatar.
By picking up the controllers again will disable hand tracking and switches back to UI laser pointers:
Accessing the VR Menu
It is also possible to access the VR menu when using finger tracking!!
To open the VR menu simply pinch for 3 seconds with your hand out in front of the HMD.
To recap we have:
Installed the Oculus branch of Mocap Fusion from Steam.
Installed the Oculus app.
Enabled developer runtime features.
Connected the Quest to the PC using a USB 3.1 cable.
Started oculus-link and launched Mocap Fusion.
Enabled finger tracking and recorded mocap.
Learned how to access the VR UI with finger tracking enabled.
Mixed Reality Passthrough
One of the coolest features in the Oculus branch is the mixed reality motion capture review. This allows reviewing motion capture in a mixed reality environment and makes avatars appear to be in the same room!
Mixed reality is a great way to see obstacles in the room when recording mocap.
To enable mixed reality click on the Passthrough toggle located at the bottom of the VR menu:
Install the Oculus Developer Hub if you want to record the Mixed Reality mocap directly from the headset:
Use the Animation Editor tools to live puppet avatars in mixed reality (with lighting and shadows):
If you are using only a HMD and controllers then the next steps are not required!
If you would like to include full body tracking using SteamVR Vive trackers then please continue reading.
Body trackers are supported in the Oculus branch but require a simple UDP server to stream data from SteamVR into Mocap Fusion.
Enter the IP to your PC or use: 127.0.0.1
Alternatively try the IP: 255.255.255.255
Body Tracking
APS OVR-Tracker Data Server
The OVR Tracker server uses the openvr api from SteamVR to stream Vive tracker coordinates data to Mocap Fusion.
You can download the Tracker Data Server from the download button. Be sure that SteamVR is running before starting the server!
SteamVR Lighthouse
Body Tracking
Debugging Window
When launching the server a window appears and prints status for each tracker as it streams data out of the UDP port.
Trackers are valid and report as "TRUE" when SteamVR is active and a VIve tracker has been connected.
Trackers report "FALSE" if there is no device connected or while SteamVR's menu is open. If all trackers show as reporting "FALSE" open the VR view and close the SteamVR menu if it is open.
Note that SteamVR must to be installed for the Vive trackers to function.
You may also wish to enable the SteamVR null-driver to avoid HMD errors.
Trouble Shooting
Note: If you notice SteamVR status reports "Standing By" and trackers are not responding (all False), try closing the APS Tracker Data Server and start the server again.
If that does not work then likely the Steam dashboard might be opened in VR and can't be closed since there is no HMD headset connected to steamVR and no controllers..
Try clicking on "Toggle Dashboard" and the trackers should resume tracking and will be valid/true.
Tracker Setup
When using Steam trackers when Quest HMDs it is necessary to calibrate the tracking space offsets of the two playspaces.
In APS you can simply assign the left and right feet trackers then calibrate T-Pose and the playspace alignment is then calibrated and the two playspaces should agree.
Optional Space Tracker
A tracker can be attached directly to the Quest HMD to help keep both playspaces aligned at all times but is not required. Read more: Space Tracker setup tutorial
Above video shows how to calibrate Trackers and Space offsets using just left and right feet tracker (center) as the alignment point.
Alternatively using the Built-In Space-Calibrator to align the playspace offsets when making a figure "∞" using the controller and tracker together.
Optional Feet Trackers
Simultaneous controllers and hand tracking (multi-modal mode) are now supported! This allows you to use Touch controllers as feet trackers for full body tracking while finger tracking is enabled.
Multi-Modal Feet Tracking Toggle
Disabling the "FEET TRACKING" toggle will disable multi-modal mode and controllers will only work when held. This disabled or enables simultaneous controllers. When multi-modal mode is enable after setting down the controllers you will see the controllers shown as trackers and listed in the Tracker setup menu.
T-Pose Calibration
Full Body Tracking + Quest Finger Tracking
Before calibrating tpose you should open a VR Mirror in front of the avatar and double check that the trackers are aligned with the avatar, also check they are moving correctly in VR!
Note: Calibrating the floor height before calibrating t-pose is often necessary!
Note: if the Space tracker has not been properly aligned on the HMD it might need slight adjusting. If the other trackers in the scene appear offset then try physically rotating the Space trackers on the HMD slightly and recalibrating.
Face Tracking
If you are using a Quest Pro you can include face tracking with when building custom avatars. Since APS_SDK (v7.8.1) - Meta Facecap Linker V2 update now includes support for all face tracking blendshapes for Quest Pro (v60 OpenXR update).
Thanks for reading ❤️