So the VR functionalities of VTK have always been a bit under realised and so I thought I’d make a quick demo video of the 3D widgets VTK uses working in VR, I implemented an in-VR menu that attaches to the controllers and then mapped the raycast of a laser from the VR controllers onto the headset projection in order to make a convincing analogue for the 2D interaction.
I also figured out how the VR Avatar works and mapped the VR controllers onto that, the inverse kinematics is a little janky and I’m still trying to come up with a better way to extract the armature. Also the fact that the controllers are initialised as Left & Right as opposed to a generic tracker class means I can’t get full-body tracking or non-body external object tracking properly implemented yet.
Augmented reality is reasonably possible by setting a background colour and then using software like Virtual Desktop to treat that as an environmental passthrough.
I’m working on some software for quick CAD modelling and so needed a better widget for manipulating the polydata, it was also fairly easy to get that working but there are a few performance issues when the text followers are enabled I need to optimise.
Currently I’ve tested this on a Quest 2 and also a Pico Ultra 4 using a PC with Virtual Desktop, it’s not a perfect system as it’s reliant on the VR streaming software working reliably over WiFi, Oculus Link works okay but I could never get Pico Connect to actually register the VR handle.
I’m planning on buying a Steam Frame as soon as it’s available and so I’d be interested in understanding what technical challenges there are on getting VTK to run natively on that hardware. I’m also interested in whether the API will be modifying the tracker class to support additional tracking data, I added the relevant entries to the OpenXR actions manifest but there’s no method to actually bind that information onto additional trackers.
Regarding technical challenges getting VTK to run on Steam Frame: Maybe I’m missing something obvious, but since Steam Frame seems like it will have support for OpenXR, I hope it would just work. Did you ever try running your application with the best practices validation layer enabled?
Maybe that would give us advanced notice if there are things we do in VTK that might cause performance problems on Steam Frame.
Regarding modifying the API to support additional tracking data: I’m not sure I follow what you mean. Is your question related to future use of Steam Frame, or do you mean right now?
So the Steam Frame will have support to run certain applications directly on the headset via the Proton compatibility layer which might allow for a version of VTK to run natively on the headset without needing a PC to stream from. It should work without issue using my current setup, but a standalone version would open up a lot more flexibility for real-world usage in the field.
I can have a look at implementing the validation layer, Virtual Desktop actually uses its own runtime (VDXR) which might make things a little more complex.
Regarding the additional trackers I’m currently using a Pico 4 Ultra which can support up to 5 additional trackers, they are mainly for body tracking but they can also be placed on physical objects for bespoke use cases. AFAIK there isn’t a way to expose the pose of additional trackers in VTK even when they’re added to the action manifest.
Also for the 3DSlicer people, a quick test of using the implicit plane as a volume clipping plane. Nothing too fancy but I figured it’d be kinda cool. Presumably I could add the image tracer widget onto the clipped plane to do some fun stuff.
The video is a little compressed given the 10 MB upload limit, but still looks pretty cool.
Regarding trackers, I agree VTK only updates left/right subaction paths. To support generic trackers, you could extend CreateSubactionPaths (here) to map vtkOpenXRManager::ControllerIndex::Generic to your json action path. Then updating for loops to add the Generic type when polling actions (here) should get you close to getting the tracker pose.
Regarding running VTK with openXR without streaming into the headset, this mainly requires the headset to support OpenGL (XR_KHR_opengl_enable extension). My experience is that Quests support GLES for instance which is why VTK can’t run directly into it but requires streaming with AirLink instead. I never checked for Steam Frame.
Supporting additional rendering backend (GLES, DirectX) in VTK-OpenXR can also be considered, but that requires some effort.
Thanks for sharing!