So I’ve revisited using VTK in VR on the Quest, this time inspired by developments in the recent blog post and some support from @LucasGandel. I’ve produced a small example to test the VR capabilities. In general the functionality is really good, but the interaction style could do with some tweaking to properly conform with modern VR standards.
class bcolors:
HEADER = '\033[95m'
OKBLUE = '\033[94m'
OKCYAN = '\033[96m'
OKGREEN = '\033[92m'
WARNING = '\033[93m'
FAIL = '\033[91m'
ENDC = '\033[0m'
BOLD = '\033[1m'
UNDERLINE = '\033[4m'
import vtk
import sys
try:
from vtkmodules.vtkRenderingOpenXR import vtkOpenXRRenderer, vtkOpenXRRenderWindowInteractor, vtkOpenXRRenderWindow
except ImportError:
print(bcolors.FAIL + "vtkOpenXR modules not found. Please ensure you have the vtkOpenXR module installed." + bcolors.ENDC)
print(bcolors.WARNING + "These modules are currently only found on Windows" + bcolors.ENDC)
print(bcolors.WARNING + "and require VTK to be built with OpenXR support." + bcolors.ENDC)
print(bcolors.WARNING + "pip install --extra-index-url https://wheels.vtk.org vtk --pre --no-cache" + bcolors.ENDC)
sys.exit(1)
def create_VR_renderer(background_color=[0.03, 0.03, 0.15]):
# Create the renderer and render window
ren = vtkOpenXRRenderer()
renwin = vtkOpenXRRenderWindow()
iren = vtkOpenXRRenderWindowInteractor()
renwin.SetInteractor(iren)
renwin.AddRenderer(ren)
ren.SetBackground(*background_color) # Set background color to a nice blue
ren.SetShowFloor(True)
ren.SetShowLeftMarker(True)
ren.SetShowRightMarker(True)
renwin.SetPhysicalViewUp(0, 0, 1) # Set the physical view up direction
renwin.SetPhysicalViewDirection(0, 1, 0) # Set the physical view direction
renwin.RenderModels() # Enable model rendering
renwin.SetPhysicalScale(1.0) # Set the physical scale for the scene, seems to get overridden later
renwin.SetBaseStationVisibility(True)
iren.SetDesiredUpdateRate(72.0) # FPS of Quest 2
renwin.SetDesiredUpdateRate(72.0)
# Set the path to the directory containing openxr_controllermodels.json
renwin.SetModelsManifestDirectory('C:/VR/')
# // Set the path to the directory containing vtk_openxr_actions.json
iren.SetActionManifestDirectory('C:/VR/')
iren.SetRenderWindow(renwin)
return ren, renwin, iren
def main():
ren, renwin, iren = create_VR_renderer(background_color=[0.03, 0.03, 0.15])
menu = make_vrmenu_widget(ren, iren)
panel = make_VR_panel_widget(ren, iren)
ren.AddActor(create_cone_actor())
# Initialize the OpenXR renderer and interactor
iren.Initialize()
renwin.SetPhysicalScale(1.0) # Set the physical scale for the scene
renwin.Render()
iren.Start()
def make_vrmenu_widget(ren, iren):
vrmenu_widget = vtk.vtkVRMenuWidget()
rep = vtk.vtkVRMenuRepresentation()
rep.VisibilityOn()
rep.SetRenderer(ren)
rep.SetNeedToRender(True)
vrmenu_widget.SetRepresentation(rep)
vrmenu_widget.SetInteractor(iren)
vrmenu_widget.SetCurrentRenderer(ren)
vrmenu_widget.SetDefaultRenderer(ren)
vrmenu_widget.SetEnabled(True)
vrmenu_widget.On()
# print(vrmenu_widget, dir(vrmenu_widget))
# print("VR Menu Widget created with representation:", rep)
# print(sorted(dir(vrmenu_widget)), sorted(dir(rep)))
return vrmenu_widget
def make_VR_panel_widget(ren, iren):
VR_panel = vtk.vtkVRPanelWidget()
rep = vtk.vtkVRPanelRepresentation()
rep.VisibilityOn()
rep.SetRenderer(ren)
rep.SetNeedToRender(True)
rep.SetAllowAdjustment(True)
rep.SetText("VR Panel test")
rep.PlaceWidget([0, 1, 0.5, 1.5, -3, -2]) # Set the position of the panel in the VR space
VR_panel.SetRepresentation(rep)
VR_panel.SetInteractor(iren)
VR_panel.SetCurrentRenderer(ren)
VR_panel.SetDefaultRenderer(ren)
VR_panel.SetEnabled(True)
VR_panel.ProcessEventsOn()
VR_panel.SetManagesCursor(True) # Manage the cursor visibility in VR
VR_panel.SetDebug(True)
VR_panel.On()
# print("VR Panel Widget created with representation:", rep)
# print(sorted(dir(VR_panel)), sorted(dir(rep)))
return VR_panel
def create_cone_actor():
coneSource = vtk.vtkConeSource()
coneSource.SetHeight(3.0)
coneSource.SetRadius(1.0)
coneSource.SetResolution(600)
coneSource.Update()
mapper = vtk.vtkPolyDataMapper()
mapper.SetInputConnection(coneSource.GetOutputPort())
actor = vtk.vtkActor()
actor.SetMapper(mapper)
# actor.GetProperty().SetColor(colors.GetColor3d("Banana").GetData())
return actor
if __name__ == "__main__":
main()
Currently I’m working on understanding how the VR interaction works, the existing implementation is a little dated (seems like it’s designed for Vive wands & one-handed control, rather than knuckle-style controllers) and so it’d be good to get more intuitive interactions that make use of the grip buttons and joystick more.
My current usage case revolves around using both a vtkBoxWidget2 and vtkImagePlaneWidget to control a reconstruction in 3D space. The box widget works great in the ‘Grab mode’ of the VR interaction but the interaction with the ImagePlaneWidget is relatively poor (no cursor support, slicing, setting the scale etc.)
It’d be nice to have the grab action bound to only the grip button, whilst the (X/Y/A/B) buttons could emulate left/right mouse click events and have the click of the joystick count as a middle-click event. Left/right on the joystick currently does nothing, neither does the dedicated menu button on the left controller.
It looks like the movement style can be changed, there’s a class named GroundMovement3D, I just need to figure out how to actually use it.
It’s nice that you can enable the floor in the VR scene to help with VR comfort, but I’d suggest that the lighting should be disabled for that object, at oblique angles the floor goes completely black which reduces its usefulness.
For some reason I had to re-set the physical world scale after adding the actors, not a major issue though.
I still need to figure out how best to add in UI elements into the scene, VRMenu and VRPanel look promising. Interestingly the 2D objects in VTK like vtkBorderWidget are still displayed which might work as a HUD with some tweaking, though I’m not sure if they properly handle the left-right perspective properly. If I can figure out how to attach a VRPanel to the controller positions then I’ll try to make a UI that way.
In terms of hardware I’m using a Quest 2 with a PrismXR Puppis S1 router to reduce latency. The GPU is an RTX5000 but I can run with an A6000 if needed. Airlink works reasonably well but I’ve found Virtual Desktop to be more stable with their VDXR implementation. Virtual Desktop also allows for hand tracking which is neat, though not super useful without the joystick.
Just thought I’d document my initial findings in case it was useful to improving the functionality in the future.