So continuing this work I implemented a callback that keeps track of the press state of the VR controller buttons to overcome the double press event in the previous post.
def right_state_flipping_callback(obj, event):
if not hasattr(obj, 'rightbuttonstate'):
rightbuttonstate = True
obj.rightbuttonstate = rightbuttonstate
else:
rightbuttonstate = not obj.rightbuttonstate
obj.rightbuttonstate = rightbuttonstate
if rightbuttonstate:
obj.InvokeEvent(vtk.vtkCommand.RightButtonPressEvent)
else:
obj.InvokeEvent(vtk.vtkCommand.RightButtonReleaseEvent)
This callback then observes the commands corresponding to a rarely used event (RightButtonDoubleClick). Ideally I’d figure out how to set a new unique custom command just so there’s no clashes with existing functionality. This callback works fine to process the events appropriately and using observers I can see that the press events and release events are being successfully invoked.
Something that I must have missed though is the fact that widgets manage the cursor and take priority over the observers in the interactor. So this method works for everywhere except where I actually want the functionality… 
I need to do some digging to understand why the widgets aren’t properly processing the events, hopefully it’s just some specific callback that’s missing. There must be some interesting code to handle the pick and drag events using the VR rays, hopefully it’ll be easy to adapt.
I’m also working on implementing a polydata-based menu panel in VR, ideally the panel will attach to the controllers positions with greater control over what’s displayed. Paraview appears to use vtkQWidgetWidget to display an image plane, translate between the VR interaction and the 2D plane, pass the translated XY information to Qt and update the image texture however it looks like this method isn’t built in the available python wheels. I might be able to replicate the functionality by rendering the Qt window as an image texture manually, then figure out code to handle the interaction.
Using the native VTK widgets seems more simple to me, using a vtkAssembly I can define a set of 3D panel widgets (vtkButtonWidget, vtkSliderWidget, etc.) that move and rotate with their respective panel, this appears to work fine in a non-VR render but I still can’t get the widgets to trigger properly. I’m also not totally sure how to get the hand pose of the controllers, it looks like the method GetHandPose() isn’t exposed to the Python bindings.