Is AR possible with an HTC Vive Pro?

VR/AR
1

Hello,

While looking for a high level framework to create some simple virtual reality applications, I stumbled upon the OpenVR module of VTK.

It looks very promising but since I have an HTC Vive Pro with two front cameras, I was wondering if it was possible to overlay virtual objects on top of the video feed to create an augmented reality app. From what I can read here, it seems that there is a single camera rendering the scene from the two points of view so I don’t know if it is feasible to set a different background image for each eye. If it’s possible, can you give me some pointers on how can I achieve this?

Thanks

P.S: I never used VTK before

1 Like
2

No one?

3

@ken-martin may know. Afaik it is not yet implemented in our support.

4

There is a MR to add this functionality (or similar) that was never merged in VTK. See
https://gitlab.kitware.com/vtk/vtk/merge_requests?scope=all&utf8=✓&state=opened&search=openvr

But I do not believe anyone is working on it right now.

5

It indeed seems to be what I’m looking for.

Do you know if it requires a great amount of work before being merged?

I’ll try to build the branch and see if it’s usable

6

No clue, but let us know how it goes.

7

I’ve talked with @LucasGandel (not the dev of this MR, he was just close to the project). He said that this work was definitelly not finished but the core of the work, recovering the image of the camera without copying the data, was already in.

8

Ok that’s somewhat good news.

The build failed because of a missing parameter in a function call so I’ll see if I can fix this easily.

Edit: The issue comes from an update in OpenVR API that requires the index of the device being used, probably to handle multiple devices at the same time. Without not thinking to much about it, I passed a 0 and the build succeeded.

9

Adam Rankin at Robarts has a working version. Last time we spoke there were still some issues like not perfect stereo vision (potentially due to difference of camera distance and interpupillary distance), but it may have been fixed since then.

10

Yes, we have it working. We use an external stereo camera as the built-in Vive Pro cameras are very low resolution.

We utilize vtkRenderer::SetLeftBackgroundTexture and vtkRenderer::SetRightBackgroundTexture.

The imperfect stereo is still present, as I have not gone back to work on it in a while.

I can’t remember if this extension works fully, or if I just utilized the python interactor in Slicer to set up the VTK pipeline.

Workflow is
Plus library stereo camera capture and broadcast -> 3D Slicer receive stereo feed and pass it to OpenVR headset via SlicerVirtualReality extension

11

I’m back on this problem and I can successfully set the background images using vtkRenderer::SetLeftBackgroundTexture and vtkRenderer::SetRightBackgroundTexture but the aspect ratio of the images is not correct and they appear to be zoomed in.

I’m getting the images from a Zed M camera since, indeed, the internal ones are pretty bad, and I’m using this to convert an OpenCV image to a VTK one. Is this the right thing to do? @adamrankin Is this how you’re doing this?