For our OpenVR + VTK based program, we noticed a strange issue in VTK 9.2.(2/5): Only the left eye actually gets the rendered image, the right eye only sees the empty surroundings. We have tested the exact same code on our end with VTK 9.1.0 and it works there.
Any pointers where we should start looking here, is there something that has changed in how to initialize the VR mode maybe?
I tried looking through the code changes in the OpenVR folder between 9.1.0 and 9.2.x, they seem to mainly concern the new common code base for OpenXR and OpenVR, right?
Hi @codeling
Indeed, many changes occured on the OpenVR module with the generalisation of many features.
The new code is tested, both by test in VTK itself and by the ParaView XRInterface plugin (and many other application using it).
So first I’d suggest trying the test on your setup to make sure they work,
then you could try to bisect the issue on your side, or take a look into the tests to understand how they differ from your app.
Maybe @Thomas_Galland or @Tiffany_Chhim have more insight
The tests work (meaning they run through successfully - are they supposed to show something as well? they just stop after a few seconds). The examples also work with VTK 9.2 (after fixing render window initialization) in them.
So I guess it’s an issue with specifically the way we’re using OpenVR - I could imagine it’s related to us having a separate thread run the VR stuff. I’ll try to come up with a minimal reproducible example extracted from our code…
Read the doc of the tests, some of them are intended to be used interactively:
https://gitlab.kitware.com/vtk/vtk/-/blob/master/Rendering/OpenVR/Testing/Cxx/CMakeLists.txt
Thanks for the info! I was using the release 9.2.5 to test, but realized that only master has those interactive settings you mentioned. The tests get reported as failed on my setup (HTC VIVE Pro on an NVidia Quadro P5000) due to multiple errors like this:
vtkOpenGLFramebufferObj:368 ERR| vtkOpenGLFramebufferObject (000002BEE65E77F0): failed at glDeleteFramebuffers 16 OpenGL errors detected
But they seem to run completely fine despite these errors being reported.
As for our original issue, as I said, I do suspect our special setup of running VR in a separate thread (i.e. not in the main application thread, because that is driving our “other” GUI) is probably at fault here, I will try and come up with a minimal reproducible example.
We are mainly testing on a Valve Index with classic RTX nvidia card here.
After many additional experiments, I think I have narrowed down our problem somewhat; it seems to be related to volume rendering,
I can at least provide a minimal example which works with VTK 9.1 but doesn’t properly with VTK 9.2:
OpenVRVol.cxx:
#include <vtkActor.h>
#include <vtkColorSeries.h>
#include <vtkColorTransferFunction.h>
#include <vtkConeSource.h>
#include <vtkImageData.h>
#include <vtkNamedColors.h>
#include <vtkNew.h>
#include <vtkOpenVRRenderWindow.h>
#include <vtkOpenVRRenderWindowInteractor.h>
#include <vtkOpenVRRenderer.h>
#include <vtkPiecewiseFunction.h>
#include <vtkPolyData.h>
#include <vtkPolyDataMapper.h>
#include <vtkProperty.h>
#include <vtkSmartVolumeMapper.h>
#include <vtkVolumeProperty.h>
int main(int, char* [])
{
int dim[3] = { 10, 10, 10 };
double spc[3] = { 0.05, 0.05, 0.05 };
vtkNew<vtkImageData> img;
img->SetDimensions(dim);
img->AllocateScalars(VTK_INT, 1);
img->SetSpacing(spc);
for (int x=0; x<dim[0]; ++x)
for (int y=0; y<dim[1]; ++y)
for (int z=0; z<dim[2]; ++z)
{
img->SetScalarComponentFromDouble(x, y, z, 0, x);
}
vtkNew<vtkColorSeries> colors;
colors->SetColorScheme(vtkColorSeries::BREWER_QUALITATIVE_SET3);
vtkNew<vtkColorTransferFunction> ctf;
for (int x = 0; x < dim[0]; ++x)
{
auto c = colors->GetColor(x);
ctf->AddRGBPoint(x, c.GetRed() / 255.0, c.GetGreen() / 255.0, c.GetBlue() / 255.0);
}
ctf->AddRGBPoint(dim[0], 1.0, 1.0, 1.0);
ctf->Build();
vtkNew<vtkSmartVolumeMapper> volMapper;
vtkNew<vtkVolume> volume;
vtkNew<vtkVolumeProperty> volProp;
volMapper->SetBlendModeToComposite();
volume->SetMapper(volMapper);
volume->SetProperty(volProp);
volume->SetVisibility(true);
volMapper->SetInputData(img);
volProp->SetColor(0, ctf);
vtkNew<vtkPiecewiseFunction> otf;
otf->AddPoint(0.0, 1.0);
otf->AddPoint(dim[0], 1.0);
volProp->SetScalarOpacity(0, otf);
volProp->Modified();
vtkNew<vtkConeSource> coneSource;
coneSource->Update();
vtkNew<vtkPolyDataMapper> mapper;
mapper->SetInputConnection(coneSource->GetOutputPort());
vtkNew<vtkActor> actor;
actor->SetMapper(mapper);
vtkNew<vtkOpenVRRenderWindow> renderWindow;
renderWindow->Initialize();
vtkNew<vtkOpenVRRenderWindowInteractor> renderWindowInteractor;
renderWindowInteractor->SetRenderWindow(renderWindow);
vtkNew<vtkOpenVRRenderer> renderer;
renderWindow->AddRenderer(renderer);
vtkNew<vtkNamedColors> namedColors;
renderer->SetBackground(namedColors->GetColor3d("ForestGreen").GetData());
renderer->AddActor(actor);
renderer->AddVolume(volume);
volume->SetPosition(0, 1, 0);
renderWindow->Render();
renderWindowInteractor->Start();
return EXIT_SUCCESS;
}
CMakeLists.txt:
cmake_minimum_required(VERSION 3.12 FATAL_ERROR)
project(OpenVRVol)
find_package(VTK COMPONENTS
CommonCore
CommonColor
CommonDataModel
FiltersSources
InteractionStyle
RenderingContextOpenGL2
RenderingCore
RenderingFreeType
RenderingGL2PSOpenGL2
RenderingOpenGL2
RenderingOpenVR
RenderingVolumeOpenGL2
)
if (NOT VTK_FOUND)
message(FATAL_ERROR "OpenVRVol: Unable to find the VTK build folder.")
endif()
# Prevent a "command line is too long" failure in Windows.
set(CMAKE_NINJA_FORCE_RESPONSE_FILE "ON" CACHE BOOL "Force Ninja to use response files.")
add_executable(OpenVRVol MACOSX_BUNDLE OpenVRVol.cxx )
target_link_libraries(OpenVRVol PRIVATE ${VTK_LIBRARIES}
)
# vtk_module_autoinit is needed
vtk_module_autoinit(
TARGETS OpenVRVol
MODULES ${VTK_LIBRARIES}
)
With VTK 9.1, the demo volume is nicely rendered above the cone:
With VTK 9.2, I only see bits and pieces of the volume if I move around; it looks as though I’m “slicing”(maybe with the near plane) through the volume, but I never see the full one, not even if I try to move further away or if I look at it from below or the other direction. A screenshot cannot really catch the full “experience”, but you get an idea:
Is my code doing something wrong here?
Was there a change in how volume rendering works with OpenVR?
Did you try outside of VR ?
Yes it works in both VTK 9.1 and 9.2, and looks (on a quick glance) the same:
Well, someone needs to go digging to figure out what is going on.
Maybe @LucasGandel has an idea.
Looks like this fix is not part of the release.
@codeling Could you please cherry-pick the above commit to see if it fixes the issue?
2 Likes
Yes this seems to solve the volume rendering issue. Thanks!
Not sure if it solved my original issue (left eye rendering only) - I’ll check back with our main program…
EDIT: this specific fix unfortunately not. I’ll have to keep looking for a minimal reproducible example (will try with git master next though)
Thanks a lot for the feedback. Do I get it correctly that both eyes are now rendered correctly in the small example you shared above, but not in your main application?
In your main program, do you use any of the following methods?
- vtkVRRenderWindow:: SetPhysicalViewDirection/ SetPhysicalViewUp/…
- vtkProp3D::SetUserTransform
Correct.
In your main program, do you use any of the following methods?
Not as far as I (/grep) can see at the moment.
These would be known for causing such issues?
Thanks a lot for checking.
I experienced a similar issue where only the left eye was rendered, depending on where the prop was placed in the scene, and I thought this was caused by the physical scale of the render window but this might just be a side-effect of the bug.
IIRC the problem was that the props were always culled because the clipping range of the vtkCamera was wrong for the right eye. Can you try adding the following to your main application to see if it fixes the issue?
renderer->RemoveCuller(renderer->GetCullers()->GetLastItem());
2 Likes
Yes, this fixes the issue with the right eye!
It doesn’t seem to be the only culling problem though:
For volume rendering, we still see some issue - experimenting with a volume (grayscale fiber rendering), it starts to show as some kind of “white fog”; by interactions I can make the volume smaller so it becomes visible as a whole; but when the volume is close enough to be cut by the the near plane, the cut doesn’t seem to happen in direction of the viewer:
I guess this is a similar or related culling issue?
EDIT: how it looks when starting is depending on the spacing of the volume (only the rendering, not the size). With sufficiently small spacings (e.g. 0.05), I don’t see the “white fog” (which appeared with spacing > 0.5) but the volume gets more or less rendered (though already then, being cut int the “wrong direction”):
(we see the front of the volume - you see the bounding box in the upper left, even that is “cut” towards the back it seems).
That the bounding box is cut is something happening only in the “initial” view - if I move the volume around, the box starts to stay visible as a whole, only the volume gets wrongly culled.
EDIT2: This volume rendering behavior can actually be reproduced by my example code above - look how you can “see through” the volume when you are actually supposed to be “in it”:
Thanks for the feedback, this is great news that the issue with the right eye is solved.
From your report, I identify the following issues:
-
Small spacing required to remove white fog
The problem here is probably the default value of the vtkVolumeProperty::ScalarOpacityUnitDistance parameter being too big. Can you try callingvolumeProperty->SetScalarOpacityUnitDistance(0.001);? -
The bounding box is cut
This could definitely be related to 1. or to the wrong camera clipping range. Does callingrenderer->ResetCameraClippingRange()help? -
Volume is wrongly cut by the camera near plane
I confirm this is an issue. I have experienced this in the past but unfortunately I don’t have a fix for it yet.
The problem is again related to the camera clipping range. The GPU volume mapper should add a clipping plane just in front of the camera near plane, in order to “close” the hole that appears in your first screenshot (see here).
I suspect that the plane is wrongly placed because the physical scale or something is not taken into account.
1 Like
Thanks for the lightning fast reply 
this is great news that the issue with the right eye is solved.
Do you have an idea how a more permanent fix of this might look like?
Regarding the volume rendering points:
- SetScalarOpacityUnitDistance doesn’t help, it was caused by similar reasons than 2:
- Yes it helped! We actually had that already in the code, but I had experimented with moving VR initialization (including ResetCamera, ResetCameraClipping and event loop start) to before adding the rendering stuff. When calling these after adding the renderers, it works again.
- Looking forward to a fix, don’t think I can help here unfortunately.
TBH I have no idea, the code was there in tests since the 1st commit that introduced OpenVR support in VTK (see here).
Maybe using vtkCameraActor/vtkFrustumSource could help to confirm that the camera frustum planes are computed correctly for the VR camera.
1 Like
What I’m wondering here is then why this was working for us without the workaround of removing the last culler for VTK < 9.2…
Also the volume cutting problem is new - with VTK 9.1 it worked as expected.
Maybe using
vtkCameraActor/vtkFrustumSourcecould help to confirm that the camera frustum planes are computed correctly for the VR camera.
If I have time I might look into it, thanks for the pointers!