Detecting visible mesh faces in a scene rendered outside of VTK

tsnoble · July 21, 2021, 12:51pm

Hi,

I’m currently attempting to select all faces of a mesh visible to a camera in a scene rendered through Qt & Ogre. Ogre doesn’t seem to provide much in the way of this, and I’m already loading the mesh into a vtkPolyData object from an STL file, so I figured I could build a vtkRenderer using the attributes from Ogre, then use a vtkHardwareSelector to get the visible faces there.

I’ve thrown together some code which I believe should work, however I am getting a color buffer depth error when calling the vtkHardwareSelector::Select() method.

The code in question is shown below. mesh_ is an attribute containing the vtkPolyData object, which is loading fine:

std::vector<int> Mesh::getVisibleFacesInRange(Ogre::Viewport* viewport, double radius, double click_x, double click_y)
{
    std::vector<int> result;
    //TODO: Pull out into Ogre Viewport -> VTK Renderer helper
    ROS_INFO("Building VTK Renderer");
    Ogre::Camera* camera = viewport->getCamera();
    Ogre::Matrix4 ogre_world;
    camera->getWorldTransforms(&ogre_world);
    double world[16];
    for (int i = 0; i < 4; i++)
    {
        for (int j = 0; j < 4; j++)
        {
            world[(i * 4) + j] = ogre_world[i][j];
        }
    }
    vtkRenderer* renderer = vtkRenderer::New();
    vtkCamera* vtk_camera = renderer->MakeCamera();
    vtk_camera->SetModelTransformMatrix(world);
    Ogre::Vector3 position = camera->getPosition();
    double pos[3] = {position.x, position.y, position.z};
    vtk_camera->SetPosition(pos);
    Ogre::Vector3 up_vec = camera->getUp();
    double up[3] = {up_vec.x, up_vec.y, up_vec.z};
    vtk_camera->SetViewUp(up);
    Ogre::Vector3 focal_point = position + camera->getDirection();
    double focal[3] = {focal_point.x, focal_point.y, focal_point.z};
    vtk_camera->SetFocalPoint(focal);
    vtk_camera->SetViewAngle(camera->getFOVy().valueDegrees());
    vtk_camera->SetFocalDistance(camera->getFocalLength());
    renderer->SetActiveCamera(vtk_camera);
    vtkRenderWindow* window = vtkRenderWindow::New();
    window->AddRenderer(renderer);
    window->SetSize(viewport->getWidth(), viewport->getHeight());
    renderer->Clear();
    vtkPolyDataMapper* mapper = vtkPolyDataMapper::New();
    mapper->SetInputData(mesh_);
    mapper->Update();
    vtkActor* actor = vtkActor::New();
    actor->SetMapper(mapper);
    renderer->AddActor(actor);
    window->Render();
    ROS_INFO("Built VTK Renderer");
    vtkOpenGLHardwareSelector* selector = vtkOpenGLHardwareSelector::New();
    selector->SetFieldAssociation(vtkDataObject::FieldAssociations::FIELD_ASSOCIATION_CELLS);
    selector->SetRenderer(renderer);
    selector->SetArea(click_x - radius, click_y - radius, click_x + radius, click_y + radius);
    vtkSelection* selection = vtkSelection::New();
    ROS_INFO("Selecting visible cells");
    selection = selector->Select();
    ...
}

Along with the logs:

[ INFO] [1626871024.589000972]: Built VTK Renderer
[ INFO] [1626871024.589327747]: Selecting visible cells
2021-07-21 13:37:04.589 (   8.983s) [        7662AA80]vtkHardwareSelector.cxx:279    ERR| vtkOpenGLHardwareSelector (0x55fdd906b1c0): Color buffer depth must be at least 8 bit. Currently: 1, 0, 2109690372

Apologies for the messy code and unfamiliarity with VTK. I’d much appreciate any help, and if there is a cleaner way of achieving this I’d love to hear it

tsnoble · July 21, 2021, 12:57pm

I’m wondering if the error is happening precisely because the scene is being rendered outside of VTK? Based on the name, I’d guess that the selector is getting face visibility info from the GPU. Since the scene is actually being rendered by Ogre, I’m wondering if the data in the GPU looks different to how VTK is expecting?

danlipsa · July 21, 2021, 1:10pm

Yes, hardware selector works by using an additional render using cell ids instead of colors and then reading the cell id at the location provided. If rendering is done in ogre the buffer with cell ids is not present anymore.
You should be able to fix this by using vtkLocationSelector instead (or keep using the hardware selector and do a render in VTK as well).

tsnoble · July 21, 2021, 1:17pm

Thanks for the quick response! Would vtkLocationSelector provide the same “visible faces only” functionality?

tsnoble · July 21, 2021, 1:23pm

Is this line not sufficient for doing the rendering in VTK?

window->Render();

danlipsa · July 21, 2021, 1:41pm

You are right - this seems to work with 3d points (not 2d). Maybe use vtkLinearSelector instead as this gives you the selection points and you can order them by depth.

tsnoble · July 21, 2021, 1:49pm

I suppose vtkLinearSelector only returns faces colliding with a single line though, as opposed to faces within a given region of the window? I wonder if I could combine vtkLocationSelector with some visible face checking logic e.g.

Find collision point of click with mesh
Find all faces a given distance around this point using vtkLocationSelector
Cull all faces where the line segment between the camera position and the center of the face collides with the mesh i.e. there’s another face occluding it

danlipsa · July 21, 2021, 2:05pm

Fixing the hardware selector seems a lot easier - the trick there is syncing the rendering parameters between ogre and vtk.

tsnoble · July 21, 2021, 2:15pm

Thanks for the guidance. I think that’d probably be simpler too! Struggling to know what parameters I’m missing though. believe I’m setting up the camera position & orientation correctly, as well as placing the mesh in the VTK scene through the renderer. Is it colour buffer options I should be looking for? What values would be suitable for VTK?

danlipsa · July 21, 2021, 2:23pm

I would make sure I get the same image when rendering in VTK and ogre.
Here is an example for the hardware selector
https://kitware.github.io/vtk-examples/site/Python/Visualization/HardwareSelector/

You will find additional usage examples in the tests for the hardware selector - grep in the VTK sources.

tsnoble · July 22, 2021, 8:01am

Didn’t have much luck figuring out the issue last night. Think my main point of confusion comes from this line of code

window->Render();

To me this suggests that VTK should be re-rendering the scene on top of whatever Ogre has rendered in the past, so I’m unsure as to why Ogre would still be affecting the state of the colour buffer after this point in time…

Will try running an isolated VTK hardware selection example in C++ just to verify that it is Ogre causing the issue, and not a purely VTK related one.

tsnoble · July 22, 2021, 8:28am

Verified that the isolated C++ example works, though I did have to add the following lines to the example code to get the window to render:

#include <vtkAutoInit.h>
VTK_MODULE_INIT(vtkRenderingOpenGL2)

tsnoble · July 22, 2021, 8:33am

Progress! I modified my example to a simpler (still failing) one which ignores the mesh geometry for now

std::vector<int> Mesh::getVisibleFacesInRange()
{
    vtkNew<vtkRenderer> renderer;
    vtkNew<vtkRenderWindow> window;
    window->SetMultiSamples(0); // Turn off anti-aliasing
    window->AddRenderer(renderer);
    window->Render();
    vtkNew<vtkOpenGLHardwareSelector> selector;
    selector->SetRenderer(renderer);
    int* temp = window->GetSize();
    unsigned int windowSize[4];
    windowSize[0] = temp[2];
    windowSize[1] = temp[3];
    windowSize[2] = temp[0];
    windowSize[3] = temp[1];
    selector->SetArea(windowSize);
    selector->SetFieldAssociation(vtkDataObject::FIELD_ASSOCIATION_CELLS);
    vtkSelection* selection = selector->Select();
    ROS_INFO("Selection has %d nodes", selection->GetNumberOfNodes());
}

Adding the autoinit lines fixed the colour buffer issue. Will begin adding the geometry back in

tsnoble · July 22, 2021, 11:00am

@danlipsa I’ve got things working better now. The scene renders in a VTK window for an instant after the user clicks and the perspective is identical to the Ogre scene. There’s still a couple of issues though.

Do you know if it’s possible to hide the VTK window?
The number of visible cells returned by the hardware selector is quite unpredictable. It seems to start off high, then settle to a more sensible number after a few clicks, even when the camera stays still. I’m using the entire window for the selection rather than an area around the click, so I don’t think it’d be due to changes in mouse position
After VTK renders the scene for the first time, Ogre stays frozen at the last frame rendered, even when the VTK window closes. I can see that the camera is still being translated / rotated in response to mouse events as the attributes continue to change (and if I click, the VTK window that pops up accounts for the updated camera) so it seems to be purely a rendering thing rather than the application crashing. I’m wondering if VTK assumes “control” of rendering (of the GPU?) the first time and just needs to indicate that it’s done once the hardware selector is executed, so that Ogre can assume control again?

danlipsa · July 22, 2021, 1:02pm

@danlipsa I’ve got things working better now. The scene renders in a VTK window for an instant after the user clicks and the perspective is identical to the Ogre scene. There’s still a couple of issues though.

Do you know if it’s possible to hide the VTK window?

SetOffScreenRendering(true)

The number of visible cells returned by the hardware selector is quite unpredictable. It seems to start off high, then settle to a more sensible number after a few clicks, even when the camera stays still. I’m using the entire window for the selection rather than an area around the click, so I don’t think it’d be due to changes in mouse position

Sounds strange. Do you have the same behavior for using an area around the click?

After VTK renders the scene for the first time, Ogre stays frozen at the last frame rendered, even when the VTK window closes. I can see that the camera is still being translated / rotated in response to mouse events as the attributes continue to change (and if I click, the VTK window that pops up accounts for the updated camera) so it seems to be purely a rendering thing rather than the application crashing. I’m wondering if VTK assumes “control” of rendering (of the GPU?) the first time and just needs to indicate that it’s done once the hardware selector is executed, so that Ogre can assume control again?

My guess is that settings you used to render in VTK change the opengl state such that ogre does not render correctly anymore. @Sankhesh Jhaveri Do you have any other suggestions?

tsnoble · July 22, 2021, 1:52pm

Just tried 1 and it works like a charm, thanks!

#include <vtkAutoInit.h>
VTK_MODULE_INIT(vtkRenderingOpenGL2)

I’m wondering if it’s these lines of code causing the issue, as Ogre (or rviz, which is the application using Ogre) loads with OpenGL 4.6, but we switch to OpenGL 2 for VTK.

I’ll do some testing with the returned number of cells using a smaller selection area.

I’m also a little confused around the output. Should I be using vtkSelection::GetNumberOfNodes() or vtkSelection::GetNode(0)::GetSelectionList::GetNumberOfTuples() to get the number of visible cells? I’ve seen the latter being used in examples, but the former also returns more than one usually

danlipsa · July 22, 2021, 2:30pm

Just tried 1 and it works like a charm, thanks!
#include <vtkAutoInit.h>

VTK_MODULE_INIT(vtkRenderingOpenGL2)
I’m wondering if it’s these lines of code causing the issue, as Ogre (or rviz, which is the application using Ogre) loads with OpenGL 4.6, but we switch to OpenGL 2 for VTK.

Both use the same opengl version. OpenGL2 is meant to mean the new VTK rendering implementation as opposed to the old one that used fixed pipeline.

I’ll do some testing with the returned number of cells using a smaller selection area.

I’m also a little confused around the output. Should I be using vtkSelection::GetNumberOfNodes() or vtkSelection::GetNode(0)::GetSelectionList::GetNumberOfTuples() to get the number of visible cells? I’ve seen the latter being used in examples, but the former also returns more than one usually

You use all nodes and all selections in each node. I think you can have the same selection in different nodes - maybe this is where your many values come from.

tsnoble · July 22, 2021, 3:04pm

Ah okay, thanks for the clarification. I’m a little stumped on why the Ogre rendering isn’t working then…

I ran some tests using the sum of tuples in all selection node lists, as opposed to the number of nodes / number of tuples in the list of the first node, and the result came out much more stable!

I’ve noticed that it still seems to pick up on some visible faces when I zoom in so much that they disappear. I think maybe I just need to pass the near clip distance from Ogre to VTK for that though

tsnoble · July 22, 2021, 3:16pm

After playing around in rviz some more after the render window freezes, I managed to get it to crash with the following error:

[ WARN] [1626966862.740065642]: OGRE EXCEPTION(3:RenderingAPIException): Zero sized texture surface on texture SelectionTexture0 face 0 mipmap 0. Probably, the GL driver refused to create the texture. in GLTexture::_createSurfaceList at /build/ogre-1.9-kiU5_5/ogre-1.9-1.9.0+dfsg1/RenderSystems/GL/src/OgreGLTexture.cpp (line 415)

Not sure if it helps much besides verifying that Ogre is having trouble using the OpenGL driver after VTK does…

tsnoble · July 22, 2021, 3:22pm

I’ve just tried using vtkRenderWindow::ReleaseGraphicsResources() and vtkRenderWindow::Finalize() in the hopes that they might clear any OpenGL state after rendering, but there was no visible difference.