vtkPropPicker in VTK 9.1 is now returning wrong world coordinates.

Paulo_Carvalho · March 15, 2022, 9:16pm

Dear people,

Recently I migrated code from VTK 8.1 to 9.1. Giving or taking minor changes (mostly related to deprecated API), most of VTK code stayed the same. However, the behavior of vtkPropPicker::GetPickPosition() has changed. It now returns the wrong world pick coordinates. Here’s the code (it was fine in VTK 8.1):

// Get pick position in 2D screen coordinates.
int* clickPos = this->GetInteractor()->GetEventPosition();

// Get the 3D object under the 2D screen coordinates (ray casting).
vtkSmartPointer<vtkPropPicker>  picker = vtkSmartPointer<vtkPropPicker>::New();
picker->Pick(clickPos[0], clickPos[1], 0, this->GetDefaultRenderer());

// Get the picked location in world coordinates.
double* pos = picker->GetPickPosition();

The XYZ value in the pos array is used to place a small red sphere in the scene to serve as a pick maker. In VTK 8.1, the pick marker appeared exactly where the user picked. In VTK 9.1, the pick marker is placed on a plane orthogonal to the camera, which is incorrect. The figure below shows a 2D grid at Z=0 facing the camera:

Picking anywere results in the correct reported position Z=0 and, consequently, the pick marker is placed correctly on the grid.

Now, if I rotate the scene and proceed to picking, the method returns the wrong pick position in world coordinates, above or below the plane, even though the picked cell is correct:

After testing a few times, the conclusion is that the returned world coordinates lies in the intersection point of the pick ray and some plane orthogonal to the camera, not on the geometry of the picked actor. It seems to me that the picking logic is applying the reverse rotation transform applied to scene when computing the world picked position.

Thanks in advance,

Paulo

Paulo_Carvalho · April 2, 2022, 2:46pm

Issue opened in VTK upstream: https://gitlab.kitware.com/vtk/vtk/-/issues/18503

lassoan · April 6, 2022, 5:50pm

In general, textured surfaces are considered semi-transparent, because it would be too expensive to check if the displayed surface patch only contains fully opaque points. Therefore, by default these surfaces are non-pickable using hardware-based pickers that rely on the depth buffer. When picking fails you get the intersection point of the view ray and the camera image plane (a plane at the camera position, orthogonal to the camera view normal), probably you saw the coordinates of this intersection point.

If you want to pick a point on a textured surface then you may need to enable ForceOpaque on the actor.

Paulo_Carvalho · April 12, 2022, 9:45pm

Well… I guess I wasn’t clear. I didn’t say picking isn’t working. I said the returned world coordinates are wrong in VTK 9. Picking is working just fine. The only problem is the world coordinates as described in details above.

lassoan · April 12, 2022, 9:47pm

Has enabling ForceOpaque resolved the issue?

Paulo_Carvalho · April 12, 2022, 9:48pm

I need NaN-valued cells to be rendered transparent. Will ForceOpaque make NaN’s opaque?

lassoan · April 12, 2022, 9:50pm

Enabling ForceOpaque will make the entire mesh handled as opaque. I don’t think cell values affect Z-buffer-based picking in any way.

Paulo_Carvalho · April 12, 2022, 9:51pm

I also wonder why everything was just fine in VTK 8. Right now I’m considering a downgrade. I need picking working without issues in my application.

Paulo_Carvalho · April 12, 2022, 9:53pm

Unfortunatelly that’s not a solution. In my line of work it’s quite common to have unvalued cells. And these need to be rendered transparent. I’ll try it to give you a feedback, but I can’t deliver the application with unvalued cells visible.

lassoan · April 12, 2022, 10:02pm

Please confirm that ForceOpaque has solved the problem of the surface being excluded from picking. Dealing with semi-transparent or transparent cells is another issue.

I accepted that picking on semi-transparent surfaces is not feasible when fast Z-buffer-based picking is used, but I guess there are solutions/workarounds for it. Maybe some additional rendering pass, filtering out transparent parts of the mesh, use a different picker, etc.

Paulo_Carvalho · April 12, 2022, 10:12pm

Of course. I’ll give it a try and get back at you soon.

Paulo_Carvalho · April 12, 2022, 10:17pm

I think that is not what’s going on. When I pick outside the surface, I get a null pointer message in the application’s message panel. Again, the picking is functioning but it is not returning correct world coordinates.

lassoan · April 12, 2022, 10:37pm

OK, keep us updated. You can add a breakpoint in the Pick function to see what’s the difference between the case of no picking case and case of picking on the camera plane. Maybe there is just some quick bounding-box based check so that if you don’t click near your image then it does not even attempt to get a value from the z buffer.

Paulo_Carvalho · April 13, 2022, 1:51pm

The ForceOpaque had no effect on the issue. To make the issue clearer, I changed the code so:

The red pick marker is cloned for each pick.
Blue markers are added at the centers of each picked cell.

Here’s the visual result:

Representing the scene above while looking down the Y axis, the issue becomes very evident:

As you can see, picking is working, but the returned world coordinates are wrong. This used to work in VTK 8.

lassoan · April 13, 2022, 2:03pm

Please debug into this a bit more. Confirm that the depth along the viewing ray is retrieved from the Z buffer. If you confirmed this then retrieve the Z buffer and inspect values in it when you render various content: your surface with various textures (RGB, RGBA, ForceOpaque on/off), and when you render a basic opaque surface without any texture. This will narrow down the issue so much that experts will have a better chance of guessing what VTK change led to the changed behavior in your application.

You’ll probably need to provide a minimal example that reproduces the changed behavior if you want any help from VTK developers. If you can show that picking broke in ParaView, too, then you may get help from ParaView developers.

Paulo_Carvalho · April 13, 2022, 2:14pm

Same issue:

I don’t think any further debugging is necessary to confirm there is a bug. The figures above speak its all. The issue exists. Now, the next step is to fix it.

True, but I can’t afford the time right now. Sorry.

Possibly. ParaView has a lot of customized/extended code. For example, the tick marker option of VTK’s scalar bar doesn’t work. So, ParaView developers have one of their own making.

Paulo_Carvalho · April 13, 2022, 2:25pm

Maybe I can get the working picking example and add the pick markers to get that minimal example in less time.

Paulo_Carvalho · April 13, 2022, 2:32pm

I think this one will do: https://kitware.github.io/vtk-examples/site/Cxx/Picking/CellPicking/

It even reports the world coordinates.

lassoan · April 13, 2022, 3:13pm

This example can be a good starting point, but it uses vtkCellPicker (that does not use hardware picking, while you do), it displays a rectangle of only two cells (so it is harder to see any trends along many cells). So, some refinement would be needed. If you have no specific preference then please use the Python scripted version (I find it easier to debug issues with Python scripting).

That’s fine, it just makes it unlikely that the investigation will move forward. This is right now only your problem, as it seems that everyone else is fine with the current VTK behavior. I’m helping because I’m nice and also because in the long term I’m interested in fast semi-transparent surface picking solutions (right now in Slicer we use vtkCellPicker, which works well - even for semi-transparent surfaces and volume rendering - but it is slow for complex meshes).

dgobbi · April 13, 2022, 3:42pm

Though this is off topic, do you use a locator with vtkCellPicker when picking large meshes? With a locator the picking is O(log(N)), but without it is O(N).