The world coordinate is changed when the shown image is rotated.

I want to know how to calculate the image pixel coordinate fro world coordinate. The code of my experiment is:

import vtk
import numpy as np
from vtk.util.numpy_support import numpy_to_vtk
def numpyToVTK(data, multi_component=False, type='float'):
    if type == 'float':
        data_type = vtk.VTK_FLOAT
    elif type == 'char':
        data_type = vtk.VTK_UNSIGNED_CHAR
    else:
        raise RuntimeError('unknown type')
    if multi_component == False:
        if len(data.shape) == 2:
            data = data[:, :, np.newaxis]
        flat_data_array = data.transpose(2,1,0).flatten()
        vtk_data = numpy_to_vtk(num_array=flat_data_array, deep=True, array_type=data_type)
        shape = data.shape
    else:
        assert len(data.shape) == 3, 'only test for 2D RGB'
        flat_data_array = data.transpose(1, 0, 2)
        flat_data_array = np.reshape(flat_data_array, newshape=[-1, data.shape[2]])
        vtk_data = numpy_to_vtk(num_array=flat_data_array, deep=True, array_type=data_type)
        shape = [data.shape[0], data.shape[1], 1]
    img = vtk.vtkImageData()
    img.GetPointData().SetScalars(vtk_data)
    img.SetDimensions(shape[0], shape[1], shape[2])
    return img
global sphereActor, textActor
sphereActor = None
textActor = None
def mouseMoveEvent(iren, event):
    x, y = iren.GetEventPosition()

    picker = vtk.vtkWorldPointPicker()
    picker.Pick(x, y, 0, render)
    worldPoint = picker.GetPickPosition()

    sphere = vtk.vtkSphereSource()
    sphere.SetCenter(worldPoint[0], worldPoint[1], worldPoint[2])
    sphere.SetRadius(2)
    sphere.Update()
    sphereMapper = vtk.vtkPolyDataMapper()
    sphereMapper.SetInputData(sphere.GetOutput())
    global sphereActor, textActor
    if sphereActor != None:
        render.RemoveActor(sphereActor)
    sphereActor = vtk.vtkActor()
    sphereActor.SetMapper(sphereMapper)
    sphereActor.GetProperty().SetColor(255, 0, 0)
    render.AddActor(sphereActor)
    render.Render()

    if textActor != None:
        render.RemoveActor(textActor)
    textActor = vtk.vtkTextActor()
    textActor.SetInput('world coordinate: (%.2f, %.2f, %.2f)'%(worldPoint[0], worldPoint[1], worldPoint[2]))
    textActor.GetTextProperty().SetColor(1, 0, 0)
    textActor.GetTextProperty().SetFontSize(15)
    render.AddActor(textActor)


    print('屏幕坐标:', (x, y), '世界坐标: ', worldPoint)

img = np.zeros(shape=[128, 128])
for i in range(128):
    for j in range(128):
        img[i, j] = i+j

vtkImg = numpyToVTK(img)
imgActor = vtk.vtkImageActor()
imgActor.SetInputData(vtkImg)

render = vtk.vtkRenderer()
render.AddActor(imgActor)
# render.Render()

renWin = vtk.vtkRenderWindow()
renWin.AddRenderer(render)
renWin.Render()
iren = vtk.vtkRenderWindowInteractor()
iren.SetRenderWindow(renWin)
iren.SetInteractorStyle(vtk.vtkInteractorStyleTrackballCamera())
iren.Initialize()
iren.AddObserver('MouseMoveEvent', mouseMoveEvent)
iren.Start()

The used vtk is 9.0.1.

  1. My first question is: why the red sphere don’t moved along with my mouse? The red sphere will update only when the mouse is stop. Does the render() cost much time?

  2. My second quesion is: the display image is converted from numpy, and the origin should be (0, 0, 0), the direction matrix should be: x(1, 0, 0), y(0, 1, 0), z(0, 0, 1). I think the z value of the world point in the display image should be 0. When the image is not rotated, the world point is actually (x, y, 0)

    image
    In the above figure, the red text show the world point of the red sphere.

    However, after I rotate the image, the z value of the world point is not 0. For example:

    image

Any suggestion is appreciated!

  1. I guess you dont need to recreate the sphere at each call, this can explain why it’s slow
  2. no idea why, but indeed it is non zero… maybe a precision issue?

You can play around with this, moving the mouse drags the sphere in real time with no much delay:

import numpy as np
from vedo import *

img = np.zeros((128,128))
for i in range(128):
    for j in range(128):
        img[j,i] = i+j

def func(evt):
    if evt.picked3d is None:
        return
    nr = np.floor(np.array(evt.picked3d)).astype(int)
    txt.text(f"屏幕坐: {evt.picked2d}\n世界坐: {evt.picked3d}\n{nr[:2]}")
    sph.pos(evt.picked3d)
    plt.render()

plt = Plotter(axes=1)
pic = Picture(img)
sph = Sphere(r=2).pickable(False)
txt = Text2D(font="LogoType")
plt.addCallback("MouseMoveEvent", func)
plt.show(pic, sph, txt)

@marcomusy Thank you for your kindly reply.

What’s your vedo version? It report some bug for my current version (vedo 2020.4.2)

  1. Picture function requre a image with 3 component.
Traceback (most recent call last):
  File "C:/Users/MLoong/Desktop/test/test.py", line 18, in <module>
    pic = Picture(img)
  File "D:\Anaconda3\envs\general\lib\site-packages\vedo\picture.py", line 36, in __init__
    nchan = obj.shape[2] # get number of channels in inputimage (L/LA/RGB/RGBA)
IndexError: tuple index out of range

I modify the code as:

img = np.zeros((128,128,1))
for i in range(128):
    for j in range(128):
        img[j,i,0] = i+j
  1. bug for Text2D:
Traceback (most recent call last):
  File "C:/Users/MLoong/Desktop/test/test.py", line 20, in <module>
    txt = Text2D(font="LogoType")
TypeError: Text2D() missing 1 required positional argument: 'txt'

I modify the code as: txt = Text2D('txt', font="LogoType").

  1. bug for func as: TypeError: func() takes 1 positional argument but 2 were given. I change the definition from def func(evt) to def func(evt, p).

  2. bug for evt.picked3d as following:

Traceback (most recent call last):
  File "C:/Users/MLoong/Desktop/test/test.py", line 10, in func
    if evt.picked3d is None:
AttributeError: 'vtkmodules.vtkRenderingUI.vtkWin32RenderWindowInte' object has no attribute 'picked3d'

and I don’t know how to fix this bug.

Moreover, why the z value of a world point is not zero? In my thought, the conversion between pixel index (i, j, z) and world point (x, y, z) should be:

worldPoint (x,y,z) = i*spacingX*directionX + j*spacingY*directionY + k*spacingZ*directionZ + originPoint

Now, the direction should be:

directionX = [1, 0, 0]
directionY = [0, 1, 0]
directionZ = [0, 0, 1]

and originPoint=[0, 0, 0], spacingX=1, spacingY=1, spacingZ=1. Thus, the conversion should be:

x = i
y = j
z = k

Since the image in this code is converted from numpy, k should be 0.

I am confuzed that the world point is (20.43, 29.55, -0.18), why the z value is not 0.

image

And I don’t think it is a precision issue. It is a huge difference between 0 and -0.18.

On the other hand, when the image is not rotated, the z value is 0. While the image is rotated, the z value is not 0. But, I think the world point should be not related with the image rotation.

Just update with pip install vedo -U. If that doesn’t do the job please open an issue on github.

If it’s world coordinates you dont need to make conversions apart from a simple scaling
(check out the example).

I don’t know about the z not being zero maybe someone here has a better answer.

@marcomusy Do you know how to acquire the image intensity according to the world point?

I recommend using vtkImageInterapolator (also see base class vtkAbstractImageInterpolator) to get the image intensity at a point.

interp = vtk.vtkImageInterpolator()
interp.Initialize(img)
interp.SetInterpolationModeToLinear()
# some images are multi-component
value = [0.0] * interp.GetNumberOfComponents()
# the xs, ys, zs here are in continuous structured coordinates
interp.InterpolateIJK((xs, ys, zs), value)
print(value)

To convert from image data physical coords to structured coords, you can use
img.TransformPhysicalPointToContinuousIndex().

If you use vtkCellPicker for picking, then you can use picker.GetMapperPosition() to get the pick position in image physical coords. Otherwise, you have to do the world_coords to image_physical_coords transformation yourself (the transformation is identity unless you have repositioned the vtkImageActor).

Also note that the vtkCellPicker will be more precise than vtkWorldPointPicker, because the vtkWorldPointPicker is limited by the precision of the GPU depth buffer.

Thank you for your kindly reply, but I still have two questions:

  1. Why the picker point is outside of image bounds? In the above figure, I have picked a world point (20.43, 29.55, -0.18). But the image bounds is: (0, 127, 0, 127, 0, 0).

  2. In this code, the TransformPhysicalPointToContinuousIndex retuan the same value for voxel coordinate with world coordinate. The TransformPhysicalPointToContinuousIndex return (20.43,29.55,-0.18) for the voxel coordinate, but the size of image is: (128,128,1). Why the z-value is -0.18?

The z-value of 0.18 is probably due to the limited precision of the depth buffer, which is used by vtkWorldPointPicker. That is why I suggested trying vtkCellPicker to see if it gives a more precise result.

I’m sure you already know why TransformPhysicalPointToContinuousIndex() returns the same value. It is because the spacing is [1,1,1], the origin is [0,0,0] and the direction is the identity matrix. Under these circumstances, do you expect TransformPhysicalPointToContinuousIndex() to give a different answer?

maybe the vtkImageActorPointPlacer is also a relevant class?
https://vtk.org/doc/nightly/html/classvtkImageActorPointPlacer.html#details

I have tried the vtkCellPicker and GetMapperPosition:

    # picker = vtk.vtkWorldPointPicker()
    picker = vtk.vtkCellPicker()
    picker.Pick(x, y, 0, render)
    # worldPoint = picker.GetPickPosition()
    worldPoint = picker.GetMapperPosition()

And the result is:

image

The obtained world point is still wrong.

Moreover, even the z-value is 0.18, if I use TransformPhysicalPointToContinuousIndex to obtain the voxel coordinate (i, j, k), is the cooridiante i/j correct?

That result does not make sense to me. Can you show me the original (x,y) screen position, the vtkWorldPointPicker.GetPickPosition(), the vtkCellPicker.GetPickPosition(), and vtkCellPicker.GetMapperPosition() all together so that I can compare them?

As I stated above, for your situation, TransformPhysicalPointToContinuousIndex() will just be the identity transformation. It does not do any clamping or anything like that, it just transforms the point based on the spacing, origin, and direction.

@dgobbi I modify the mouseMoveEvent as:


    x, y = iren.GetEventPosition()

    worldPicker = vtk.vtkWorldPointPicker()
    worldPicker.Pick(x, y, 0, render)
    worldPickPoint = worldPicker.GetPickPosition()

    cellPicker = vtk.vtkCellPicker()
    cellPicker.Pick(x, y, 0, render)
    cellPickPoint = cellPicker.GetPickPosition()
    cellMapperPickPoint = cellPicker.GetMapperPosition()

    ...

    textActor = vtk.vtkTextActor()
    textActor.SetInput('screen pos: (%.1f, %.1f)\n'
                       'vtkWorldPointPicker position: (%.3f, %.3f, %.3f)\n'
                       'vtkCellPicker GetPickPosition: (%.3f, %.3f, %.3f)\n'
                       'vtkCellPicker GetMapperPosition: (%.3f, %.3f, %.3f)'%(x, y, worldPickPoint[0], worldPickPoint[1], worldPickPoint[2], cellPickPoint[0], cellPickPoint[1], cellPickPoint[2],
                                                                              cellMapperPickPoint[0], cellMapperPickPoint[1], cellMapperPickPoint[2]))

And the shown result is:

All picked point has non-zero z-value.

Good. For the CellPicker, the World and Mapper positions are exactly the same, which is what I expected for your data.

The pick is probably hitting the sphere instead of hitting the plane. Try setting PickableOff() on the sphere, and also on the text.

sphereActor.PickableOff()
textActor.PickableOff()

The WorldPointPicker position isn’t far different (we don’t expect the WorldPointPicker and the CellPicker to return exactly the same result, since they use different methods).

You should not expect the Z to be exactly zero because there might be roundoff error in the pick calculations. The tolerance of vtkCellPicker is plus or minus 0.025 by default, but you can call picker->SetTolerance(…) to change it. The tolerance of vtkWorldPointPicker depends on the rendered pixel size and the depth buffer precision.

2 Likes

After setting sphereActor.PickableOff(), the z-value of vtkCellPicker is about zero.

It shows that vtkCellPicker is more accurate than vtkWorldPicker.

@dgobbi Really thank you for your kindly reply.
@marcomusy Maybe vtkCellPicker is also accurate than vtkPropPicker.