What about if the view is set to perspective? Is there any functionality in VTK to assist in these calculations?

Currently, I’ve gotten myself down a rabbit hole where I think I may be close to the desired result:

```
...# Got the image from the rendering scene as `img` NumPy image array
import numpy as np
import vtki
import matplotlib.pyplot as plt
plt.imshow(img)
plt.colorbar()
plt.show()
```

Then I get a point cloud:

```
def get_depth_pc(plotter,):
ren_source = vtk.vtkRendererSource()
ren_source.SetInput(plotter.renderer)
ren_source.WholeWindowOff()
ren_source.DepthValuesOn()
ren_source.Update()
pc = vtk.vtkDepthImageToPointCloud()
pc.SetCamera(plotter.renderer.GetActiveCamera())
pc.SetInputConnection(ren_source.GetOutputPort())
pc.Update()
return vtki.wrap(pc.GetOutput())
point_cloud = get_depth_pc(p)
```

Then I transform the points to the camera’s reference frame

```
def get_numpy_mtx(mtx):
vals = np.empty((3,3))
for i in range(3):
for j in range(3):
vals[i,j] = mtx.GetElement(i, j)
return vals
matrix = p.camera.GetModelViewTransformMatrix()
mtx = get_numpy_mtx(matrix)
points = point_cloud.points.dot(mtx)
```

Then I attempt to produce the depth mapped image… but it doesn’t look right:

```
idx = np.argwhere(img.ravel() != 1.0).ravel()
depths = np.empty_like(img.ravel())
depths[:] = np.nan
depths[idx] = points[:,-1]
plt.imshow(depths.reshape(img.shape))
plt.colorbar()
plt.show()
```