The vtkFixedPointVolumeRayCastMapper only works with vtkImageData, so vtkUnstructedGrid is not useful for this situation.
One simple way to apply a mask is by multiplying with vtkImageMathematics. This might require changing the mask data type to the image data type, with vtkImageCast.
Or, another way to apply a mask is vtkImageStencil:
imageToStencil = vtk.vtkImageToImageStencil()
stencilFilter = vtk.vtkImageStencil()
# set "background" to a value that will be transparent for volume rendering,
# for example 0 for MRI, or -1000 for CT
I guess this question related to this topic created by you : How to crop 3D image from a 2D free-hand ROI? - #5 by Linfeng_Song . Have you manged to implement the free-form cut function ? I am implementing this function by just only using VTK code. Somehow it works with data with slicing (1, 1, 1), but for volume with different slicing the cut position is always not correct. Can you please help to share your algorithm of implementing free-form cut ? Thank in advance. BTW, is there any opportunity that I can know your contact and have a conversation about this topic.