After applying the mask, the 3D image appears jagged or aliased at the edges. Any suggestions on how to fix this

This is actually nice idea, and I will certainly give it a try. However, it may not entirely align with the existed logic, as I am using the mask as a binary in vtkOpenGLGPUVolumeRayCastMapper. Thanks