After applying the mask, the 3D image appears jagged or aliased at the edges. Any suggestions on how to fix this

I have a 3D image of a vertebra before applying the mask
image
And after
image
As you can see, the edges of the image become heavily jagged. I am considering ways to fix this. Perhaps some form of post-processing on the GPU or applying some processing to the masked data before its presentation could work. Post-processing techniques like MSAA or Gaussian blur did not yield the desired results, as they operate on pixels, while the jaggedness here is at the voxel level, but perhaps I am missing something. Does the community have any ideas?
Thanks!

The jaggedness comes from the mask, so the way to get rid of the jaggedness is to smooth the mask before applying it to the image. Any smoothing applied after the masking will degrade the image.

The textbook approach is to apply a bit of smoothing to the mask (a Gaussian blur will work fine, you might need to convert the mask to floating-point first), and then multiply the image by the smoothed mask. Whether this approach will work for you will depend on what you use the image for after you mask it.

1 Like

This is actually nice idea, and I will certainly give it a try. However, it may not entirely align with the existed logic, as I am using the mask as a binary in vtkOpenGLGPUVolumeRayCastMapper. Thanks