We experimented with basic binary labelmap based masking and the results were really bad. It can only be useful if the boundary of the masked region is already empty and you only need to use a single color.
Instead, we developed a new method in 3D Slicer that works very well. It dilates labels and masks the volume with smooth transition, and generates an RGBA volume as output that VTK can directly render using raycasting.
Full implementation is available in 3D Slicer in “Colorize volume” module (in Sandbox extension) - see full source code.
Some more images and discussion is available here:
- VTK multivolume/cinematic volume rendering - Support - 3D Slicer Community
- New Colorize Volume module - Success stories - 3D Slicer Community
The processing (except the labelmap dilation) is very simple, so it would not be too hard to port to work on GPU, but it is probably not worth the trouble since processing takes only a few seconds on the CPU, volume rendering pipeline is already very complex, and shaders are a bit moving target in VTK now.