You can create a 3D frustum shape based on the 2D polygon drawn on the screen and the camera parameters, then convert this closed surface to a binary image. This binary image can be either used as a mask in the volume raycast mapper, or you can use it to blank out voxels of the rendered volume.
This is implemented using VTK in 3D Slicer. It is called the Scissors tool (a.k.a. Scalpel in other software):
You can use the feature as is in 3D Slicer using GUI or in your custom C++ modules or Python scripts. Or you can copy-paste the code into your application (Slicer’s permissive license allows using any parts of it for any purpose in any other software). The implementation in C++ is available here, for both slice and 3D views, perspective and parallel projections:
