The figure below illustrates how one uses a transform matrix to calculate the result:
The left hand term is the new XYZ of one point of your geometry. The 4x4 matrix is your transform matrix which, in this case, is a rotation about the Z axis. The right hand term is the original XYZ location of one point of your geometry. The 4th row/column in the matrices for 3D transforms are needed to allow projective geometry computations such as the one you need.
Now, replace the rotation matrix above with VTK’s projection matrix and perform this calculation for every vertex of your geometry that lie inside the frustum. If you want to delve into the wonderful world of Computer Graphics/Geometry, I recommend reading a primer such as Graphics Gems (Amazon.com).