Rotating my volume using setUserMatrix results in a skewed volume

All these computations are implemented in vtkTransform. I would recommend to use that.