Judging from your secreen captures, it’s not rotation, but handedness. These are two different concepts. A 3D coordinate system can either be right-handed or left-handed:
Either this or there is something ill-coded in your QVtkItem class. It is possible that you’re not properly handling the fact that the origin of screen coordinates is in the top left corner of the widget. Check the mirrorVertically property of your class: QQuickFramebufferObject Class | Qt Quick 5.15.6