A typical reason for such issues (but unspecific to VTK) is uninitialized variables (as in debug mode for example under Windows, all variables get initialized automatically to a predefined value.
Can you provide example code that reproduces this behavior? What interaction style are you using? Are you intercepting the Qt mouse move events or the VTK ones?