Does it really need to be a subclass of vtkRenderWindow? You might be better of with a custom class that sets everything up.
Since you subclass vtkRenderWindow, you’re opting out of the factory overrides which support opengl. You might have better luck sub-classing vtk[X,Cocoa,Win32]OpenGLRenderWindow instead.