Oh.
Since the runtime detection occurs while constructing an instance of vtkRenderWindow, the class cannot know if OffScreenRendering is going to be set or not. This could be solved if VTK C++ classes accepted constructor keyword arguments.
By the way, OffScreenRendering (which is deprecated in favor of UseOffScreenBuffers and ShowWindow) does not mean that VTK will never create a real window. It only means that the geometry will be rendered into a framebuffer which is not related to the window.
As you’re interested in programatically influencing the render window selection, it would be nice if we made use of vtkGraphicsFactory::OffScreenOnlyMode. Then you can benefit from the feature like this:
vtkGraphicsFactory::SetOffScreenOnlyMode(true);
vtkNew<vtkRenderWindow> window;