VTK doesn’t use GLUT. On Windows, It uses the native Win32 API and WGL.
The place in the code where VTK creates a native Win32 window is here (in vtkWin32RenderWindow) and if my understanding is correct, any limitations on the size you get are due to display/desktop limits and are unrelated to OpenGL.
Since vtkWin32RenderWindow uses old Win32 API calls, I’m not sure how it will respond to recent Windows features like the “Scale” the Display Settings. It would be worth going to your display settings and changing the “Scale” to 100% to see if it changes the behavior of VTK. I notice, for example, that 1536 is 1920/1.25, so maybe your current “Scale” setting is 125%?