As far as I understand, VTK itself doesn’t restrict the size of the window. VTK simply requests a certain size of window from the operating system, and then accepts whatever size the operating system provides.
It might be possible to find the maximum size by calling GetScreenSize() and then subtracting the number of vertical and horizontal pixels needed for menus, borders, and other decorations. But this is fiddly and not very robust.
VTK applications often rely on a GUI toolkit such as Qt to handle window sizing in situations like the one that you’re describing.