Remote modules vs external modules

[dgobbi (David Gobbi)]

Over the weekend, I started updating vtk-dicom to the new VTK 9 build system.

For previous versions of VTK, it was possible to build vtk-dicom either as part of the VTK build (i.e. as a remote module, thanks to Bill Lorensen), or as an external package against an existing VTK build. External builds have always been challenging due to the fact that, historically, the VTK cmake commands for wrapping are neither streamlined nor stable between VTK versions.

I plan to continue supporting both remote and external builds of vtk-dicom for VTK 9. But there seems to be a hitch. For remote modules, vtk.module must be in the base directory of the repository, but for external packages, vtk.module must be in a subdirectory (so that it can be found by the main CMakeLists.txt for the package). I can solve this problem by having two identical copies of vtk.module, one in the base directory and another in the directory that contains the source code for the module. But it’s a rather ugly solution.

The best solution would probably be to expand the idea of “remote module” to also allow for “remote packages”, where a package can contain one or more modules (e.g. as subdirectories). For vtk-dicom, at least, I think it would make it easier for me to maintain things moving forward.

Does anyone have opinions on this?

[lassoan (Andras Lasso)]

I don’t know what is the best approach to implement this, but I can confirm that building VTK remote modules after VTK was already built, in a separate build tree is a very important use case for 3D Slicer extensions, too.

For example, in SlicerVirtualReality extension builds VTKRenderingOpenVR VTK remote module after VTK has been already built, in a separate build tree.

This is the main reason we keep backporting essential VTK patches instead of switching to a recent VTK master.

[dgobbi (David Gobbi)]

Hi Andas, I’m looking at the cmake files for SlicerVirtualReality, and I can’t figure out where it gets VTKRenderingOpenVR from. It’s marked as an external project, but it doesn’t seem to be downloaded, nor is it inside the SlicerVirtualReality repository. Where is it located?

[ben.boeckel (Ben Boeckel (Kitware))]

@dgobbi

For remote modules, vtk.module must be in the base directory of the repository

I don’t think that’s true? We recursively glob for the file.

@lassoan

I can confirm that building VTK remote modules after VTK was already built, in a separate build tree is a very important use case for 3D Slicer extensions, too.

I’d prefer to see these stay as external modules (possibly with contract testing in VTK).

@lassoan

For example, in SlicerVirtualReality extension builds VTKRenderingOpenVR VTK remote module after VTK has been already built, in a separate build tree.

I consider this an unsupported configuration; you’ll need to mangle your VTK::RenderingOpenVR module if VTK isn’t building it (or accept the one VTK came with).

[dgobbi (David Gobbi)]

@ben.boeckel

I don’t think that’s true? We recursively glob for the file.

If so, that makes things easier. Then I can arrange things so that the CMakeLists.txt in my base directory is only used when building as an external project.

[lassoan (Andras Lasso)]

@ben.boeckel

I consider this an unsupported configuration; you’ll need to mangle your VTK::RenderingOpenVR module if VTK isn’t building it (or accept the one VTK came with).

At the time VTK is built, we don’t know what remote modules various extensions will need.

How do you mangle a module?

[ben.boeckel (Ben Boeckel (Kitware))]

If HDF5 is built without MPI, do you go and build its MPI bits if you need them? No, you error saying “the HDF5 you gave me is not suitable for what I need”. Why are we trying to make VTK so different?

As for proper mangling, things that need changed:

• module name (something like ITK::VTKRenderingOpenVR)
• symbol names (a mechanism will need to be added for this)
• header locations, library name (controllable through vtk_module_build already)

[lassoan (Andras Lasso)]

I agree that it is not reasonable to change VTK build options externally (e.g., enable Qt or change the rendering backend). What we need is to be able to leverage VTK’s build system to build VTK-based external libraries both as part of VTK build; and after VTK build as a separate library.

It is of course possible to implement handling of different build modes in each remote module, as it is done in vtkDICOM. However, that would mean copy-pasting, slightly modifying, and maintaining significant amount of non-trival CMake code between all remote modules.

Instead, it would be much better if VTK would contain common CMake files that all remote modules could include. It would simplify the work of both remote module developers (they would need to develop and maintain less CMake code) and VTK developers (as build system of remote modules would be more uniform and they could be tuned all at once by modifying scripts in the main VTK repository).

OpenVR may be a special case, because it is a remote module stored in the VTK source tree (at least this was the status before the large VTK build system rework). OpenVR may have to be converted to a proper remote module or a regular local VTK module.

[ben.boeckel (Ben Boeckel (Kitware))]

@lassoan

What we need is to be able to leverage VTK’s build system to build VTK-based external libraries both as part of VTK build; and after VTK build as a separate library.

Why does it need to support being built inside VTK? Why not always just do find_package(VTK) just like any other dependency? I’d prefer these specialized modules become separate repositories that rely on VTK just like VTK relies on Qt: an external dependency.

[lassoan (Andras Lasso)]

@ben.boeckel

Why does it need to support being built inside VTK?

By building inside VTK I mean that developers can add remote modules to VTK by changing CMake options when configuring VTK. There are many reasons why it is useful:

• convenience feature for developers: many small projects don’t even use superbuild to build dependencies, so cutting down number of libraries they have to configure manually is a big deal
• flexibility for VTK maintainers: if VTK maintainers decide that they want to reduce the VTK tree size or don’t want to maintain some VTK modules anymore then modules can be moved to external repositories without impacting projects that rely on that VTK module
• dependency between modules can be automatically handled (e.g., if remote modules depend on other remote modules)

@ben.boeckel

I’d prefer these specialized modules become separate repositories that rely on VTK just like VTK relies on Qt: an external dependency

Do you mean that you would prefer if remote modules infrastructure would go away entirely? That would not be good for anyone, as it would mean less flexibility and increased maintenance burden for both VTK and remote module developers.

[ben.boeckel (Ben Boeckel (Kitware))]

@lassoan

By building inside VTK I mean that developers can add remote modules to VTK by changing CMake options when configuring VTK.

With the new system, VTK’s options have documented behaviors. I really don’t want to even allow remote modules to interfere with such things. Being inside the build is too close; it’s basically an implicit friend declaration for VTK’s own build system. It was not designed with such access in mind.

@lassoan

convenience feature for developers: many small projects don’t even use superbuild to build dependencies, so cutting down number of libraries they have to configure manually is a big deal

We’re planning on providing packages built by VTK more often starting with 9.0. It should be as easy as pip install https://vtk.org/... once that happens for Python. Not sure about C++ yet; maybe Anaconda, .dpkg, and .rpm packages is good enough there.

@lassoan

flexibility for VTK maintainers: if VTK maintainers decide that they want to reduce the VTK tree size or don’t want to maintain some VTK modules anymore then modules can be moved to external repositories without impacting projects that rely on that VTK module

If that were happening, I’d do (for example):

find_package(vtkio REQUIRED)
add_library(VTK::IOXML INTERFACE)
target_link_libraries(VTK::IOXML INTERFACE vtkio::xml)

Not build it as part of VTK.

@lassoan

dependency between modules can be automatically handled (e.g., if remote modules depend on other remote modules)

More find_package. Just like every other project out there. You’re already going to have to support that otherwise you’re forcing everyone to build a custom VTK to build your project (since it’s not going into any of the packages that are part of the packaging mentioned above). That’s not nice either.

@lassoan

That would not be good for anyone, as it would mean less flexibility and increased maintenance burden for both VTK and remote module developers.

Yes, I, personally, want it to go away. I’ve yet to hear an argument for why it should be possible that isn’t basically “that sounds like a lot of work”. And before, I could see why. But now that the API VTK uses is just as usable outside of it, that sounds fine to me. If you use the module system, it even helps you write your package configuration file for your dependencies! It can’t write your top-level config file, but it does what it can. Plus, what VTK doesn’t export into its install tree for is explicitly not part of any API guarantee and I won’t care if it ends up breaking a remote module.

As for burdens, it’d mean less for VTK. find_package(VTK) has to work anyways, so that’s not a problem. It also means we don’t have to halt development when remote modules don’t follow VTK closely enough (new testing machines being broken with remote modules, having remote modules rely on accidentally-public stuff that is only accessible in a build, etc.). Yes, it’d mean more for remote module developers, but why is VTK so different than HDF5 or any of your other dependencies?

[dgobbi (David Gobbi)]

For vtkDICOM, if I was forced to choose between supporting external builds vs. building as a remote module, then I would definitely go for external build. However, I can’t deny that I like the convenience of remote modules and don’t want them to go away.

One concern that I have is packaging (pypi). Are the remote modules going to be included in the main vtk package? I kinda think they shouldn’t, even though it means I’ll have to make my own vtkDICOM pypi package. Neither does it make sense for them to be built by default, except for by the dashboards for testing.

Also, for Python, I don’t know if the package structure is resolved yet. The python modules for external packages obviously cannot accessed by “import vtk.<module>” or “import vtkmodules.<module>”. So should things be set up to that people use “import vtk<package_name>.<module>” or simply "import vtk<module>"?

[ben.boeckel (Ben Boeckel (Kitware))]

@dgobbi

However, I can’t deny that I like the convenience of remote modules and don’t want them to go away.

What I want to do is convert them to contract tests.

@dgobbi

One concern that I have is packaging (pypi). Are the remote modules going to be included in the main vtk package?

I really don’t want to do that. You’ll be subject to VTK’s version scheme at that point. And since development for them happens all over the place, respinning releases because a bug was fixed in vtkDICOM seems a bit much to me.

@dgobbi

Also, for Python, I don’t know if the package structure is resolved yet. The python modules for external packages obviously cannot accessed by “ import vtk.<module> ” or “ import vtkmodules.<module> ”. So should things be set up to that people use “ import vtk<package_name>.<module> ” or simply " import vtk<module>" ?

Python is where it gets really hairy. There’s no real way to distribute a package that can live in one of two places which doesn’t force everyone to do:

try:
    import vtkdicom
except ImportError:
    import vtkmodules.dicom as vtkdicom

without making a third package/module to do it for you and having everyone use that one. The vtk and vtkmodules bits built as part of ParaView has this issue; it cannot be vendored in any sensible way without affecting everyone who wants to be flexible over how VTK is provided. (A big reason I really want ParaView to rely on VTK releases rather than a submodule, but that’s a long road.)