vtkWrapPython disappeared from VTK-8.90

In 3D Slicer’s build system we rely on vtkWrapHierarchy.cmake and vtkWrapPython.cmake to wrap our VTK-based C++ classes. However, these macros seem to be refactored into a new macro, vtkModuleWrapPython.cmake. We already have our own module system, which is quite different from VTK’s (each module has VTK-based data and logic components and a Qt-based GUI component), there are many modules (there are hundreds of modules in Slicer core and there are about 150 remote extensions, each containing several modules, with various dependencies between them), so it would not be feasible to build all these as VTK modules.

@dgobbi @ben.boeckel Would it be possible to restore the original vtkWrapHierarchy and vtkWrapPython features and implement vtkModuleWrapPython.cmake to use them? Do you have recommendation for an alternative solution?

Both have been adapted to rely on the properties provided by the module system; they are quite linked. If the old macros were sufficient for ITK, feel free to copy the old versions into ITK and reuse the wrapping binaries from VTK (they haven’t changed with the new system and I don’t expect that to be any different in the future). What I don’t want to do is add back the old (and what would be untested) CMake wrapping API to VTK itself.

The benefits the new APIs that the old ones just do not provide:

  • knowing where a hierarchy file for an external module is (the old one just kind of guessed)
  • ability to depend on external modules in your Python wrapped libraries (the Python module path is now available on the target itself so it knows what to import before bringing in your module)
  • ability to wrap external modules (vtkWrapPython could conceivably be a completely separate project and wrap a hypothetical Python-less VTK just fine)
  • removal of the PythonD libraries; I don’t know that its removal was ever backported to the old macros
  • Python static builds work properly with Python3

So no, I don’t think it’s possible to rebase the current feature set on top of the old macros. However, if the old ones work for you, please feel free to take them; the tool APIs have not changed, just the CMake API making them available in CMake code.

That said, enhancing the VTK module system to also work for ITK/Slicer probably isn’t too hard. VTK conventions may be there as defaults, but they should be able to be avoid or, if not, at least relaxed to allow ITK/Slicer to use it more easily.

1 Like

Hi Andras, I’m in the same situation as you, but to a lesser degree. I also wrote my own module system a few years ago, because I needed a good way to express the dependencies of our packages at work on Qt, VTK, and ITK (and on each other). And, likewise, adopting VTK modules currently seems to be a no-go because they are too VTK-specific. Just to make things clear, I wasn’t part of the design of the current module system. But having used it for vtk-dicom, I can say that it works well (for VTK stuff, at least).

My recommendation is the same as Ben’s: bring vtkWrapPython.cmake and vtkWrapHierarchy.cmake into Slicer. But the ones from VTK 8.2 will definitely not work. It is necessary to use the ones from the following MR, which match the current wrapping tools:
https://gitlab.kitware.com/vtk/vtk/merge_requests/3075

The big change in the Python wrappers between 8.2 and 8.90 was the elimination of the “PythonD” shared libraries. This change in the wrappers required significant changes to the cmake code, which are reflected in the MR above.

2 Likes

@ben.boeckel @dgobbi Thanks for your useful insights and advice.

We have a few other projects that relied on vtkWrapPython and I guess there are many others out there. Copy-pasting would work well in the short term, but it would be hard to maintain in the long term (we would end up with many slightly different variants of the script in various projects). What do you think about these options:

  • Maintain vtkWrapPython macros in a separate repository
  • Maintain the entire VTK\Wrapping folder and vtkWrapPython macros in a separate repository
  • Add back vtkWrapPython macros to VTK with some automated tests to make sure they remain functional, even if VTK modules don’t rely on them (I know that @ben.boeckel wrote that this is not a preferred option but actually this seems to be the simplest and least amount of work overall)

@lassoan I’m very much in favor of externalizing the VTK wrapping, but I don’t see it happening before VTK 9. There is a github project called WrapVTK that I wrote for VTK 5.8 and have maintained through VTK 8.2, it generates an XML description of the VTK API rather than the wrappers themselves, but it serves as a proof-of-concept that external wrapping is doable.

2 Likes

OK. So, maybe we’ll implement some short-term solution (copy-paste) now and revisit this question after VTK9 is released.

I’ve also asked @jcfr to comment on this thread and/or talk about this with @ben.boeckel when they meet later this week.

This would be OK. I’m more worried about it being untested.

What about it was too specific? Relaxing VTK assumptions is OK with me, but I’ve only relaxed beyond my initial thoughts where they’ve affected other projects.

Really just that everything that a module lists in DEPENDS must, itself, be a module of the same type. This might be an unfair assessment, but there doesn’t appear to be any support for heterogeneity (e.g. listing ITK modules or Qt packages in DEPENDS).

That’s because the targets listed in DEPENDS are expected to have all the module metadata attached to them. Their own dependencies, wrapping information, etc.

What would you expect a non-module in that list to do? How would vtkModule.cmake, upon seeing itkModule as a dependency know to say “hey, you want to build that because I want it built”? How would it affect cache variable selections/exposure? Nothing stops you from vtk_module_link() for ITK modules.

The expectation is that any non-module (specifically, any non-native-module) in the list is an external dependency. If the package that should supply it doesn’t supply it, then an error is raised.

Is there a reason that vtk_module_link isn’t sufficient?

What could be done is something like:

TARGET_DEPENDS
  PUBLIC
    foobar
  PRIVATE
    itkModule

which is basically a shortcut for calling vtk_module_link with those arguments. I don’t think they should be expected to be targets since that would disallow genexes. Or maybe that kind of stuff should be forced to be in the CMakeLists.txt. Thoughts?

Of course vtk_module_link is sufficient, it just isn’t as convenient. The use of PRIVATE is a no-go for people who want to make a module that provides ITK interface classes.

The module system that I wrote for work just compares all the listed dependencies against

  • the list of modules provided by the package that is being built, and
  • the lists of modules/libraries made available by all external packages (I have package-level macros for wrangling those)

But I don’t have to deal with wrapping at work (ironic, I know), and that simplifies things.

That was merely an example; itkModule could just as easily be added to a PUBLIC or INTERFACE section too. The module system would likely just take those and make its own vtk_module_link call internally; not much else to do AFAICT.

SMTK is a project that has VTK code, but is not built using the VTK module system everywhere. Just its VTK code is in VTK modules. The interface logic is here. I can explain the details if anyone is curious about things.