RFC: Toward supporting distribution of VTK based modules on PyPI

jcfr · December 21, 2020, 7:40pm

In a nutshell, we would like to officially support the ability to package and distribute any VTK based modules on PyPI (Python Package Index).

This means we would maintain a template to allow anyone in the community to do so.

There is of course a lot of technical details we will have to consider but before putting together a technical proposal, we would like to gather input from the community to make sure we consider all aspects.

In recent years, Python has become the most widely used programming language (and in particular, in important areas, such as machine learning [1]). Since numerous Python packages are available on PyPI for all imaginable computational tasks, libraries that are not available on PyPI get simply ignored. Algorithm developers who were happy for many years with just making C++ source code available now need a way to distribute their work on PyPI. The ability to easily build VTK modules against VTK itself and allow anyone in the community to distribute VTK based modules will be key. We think it is currently an unmet need.

Thanks for your input,

J-Christophe, Will, Utkarsh, and Stephen

[1] https://medium.com/analytics-vidhya/machine-learning-technology-trends-2020-2d0212813c05

utkarshayachit · December 21, 2020, 8:30pm

Generally-speaking, I am in support of this initiative. I’d revise the requirement, however, as “ability to build ~~VTK~~ modules against VTK ~~itself~~ and allow anyone in the community to distribute VTK-based modules”.

When you say a VTK-module, to me it sounds like it’ll be in VTK. So I should able to do from vtkmodules import MyNewModule. I am not sure that needs to be a requirement. Maybe making ability to support import MyNewModule easy is all that’s needed.

pieper · December 21, 2020, 8:32pm

This would be incredibly valuable and would open up use of VTK to a much broader community.

lassoan · December 21, 2020, 9:51pm

This would be awesome. It would allow us to make available lots of VTK-based tools and algorithms that we developed over the years to developers/researchers who do most of their work in Python - which are most of the new graduates in our institution (and probably at many other places, too).

I agree that there is a difference between VTK core modules (stored in VTK repository, can only be built as part of VTK) and VTK remote modules (stored in external repositories and can be built against a VTK build tree and maybe even install tree). So, we could say that there are two requirements:

Ability to easily build VTK remote modules against VTK and allow anyone in the community to distribute VTK based remote modules.
Ability to easily build VTK core modules against VTK and allow anyone in the community to distribute VTK based modules.

We all already agree on necessity of requirement 1.

Requirement 2 is needed because there are several non-essential modules in VTK repository that would not be practical to include in the main VTK Python package on PyPI as they would make VTK package bloated with features that very few people need and/or would require special hardware or software configurations to be usable. For example, all the domain-specific modules (Chemistry, ChemistryOpenGL2, Microscopy, ParallelChemistry, GeoVis, …), some of the parallel modules, exotic IO modules, OpenVR, all the GUI support (Qt, MFC, Tk, …) modules should be probably distributed as separate packages on PyPI (e.g., vtk-chemistry, vtk-geovis, vtk-openvr, …).

An alternative solution would be to move out all non-essential VTK modules to remote modules. This would be useful also because it would decrease size, complexity, and maintenance workload of the main VTK repository. Important VTK modules could be still hosted in Kitware’s gitlab and VTK developers could have direct write access to them.

kcburns0428 · December 22, 2020, 12:12am

We use VTK in Jupyter Notebooks within in an Anaconda environment on Debian. We will only use VTK modules that are installed via Anaconda’s Conda package manager, and thus are contained within the environment.

This helps us maintain package compatibility within the Anaconda environment, and helps us keep the system-wide Debian clean and minimal.

Thank you,

Ken

lassoan · December 22, 2020, 12:29am

Is your point that not just PyPI but conda package manager should be considered? Do you impose any limitations on using conda-forge recipes, building packages created from recipes obtained from PyPI, or installing packages using pip?

kcburns0428 · December 22, 2020, 4:38am

The Conda package manager has a default channel, and that it what we use in order to ensure package compatibility; so that is what we would prefer you set as your target. One big downside of this from your perspective is that you have to stay in-line with them as to what versions of dependent packages you use.

I understand that conda-forge and other channels are available, but using these could change (update or downgrade) some existing packages is such a way as to break package-compatibility within the environment… Also, we maintain security by periodically updating our Anaconda environment using Conda’s default channel, so would run the risk of breaking any packages installed from elsewhere (depending on their dependent package versions). So, packages from conda-forge and other channels are only evaluated under duress, and very seldom get deployed.

We do not use pip due to package compatibility concerns.

Aptitude is what we use for maintaining the system-wide Debian (and, again, we restrict ourselves to the Debian-provided repositories). Also, we keep our system-wide Debian installation as small as possible by restricting our application-related software to its environment. This allows us to maintain security by periodically updating our Debian without breaking package-compatibility.

Obviously, we are intent on maintaining system stability; and are willing to sacrifice functionality to do so.

Nobody likes a system that used to work.

Thanks,

Ken

phcerdan · December 22, 2020, 4:10pm

Hi @jcfr, sounds great. Could you explain the technicalities a bit?

Do you mean that any python app using VTK with that template would link dynamically to the main VTK cpython library, and would be able to operate with objects from proper VTK python wrappings (import vtk)?

In my recent experience, I have used VTK/ITK internally, and exposed my own wrappings of a few VTK objects when needed. The problem of course is that my_wrapped_vtk_image is not binary compatible with the same VTK object from import vtk. So I have used bridge functions in pure python (usually via numpy) to convert between these.

Just trying to understand better, I like the idea of course.

ben.boeckel · December 22, 2020, 4:52pm

Any module which requires a build tree and doesn’t work with a build tree is either:

using internal APIs (bad module, don’t do that); or
a bug in VTK’s install tree

Please file bugs when this happens.

This sounds like a very good line to use for moving some code out of VTK and into its own repository (or multiple!). They’d also serve as an example of “see, here’s how to make a wheel of a VTK-using project”. Actually doing the split is likely to be a huge undertaking (and ParaView will need to support an external VTK first in any case for this to become a practical reality).

(replying to the overall effort here):

I do have issues with wheels in particular because shipping C++ APIs with wheels is a very unsolved problem that I haven’t seen anyone working on it. Conda does support non-Python code and is likely more suitable for a proper solution for this (though if wheels do end up having ways of shipping non-Python APIs reliably, we can look at that again). Things to consider:

where do headers live within a wheel?
where should the vtk-config.cmake and associated files go?
VTK hierarchy files also need to go into the package

Now the new VTK build system should be able to deal with any such layout, but making it Just Work is up to conventions in the wheel-using ecosystem.

lassoan · May 8, 2021, 8:37pm

Just a quick update. We have of a simple VTK-based library that can be used as an example for VTK8/9 compatible Python wrapping by reusable generic macros: vtkAddon. @jcfr has started to work on VMTK, which is a quite big library, with superbuild, continuous build on github, Python wrapping, used within a C++ application (3D Slicer) and also distributed via PyPI. We expect it to be ready in a couple of weeks and it should serve as a good template for developing VTK9-based tools.

jcfr · June 30, 2021, 6:14pm

VTK wheels

Thanks for the outstanding work of @ben.boeckel

a month a ago, we added support for building VTK wheels on Linux, macOS and Windows (vtk mr-7898)
and we are now updating the release process to have VTK wheels uploaded. See vtk mr-8128

Externally build VTK remote or built-in modules

To put things in context, VTK modules can be built in the following scenarios:

Module sources and generated buildsystem are respectively in VTK source tree and VTK build tree (aka built-in case)
Module sources are external and module generated buildsystem is in VTK build tree (aka built-in case for a VTK remote module)
Module sources and module generated buildsystem are both external
Module sources are in VTK source tree and module generated buildsystem is external

To support use case (3) and (4), we created a project called VTKExternalModule.

It is based on prior work of @dgobbi

See https://github.com/KitwareMedical/VTKExternalModule

VTKExternalModule: Highlights

In a nutshell, you can now externally build any VTK remote module like kitware/LookingGlassVTKModule, or any built-in module like Rendering/OpenVR, …
This is foundational work for an upcoming VTKModuleTemplate project like ITKModuleTemplate that @thewtex and many others have been working on.
Building against VTK install or build tree works

VTKExternalModule: Next steps

work with @ben.boeckel to have publish VTK SDK
create VTKModuleTemplate and add relevant GitHub actions for continuous integration (CI)
add wheel building support
support for selecting install components to associate with default install target
support for specifying dependent VTK modules
…

VTKExternalModule: An example

Assuming you have a build of VTK with python enabled (with mr-8123), HoloPlayCore downloaded, …

VTK_DIR=/home/jcfr/Projects/VTK-install/lib/cmake/vtk-9.0/
# VTK_DIR=/home/jcfr/Projects/VTK-build

git clone git://github.com/KitwareMedical/VTKExternalModule /tmp/VTKExternalModule
git clone git://github.com/kitware/LookingGlassVTKModule /tmp/LookingGlassVTKModule

VTK_EXTERNAL_MODULE_SOURCE_DIR=/tmp/VTKExternalModule
VTK_MODULE_NAME=RenderingLookingGlass
VTK_MODULE_SOURCE_DIR=/tmp/LookingGlassVTKModule
VTK_MODULE_CMAKE_MODULE_PATH=

VTK_EXTERNAL_MODULE_BUILD_DIR=/tmp/${VTK_MODULE_NAME}-build
VTK_EXTERNAL_MODULE_INSTALL_DIR=/tmp/${VTK_MODULE_NAME}-install

HoloPlayCore_DIR=/home/jcfr/Projects/SlicerLookingGlass-r/HoloPlayCore
HoloPlayCore_INCLUDE_DIR=${HoloPlayCore_DIR}/HoloPlayCore/include
HoloPlayCore_LIBRARY=${HoloPlayCore_DIR}/HoloPlayCore/dylib/linux/libHoloPlayCore.so


~/Software/cmake-3.20.1-linux-x86_64/bin/cmake \
  -DHoloPlayCore_INCLUDE_DIR:PATH=${HoloPlayCore_INCLUDE_DIR} \
  -DHoloPlayCore_LIBRARY:FILEPATH=${HoloPlayCore_LIBRARY} \
  -DVTK_DIR:PATH=${VTK_DIR} \
  -DVTK_MODULE_NAME:STRING=${VTK_MODULE_NAME} \
  -DVTK_MODULE_SOURCE_DIR:PATH=${VTK_MODULE_SOURCE_DIR} \
  -DVTK_MODULE_CMAKE_MODULE_PATH=${VTK_MODULE_CMAKE_MODULE_PATH} \
  -DCMAKE_INSTALL_PREFIX:PATH=${VTK_EXTERNAL_MODULE_INSTALL_DIR} \
  -B ${VTK_EXTERNAL_MODULE_BUILD_DIR} \
  -S ${VTK_EXTERNAL_MODULE_SOURCE_DIR}

cmake --build ${VTK_EXTERNAL_MODULE_BUILD_DIR} --config Release --target install -- -j4

cd ${VTK_EXTERNAL_MODULE_INSTALL_DIR}
find . -type f

./lib/cmake/vtk/vtkRenderingLookingGlass-targets-noconfig.cmake
./lib/cmake/vtk/vtkmodules-vtk-python-module-properties.cmake
./lib/cmake/vtk/vtkRenderingLookingGlass-vtk-module-properties.cmake
./lib/cmake/vtk/vtkRenderingLookingGlass-targets.cmake
./lib/vtk/hierarchy/vtkRenderingLookingGlass/vtkRenderingLookingGlass-hierarchy.txt
./lib/libvtkRenderingLookingGlass.so.0.1.0
./lib/python3.6/site-packages/vtkmodules/vtkRenderingLookingGlass.so
./include/vtkrenderinglookingglass-0.1/vtkLookingGlassPass.h
./include/vtkrenderinglookingglass-0.1/vtkLookingGlassInterface.h
./include/vtkrenderinglookingglass-0.1/vtkRenderingLookingGlassModule.h

ben.boeckel · June 30, 2021, 8:23pm

jcfr:

./lib/cmake/vtk/vtkRenderingLookingGlass-targets-noconfig.cmake
./lib/cmake/vtk/vtkmodules-vtk-python-module-properties.cmake
./lib/cmake/vtk/vtkRenderingLookingGlass-vtk-module-properties.cmake
./lib/cmake/vtk/vtkRenderingLookingGlass-targets.cmake

This is wrong. These are not VTK and should not be found via find_package(VTK). If one wants something like this, find_package(VTKLookingGlass) is how it should work. Of particular interest is that the python-module-properties.cmake file just clobbered VTK’s copy of that file. How are these files to be included anyways? vtk-config.cmake won’t look at them.

I’m really not interested in supporting this. The code in VTK builds as part of VTK. If that’s not suitable, let’s move it out of VTK and make it its own repo so that it can be used separately and can be expected to work separately. This appears to just be clobbering parts of VTK’s installed CMake package.

I really don’t want to support this either. VTK does a lot of work to make itself useful for VTK-using projects and expecting VTK to pick up the slack for making a proper CMake package is too far. Any module that only supports being built this way will not appear in official wheels or builds at all because this code is outside of VTK’s software process or quality control.

This is fine…it’s just normal CMake project. Why is VTKExternalModule helpful here?

This is also not correct…it’s not VTK code and should be in its own package. This is just like if VTK shipped a numpy.vtk module…it’s not NumPy’s code and we shouldn’t inject ourselves into its package.

jcfr · March 21, 2024, 2:52pm

Update:

To streamline the development and dissemination of VTK-based modules on the Python Packaging Index (PyPI), with help of @ben.boeckel and @Patrick_Avery we already worked on the creation of the vtk-wheel-sdk^[1]

This initiative has facilitated the creation of several projects such as:

vtk-u3dexporter
vtk-addons
vtk-cleaver
vtk-lookingglass

While this has been useful, the current approach requires to maintain a substantial amount of boilerplate code within each project. This lack of scalability hinders the wider adoption of VTK as a foundational tool for building and disseminating Python-based modules.

As a first step to address this, we are now working on creating a first-class vtk-sdk wheel. This wheel is intended to distribute the content of the existing VTK wheel SDK while also introducing a scikit-build-core cmake.module entrypoint^[2].

While our ultimate goal involves updating the VTK build-system and GitLab CI, we will approach this in stages. As a next step, we will move forward with the creation of the dedicated GitHub project to support creation of the vtk-sdk wheels.

Thanks @alexy.pellegrini and @finetjul for the help.