CMake flags used to compile Python wheel

Support
1

Hi,
I was trying to go through the .ci/*.cmake files to understand how exactly the Python3 wheels (mainly interested in Linux) are generated, but I found that many optional flags are not mentioned (thus I’m assuming set to off?). I’m mainly looking at VTK_PYTHON_FULL_THREADSAFE (that I’m assuming is off) and VTK_SMP_IMPLEMENTATION_TYPE (also sequential?). I have been wondering why, considering how hard it is to achieve multithreading in Python due to the GIL, these flags are not turned on by default to have multithreaded code on the C++ side of things.

Francesco

2

Oddly enough, the VTK_PYTHON_FULL_THREADSAFE setting can only be configured when building Paraview. It doesn’t appear when building VTK itself.

I agree that VTK_PYTHON_FULL_THREADSAFE should be ON by default, and last year I submitted a patch (!9223) to VTK. The same patch also releases the GIL during long stretches of C++ code execution, in order to unblock the execution of python threads. If you’re willing to read through the patch and give your opinion, that might help to give it enough momentum to actually get fully reviewed and merged.

Regarding C++ threading, the wheels provide both Sequential and STDThreads as SMP backends. The VTK_SMP_IMPLEMENTATION_TYPE only indicates the default backend. The full set of defaults for VTK (and the wheels) is as follows:

VTK_SMP_IMPLEMENTATION_TYPE=Sequential
VTK_SMP_ENABLE_SEQUENTIAL=ON
VTK_SMP_ENABLE_STDTHREAD=ON
VTK_SMP_ENABLE_TBB=OFF
VTK_SMP_ENABLE_OPENMP=OFF

The STDThread backend can be turned on at runtime like so:

>>> vtkSMPTools.SetBackend('STDThread')

It isn’t the default because it has performance issues (and potential stability issues) which are being worked on in !10027.

The VTK imaging pipeline, at least, is parallel on all platforms/configurations because it falls back to native threads (e.g. posix threads on linux).

1 Like
3

That is strange…though it was added in 2015 (MR 582) in just the code. I suspect it was pushed from the ParaView side, never got a VTK exposure, and got missed when I rewrote things in ParaView to have a PARAVIEW_ prefix…

Anyways, exposing it to VTK probably isn’t the worst thing, though I do remember someone trying to do it and I wondered why it should be an option at all…

4

Thank you for your replies!
The VTK_PYTHON_FULL_THREADSAFE was discussed in a pipermail thread of VTK many years ago, but I can’t seem to find it…
I’ll give a look at the patch and comment about it. I don’t really see many reasons why it shouldn’t be the default. And good to know that the multithreading can be enabled at runtime, reading VTK_SMP_IMPLEMENTATION_TYPE sounds very “definitive” for a given build. Thank you!

5

Yes, I pushed for that to avoid adding yet another axis to VTK’s build configuration.