Cannot import `_build_paths`

I am building VTK from source on Windows 10 using Ninja. After building, however, I cannot import vtk as it cannot import the _build_paths. Here is my setup:

Hardware


OS: Windows 10 Enterprise, x64-bit, Build 1909
CPU: 2x Intel(R) Xeon(R) Gold 6248R
Disk: 2TB NVMe M.2 SSD
RAM: 192 GB DDR4
Compute GPUs: 2x NVIDIA Quadro RTX8000
Display GPU: 1x NVIDIA Quadro RTX4000


Software


Ninja: 1.10.2 (Extracted from ninja-win.zip into C:\WINDOWS)
Visual Studio: Visual Studio 16 2019 Community
Windows SDK: 10.0.19041.0, targeting 10.0.18363
Python: 3.8.10 x64-bit
CMake: 3.21.1 (I use CMake-GUI)
vcpkg: 2021-08-12
VTK Source: Master branch, cloned as follows:

> cd C:\VTK
> git clone --recursive https://gitlab.kitware.com/vtk/vtk.git src

TBB: tbb2018_20170919oss


Steps to Reproduce

  1. Open x64 Native Tools Command Prompt for VS 2019
  2. Make C:\VTK folder and git clone source into subfolder src as shown above
  3. Start CMake-GUI by running cmake-gui
  4. In CMake-GUI, choose C:\VTK\src for the source folder and C:\VTK for the build folder
  5. Press Configure and choose Ninja as the build tool
  6. Change the following settings:
VTK_RELOCATABLE_INSTALL  OFF
VTK_GROUP_ENABLE_Imaging  WANT
VTK_GROUP_ENABLE_MPI  WANT
VTK_GROUP_ENABLE_Qt  WANT
VTK_GROUP_ENABLE_Rendering  WANT
VTK_GROUP_ENABLE_StandAlone  WANT
VTK_MAX_THREADS  96
VTK_MPI_NUMPROCS  48
VTK_PYTHON_VERSION  3
VTK_QT_VERSION  5
VTK_SMP_IMPLEMENTATION_TYPE  TBB
VTK_USE_MPI  ON
VTK_WRAP_PYTHON  ON
Python3_EXECUTABLE  C:/Program Files/Python38/python.exe
Python3_INCLUDE_DIR  C:/Program Files/Python38/include
Python3_LIBRARY  C:/Program Files/Python38/libs/python38.lib
MPIEXEC_MAX_NUMPROCS  48
MPI_C_HEADER_DIR  C:/Program Files (x86)/Microsoft SDKs/MPI/Include
MPI_msmpi_LIBRARY  C:/Program Files (x86)/Microsoft SDKs/MPI/Lib/x64/msmpi.lib
CMAKE_CXX_MP_FLAG  ON
CMAKE_BUILD_TYPE  Release
CMAKE_INSTALL_PREFIX  C:/VTK
TBB_DIR  C:/tbb2018_20170919oss/cmake
  1. Reconfigure and then press Generate. Only 1 warning appears during Configure, which seems harmless
CMake Warning (dev) at GUISupport/QtQuick/qml/CMakeLists.txt:100 (message):
  Qt5 is configured in both Debug and Release modes.  Due to Qt issue 47774
  (https://bugreports.qt.io/browse/QTBUG-47774), skipping generation of
  qmltypes file.  Using the one provided with the source tree instead.
This warning is for project developers.  Use -Wno-dev to suppress it.
  1. Back in x64 Native Tools Command Prompt for VS 2019, run ninja from C:\VTK directory
  2. After build completes, run ninja install
  3. Add the following to PYTHON_PATH and PATH environment variableS:
C:\VTK  
C:\VTK\bin  
C:\VTK\bin\Lib\site-packages\  
C:\VTK\include  
C:\VTK\include\vtk-9.0\  
C:\VTK\lib  
C:\VTK\lib\cmake
  1. Open a new python interpreter session in PowerShell and attempt to import vtk

Problem

Trying to import vtk in a Python38 interpreter started in PowerShell yields the following error:

❯ py
Python 3.8.10 (tags/v3.8.10:3d8993a, May  3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import vtk
Traceback (most recent call last):
  File "C:\VTK\bin\Lib\site-packages\vtkmodules\__init__.py", line 54, in <module>
    from . import vtkCommonCore
ImportError: DLL load failed while importing vtkCommonCore: The specified procedure could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\VTK\bin\Lib\site-packages\vtk.py", line 31, in <module>
    all_m = importlib.import_module('vtkmodules.all')
  File "C:\Program Files\Python38\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "C:\VTK\bin\Lib\site-packages\vtkmodules\__init__.py", line 56, in <module>
    import _vtkmodules_static
ModuleNotFoundError: No module named '_vtkmodules_static'
>>>

Using print statements, I’ve narrowed down the issue to the following line in C:\VTK\bin\Lib\site-packages\vtkmodules\__init__.py giving an import error

from . import _build_paths

e.g. If I put corresponding print statements in the _windows_dll_path function of C:\VTK\C:\VTK\bin\Lib\site-packages\vtkmodules\__init__.py like so:

    # Build tree support.
    try:
        print("Try import...")
        from . import _build_paths
        print("I worked!")

        # Add any paths needed for the build tree.
        for path in _build_paths.paths:
            if os.path.exists(path):
                _ = os.add_dll_directory(path)
    except ImportError:
        # Relocatable install tree (or non-Windows).
        pass

the result is

❯ py
Python 3.8.10 (tags/v3.8.10:3d8993a, May  3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import vtk
Try import...
Traceback (most recent call last):
  File "C:\VTK\bin\Lib\site-packages\vtkmodules\__init__.py", line 54, in <module>
    from . import vtkCommonCore
ImportError: DLL load failed while importing vtkCommonCore: The specified procedure could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\VTK\bin\Lib\site-packages\vtk.py", line 31, in <module>
    all_m = importlib.import_module('vtkmodules.all')
  File "C:\Program Files\Python38\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "C:\VTK\bin\Lib\site-packages\vtkmodules\__init__.py", line 56, in <module>
    import _vtkmodules_static
ModuleNotFoundError: No module named '_vtkmodules_static'
>>>

How can I fix this problem?

CC @ben.boeckel @mwestphal @dgobbi

Observing C:\VTK\src\Wrapping\Python\CMakeLists.txt, lines 169-182

set(vtk_build_paths_module
  "${CMAKE_BINARY_DIR}/${VTK_PYTHON_SITE_PACKAGES_SUFFIX}/vtkmodules/_build_paths.py")
if (WIN32 AND VTK_DLL_PATHS)
  string(REPLACE ";" "',\n    '" vtk_dll_paths_python "${VTK_DLL_PATHS}")
  file(WRITE "${vtk_build_paths_module}"
    "paths = [\n    '${vtk_dll_paths_python}',\n]\n")
  # Add the file to the zip only if required in the install tree.
  if (NOT VTK_RELOCATABLE_INSTALL AND NOT VTK_UNIFIED_INSTALL_TREE)
    list(APPEND _vtk_python_files
      "${vtk_build_paths_module}")
  endif ()
elseif (EXISTS "${vtk_build_paths_module}")
  file(REMOVE "${vtk_build_paths_module}")
endif ()

I notice VTK_DLL_PATHS is an empty string in CMake-GUI. Does this have to be set explicitly during configure/generate? Also, why isn’t this set by default if VTK_RELOCATABLE_INSTALL is set to OFF?

Even setting

VTK_DLL_PATHS  C:\VTK\bin

the Python module _build_paths.py is now created, which contains nothing more than

paths = [
    'C:/VTK/bin',
]

but the error still occurs.

@ben.boeckel @mwestphal @dgobbi I really need your help on this. Thank you in advance!

The reason that vtkCommonCore doesn’t import is because it depends on DLLs that aren’t in Python’s DLL search path.

The first thing you need to do (and I think this has been suggested before), is to use a good, modern dependency tracker to find all the DLLs that are needed. You can download the DependenciesGUI program, add it to your PATH, and then run it from within your bin directory:

    DependenciesGUI vtkCommonCore-9.0.dll

Or you can run it from within the vtkmodules directory:

    DependenciesGUI vtkCommonCore.cp38-win_amd64.pyd

Once you know what DLLs are needed, make sure they are all in the VTK_DLL_PATH. Just C:\VTK\bin by itself will be insufficient, it also needs the paths for the MPI DLLs, the TBB DLLs, and whatever other libraries you have linked to that aren’t in \Windows\System32.

@dgobbi You should notice that in the problem I report the issue is specifically with vtkCommonCore, which has solely to do with the core vtk dll, which does in fact exist in C:\VTK\bin. That being said, I also have the other pertinent dll’s in my PATH as well.

Why is it that vtkCommonCore would not load even though it is in my PATH?

The reason the vtkCommonCore DLL doesn’t load even though it is in the path, is that it needs other DLLs that are not in the path.

In a bare-bones VTK build, vtkCommonCore will only depend on system libraries and VTK’s built-in utility libraries like KWSys. But you mentioned that you built with MPI (and perhaps other external third-party libraries?). The DLLs for all those additional libraries must be in the path.

Yes, Python’s error message for this has always been terrible and there’s nothing we can do about that. However, 3.10 is said has useful information instead of misleading information in it.