Building VTK modules with dependencies results in race condition in make

Development
1

I have a project that builds multiple VTK modules, and some depend on other modules built in the same project. There seems to be some kind of race condition in the resulting Makefiles, that causes an error if a module ModuleB depends on a ModuleA.

vtkWrapHierarchy: couldn't open file .../build/lib/vtk/hierarchy/MyProject/ModuleA-hierarchy.txt
make[2]: *** [ModuleB/CMakeFiles/ModuleB-hierarchy.dir/build.make:66: lib/vtk/hierarchy/MyProject/ModuleB-hierarchy.txt] Error 1
make[1]: *** [CMakeFiles/Makefile2:234: ModuleB/CMakeFiles/ModuleB-hierarchy.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

Running make without invoking CMake again multiple times finally gets rid of this error; and when I’m lucky it works on the first try (therefore I think this is a race condition of some sorts).

I have broken down the problem to a minimal example (vtk_module_wrap_python is required, otherwise the hierarchies are not built). The project contains two subdirectories, where ModuleA and ModuleB are defined (they contain no code at all in the minimal example). The essential parts are listed below.
The full source is here: vtk9_race.tar.gz (901 Bytes)

I have tested this with VTK from the master branch and with VTK contained in ParaView-v5.7.0-RC2.

  • The error seems to only happen when using make for building (with -j1 and without), it seems to work fine using ninja.
  • It also works, if I first build target ModuleA-hierarchy, i.e., run make ModuleA-hierarchy; make.
  • The problem only occurs in CMake > 3.12. I successfully tested versions 3.8.0 and 3.12.2, but it breaks using versions 3.13.4 and 3.15.3.

CMakeLists.txt

cmake_minimum_required(VERSION 3.8 FATAL_ERROR) 

project(MyProject)
find_package(VTK REQUIRED)
option(BUILD_SHARED_LIBS "Build shared libraries" ON)

include(GNUInstallDirs)
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_BINDIR}")
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_LIBDIR}")
set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_LIBDIR}")
file(RELATIVE_PATH python_module_path "${VTK_PREFIX_PATH}" "${VTK_PYTHONPATH}")

vtk_module_find_modules(vtk_module_files "${CMAKE_CURRENT_SOURCE_DIR}")
vtk_module_scan(
    MODULE_FILES        ${vtk_module_files}
    PROVIDES_MODULES    VTK_MODULES
    WANT_BY_DEFAULT     ON
)
vtk_module_build(
    MODULES             ${VTK_MODULES}
    INSTALL_HEADERS     OFF
)
vtk_module_wrap_python(
    MODULES             ${VTK_MODULES}
    INSTALL_HEADERS     OFF
    PYTHON_PACKAGE      ${CMAKE_PROJECT_NAME}
    MODULE_DESTINATION  ${python_module_path}
)

ModuleA/vtk.module

NAME
  ModuleA

ModuleB/vtk.module

NAME
  ModuleB
DEPENDS
  ModuleA
2

So, I did some digging in the VTK source, and found this comment:

# Non-alias target modules are external and should already have their file
# generated.

The solution apparently is to give all modules names like MY::ModuleA and use these (aliased) targets instead (see below).
No idea though, why the original setup still works with older CMake versions.

ModuleA/vtk.module

NAME
  MY::ModuleA
LIBRARY_NAME
  myModuleA

ModuleB/vtk.module

NAME
  MY::ModuleB
LIBRARY_NAME
  myModuleB
DEPENDS
  MY::ModuleA
3

This is probably an oversight in the logic around that comment. Looking into it.

4

Please try this change: https://gitlab.kitware.com/vtk/vtk/merge_requests/6039

5

After some tests on my side, it seems another solution could be to force the target creating the hierarchy file to be executed prior to the main target.
On my case, for each module I have a:

add_dependeencies(${NAME} ${NAME}-hierarchy)

Also, you can also call cmake --taget ModuleB-hierarchy explicitely to force the computation of the missing file.

6

That’s not a valid solution; the module system should Just Work™ at this level. Does the MR above fix the problem for you?

7

I think the NOT in line 2685 of your patch is not supposed to be there.
Using the following patch, the problem is fixed for me:

--- a/CMake/vtkModule.cmake   2019-08-13 10:21:06.614636564 +0200
+++ b/CMake/vtkModule.cmake   2019-10-08 09:51:20.888553903 +0200
@@ -2677,13 +2677,12 @@
       "${_vtk_hierarchy_depend_hierarchy}")
 
     # Find the hierarchy target of the module.
-    get_property(_vtk_hierarchy_module_is_alias
+    get_property(_vtk_hierarchy_module_is_imported
       TARGET    "${_vtk_hierarchy_depend}"
-      PROPERTY  ALIASED_TARGET
-      SET)
+      PROPERTY  IMPORTED)
     # Non-alias target modules are external and should already have their file
     # generated.
-    if (NOT _vtk_hierarchy_module_is_alias)
+    if (_vtk_hierarchy_module_is_imported)
       continue ()
     endif ()
8

Oops. Thanks for the test. I’ve updated the branch and submitted it for testing (now that the atomic stuff has been addressed).