python 3.8 wheel on Mac OS X

Support
#1

Hi - I’ve been trying to figure out how to get vtk to work with python 3.8, and in particular in making a wheel for easy installation. The basic vtk functionality works when compiling from source (using the patch from https://gitlab.kitware.com/vtk/vtk/uploads/cf7f517b4f647de1997c4e8cddb54de9/0001-Compatibility-for-Python-3.8.patch), including the python wrapping module which allows my python application to work. I was also able to modify the wheel building infrastructure from https://github.com/KitwareMedical/VTKPythonPackage and got it to compile and apparently install. The changes were minimal (and I’d be happy to post the patch if it’s useful), mostly applying the source patch above, changing venv to python3 -m venv, and changing some paths and version numbers. I also created a dummy libpython-not-needed-symbols-exported-by-interpreter that is actually a library, not an empty file, to deal with the problems mentioned in http://vtk.1045678.n5.nabble.com/vtk-8-2-0-rc2-problem-building-wheels-td5749301.html.

This process creates a wheel that I can install with pip, but the resulting directory structure is very weird. When I directly install from source I get a bunch of libvtk*.so files in PREFIX/lib, and vtk.py and vtkmodules in PREFIX/lib/python3.8/site-packages. If I add those to DYLD_LIBRARY_PATH and PYTHONPATH things works fine. However, the wheel (which is installing to the Mac OS user directory PREFIX=~/Library/Python/3.8), creates a vtk/ directory in PREFIX/lib/python/site-packages. In that vtk/ directory it creates an empty stub __init__.py, a bunch of libvtk*dylib files, and a python3.8/site-packages directory which containts vtk.py and vtkmodules/. With this setup I can import vtk, but import vtk.util fails with a ModuleNotFound error. I’m pretty sure that if I manually move the files into the a directory structure identical to that created by the build-from-source process, it works, so the actual library and python files are probably OK. It just looks like the wheel building process appends an extra vtk/python3.8/site-packages to the path, or something like that.

Is anyone still interested in creating a wheel file for vtk with python 3.8 (as that old mailing list thread suggested people were at one point)? Does anyone know how the os x wheel-creating-script (VTKPythonPackage/scripts/macos_build_wheels.py) decides on the directory structure?

#2

VTK now contains wheel-building logic in its own master branch now (9.0 will have it too). This is done so that you can build any VTK you want with whichever modules you want. I need to write docs for how to use it, but I sketched the outlines of what is necessary in this post: https://discourse.vtk.org/t/vtkxdmf3writer-not-in-python-pypi-distribution/2470/2.

#3

Great, thanks. I hadn’t realized that make it into master. It took me a moment to find the VTK_WHEEL_BUILD option, but eventually I discovered advanced mode in the ccmake interface.

#4

No, it’s a CMake option. It’s not in the cache because it’s not really something meant to be turned on and off once the initial configure goes through since it’s effectively a completely separate build mode/structure (i.e., make install will not make anything useful).

#5

Oops, race condition during the comment edit process. Thanks again.

#6

I’ve now tested master, and it almost works. The compilation and installation process appears to work fine, but the resulting installation in ~/Library seems not quite right. If I try to just run python and import vtk, I get

>>> import vtk
Traceback (most recent call last):
  File "/Users/bernstei/Library/Python/3.8/lib/python/site-packages/vtkmodules/__init__.py", line 13, in <module>
    from . import vtkCommonCore
ImportError: dlopen(/Users/bernstei/Library/Python/3.8/lib/python/site-packages/vtkmodules/vtkCommonCore.cpython-38-darwin.so, 2): Library not loaded: @rpath/libvtkWrappingPythonCore.1.dylib
  Referenced from: /Users/bernstei/Library/Python/3.8/lib/python/site-packages/vtkmodules/vtkCommonCore.cpython-38-darwin.so
  Reason: image not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/bernstei/Library/Python/3.8/lib/python/site-packages/vtk.py", line 30, in <module>
    all_m = importlib.import_module('vtkmodules.all')
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/Users/bernstei/Library/Python/3.8/lib/python/site-packages/vtkmodules/__init__.py", line 15, in <module>
    import _vtkmodules_static
ModuleNotFoundError: No module named '_vtkmodules_static'

If I add ~/Library/Python/3.8/lib/python/site-packages/vtkmodules to DYLD_LIBRARY_PATH, it works well enough to run my application. I get the impression that ths isn’t supposed to be necessary. Should I open a github issue?

#7

A GitLab issue is better. Could you send the output of otool -l output on one of the Python modules? I’m interested in the value of the LC_RPATH load command. It should have an entry for something like @loader_path/../some/relative/path which ends up resolving to where the real libraries live.

#8

Here’s the output of otool -l (sorry, it won’t let me upload a file). Note that in the python 3.7 vtk 8.1 wheel the libraries were placed in site-packages/vtk along side __init__.py and vtkmodules/, while now (python 3.8m, vtk master) they’re placed in site-packages/vtkmodules/, one directory below vtk.py. I’ll also open a gitlab issue.

_SOME_HOME_DIR_/Library/Python/3.8/lib/python/site-packages/vtkmodules/libvtkDomainsChemistry.dylib:
Mach header
      magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
 0xfeedfacf 16777223          3  0x00           6    29       2640 0x00118085
Load command 0
      cmd LC_SEGMENT_64
  cmdsize 632
  segname __TEXT
   vmaddr 0x0000000000000000
   vmsize 0x000000000003c000
  fileoff 0
 filesize 245760
  maxprot 0x00000005
 initprot 0x00000005
   nsects 7
    flags 0x0
Section
  sectname __text
   segname __TEXT
      addr 0x0000000000005e50
      size 0x000000000002ccc2
    offset 24144
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x80000400
 reserved1 0
 reserved2 0
Section
  sectname __stubs
   segname __TEXT
      addr 0x0000000000032b12
      size 0x00000000000006b4
    offset 207634
     align 2^1 (2)
    reloff 0
    nreloc 0
     flags 0x80000408
 reserved1 0 (index into indirect symbol table)
 reserved2 6 (size of stubs)
Section
  sectname __stub_helper
   segname __TEXT
      addr 0x00000000000331c8
      size 0x0000000000000b00
    offset 209352
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x80000400
 reserved1 0
 reserved2 0
Section
  sectname __gcc_except_tab
   segname __TEXT
      addr 0x0000000000033cc8
      size 0x00000000000029c0
    offset 212168
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __const
   segname __TEXT
      addr 0x0000000000036690
      size 0x0000000000001eff
    offset 222864
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __cstring
   segname __TEXT
      addr 0x000000000003858f
      size 0x0000000000002ff1
    offset 230799
     align 2^0 (1)
    reloff 0
    nreloc 0
     flags 0x00000002
 reserved1 0
 reserved2 0
Section
  sectname __unwind_info
   segname __TEXT
      addr 0x000000000003b580
      size 0x0000000000000a80
    offset 243072
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Load command 1
      cmd LC_SEGMENT_64
  cmdsize 632
  segname __DATA
   vmaddr 0x000000000003c000
   vmsize 0x0000000000006000
  fileoff 245760
 filesize 24576
  maxprot 0x00000003
 initprot 0x00000003
   nsects 7
    flags 0x0
Section
  sectname __nl_symbol_ptr
   segname __DATA
      addr 0x000000000003c000
      size 0x0000000000000008
    offset 245760
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000006
 reserved1 286 (index into indirect symbol table)
 reserved2 0
Section
  sectname __got
   segname __DATA
      addr 0x000000000003c008
      size 0x00000000000000c0
    offset 245768
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000006
 reserved1 287 (index into indirect symbol table)
 reserved2 0
Section
  sectname __la_symbol_ptr
   segname __DATA
      addr 0x000000000003c0c8
      size 0x00000000000008f0
    offset 245960
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000007
 reserved1 311 (index into indirect symbol table)
 reserved2 0
Section
  sectname __mod_init_func
   segname __DATA
      addr 0x000000000003c9b8
      size 0x0000000000000088
    offset 248248
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000009
 reserved1 0
 reserved2 0
Section
  sectname __const
   segname __DATA
      addr 0x000000000003ca40
      size 0x0000000000004c60
    offset 248384
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __bss
   segname __DATA
      addr 0x00000000000416a0
      size 0x0000000000000040
    offset 0
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000001
 reserved1 0
 reserved2 0
Section
  sectname __common
   segname __DATA
      addr 0x00000000000416e0
      size 0x0000000000000008
    offset 0
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000001
 reserved1 0
 reserved2 0
Load command 2
      cmd LC_SEGMENT_64
  cmdsize 72
  segname __LINKEDIT
   vmaddr 0x0000000000042000
   vmsize 0x000000000001d000
  fileoff 270336
 filesize 115676
  maxprot 0x00000001
 initprot 0x00000001
   nsects 0
    flags 0x0
Load command 3
          cmd LC_ID_DYLIB
      cmdsize 64
         name @rpath/libvtkDomainsChemistry.1.dylib (offset 24)
   time stamp 1 Wed Dec 31 19:00:01 1969
      current version 9.0.0
compatibility version 1.0.0
Load command 4
            cmd LC_DYLD_INFO_ONLY
        cmdsize 48
     rebase_off 270336
    rebase_size 512
       bind_off 270848
      bind_size 11336
  weak_bind_off 282184
 weak_bind_size 1008
  lazy_bind_off 283192
 lazy_bind_size 13544
     export_off 296736
    export_size 10128
Load command 5
     cmd LC_SYMTAB
 cmdsize 24
  symoff 307672
   nsyms 1305
  stroff 330940
 strsize 55072
Load command 6
            cmd LC_DYSYMTAB
        cmdsize 80
      ilocalsym 0
      nlocalsym 602
     iextdefsym 602
     nextdefsym 271
      iundefsym 873
      nundefsym 432
         tocoff 0
           ntoc 0
      modtaboff 0
        nmodtab 0
   extrefsymoff 0
    nextrefsyms 0
 indirectsymoff 328552
  nindirectsyms 597
      extreloff 0
        nextrel 0
      locreloff 0
        nlocrel 0
Load command 7
     cmd LC_UUID
 cmdsize 24
    uuid 45B2B59A-956C-3BF0-B6A5-F64069AC6F07
Load command 8
       cmd LC_BUILD_VERSION
   cmdsize 32
  platform macos
       sdk 10.15
     minos 10.14
    ntools 1
      tool ld
   version 530.0
Load command 9
      cmd LC_SOURCE_VERSION
  cmdsize 16
  version 0.0
Load command 10
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name @rpath/libvtkIOLegacy.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 9.0.0
compatibility version 1.0.0
Load command 11
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name @rpath/libvtkIOXMLParser.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 9.0.0
compatibility version 1.0.0
Load command 12
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name @rpath/libvtkRenderingCore.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 9.0.0
compatibility version 1.0.0
Load command 13
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name @rpath/libvtkFiltersSources.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 9.0.0
compatibility version 1.0.0
Load command 14
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name @rpath/libvtkIOCore.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 9.0.0
compatibility version 1.0.0
Load command 15
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name @rpath/libvtkFiltersGeneral.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 9.0.0
compatibility version 1.0.0
Load command 16
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name @rpath/libvtkFiltersCore.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 9.0.0
compatibility version 1.0.0
Load command 17
          cmd LC_LOAD_DYLIB
      cmdsize 72
         name @rpath/libvtkCommonExecutionModel.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 9.0.0
compatibility version 1.0.0
Load command 18
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name @rpath/libvtkCommonDataModel.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 9.0.0
compatibility version 1.0.0
Load command 19
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name @rpath/libvtkCommonTransforms.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 9.0.0
compatibility version 1.0.0
Load command 20
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name @rpath/libvtkCommonMisc.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 9.0.0
compatibility version 1.0.0
Load command 21
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name @rpath/libvtkCommonMath.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 9.0.0
compatibility version 1.0.0
Load command 22
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name @rpath/libvtkCommonCore.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 9.0.0
compatibility version 1.0.0
Load command 23
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name @rpath/libvtksys.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 9.0.0
compatibility version 1.0.0
Load command 24
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name /usr/lib/libc++.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 800.7.0
compatibility version 1.0.0
Load command 25
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/lib/libSystem.B.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 1281.0.0
compatibility version 1.0.0
Load command 26
          cmd LC_RPATH
      cmdsize 24
         path $ORIGIN (offset 12)
Load command 27
      cmd LC_FUNCTION_STARTS
  cmdsize 16
  dataoff 306864
 datasize 792
Load command 28
      cmd LC_DATA_IN_CODE
  cmdsize 16
  dataoff 307656
 datasize 16
#9

Well, this indicates that something is doing rpath logic, but not special casing macOS rpath logic and instead using typical ELF rpath placeholders instead.

#10

gitlab issue. I’m happy to test any possible fixes. I’ll try to keep an eye on both the issue and this discourse topic.