Hi - I’ve been trying to figure out how to get vtk to work with python 3.8, and in particular in making a wheel for easy installation. The basic vtk functionality works when compiling from source (using the patch from https://gitlab.kitware.com/vtk/vtk/uploads/cf7f517b4f647de1997c4e8cddb54de9/0001-Compatibility-for-Python-3.8.patch), including the python wrapping module which allows my python application to work. I was also able to modify the wheel building infrastructure from https://github.com/KitwareMedical/VTKPythonPackage and got it to compile and apparently install. The changes were minimal (and I’d be happy to post the patch if it’s useful), mostly applying the source patch above, changing venv to python3 -m venv, and changing some paths and version numbers. I also created a dummy libpython-not-needed-symbols-exported-by-interpreter that is actually a library, not an empty file, to deal with the problems mentioned in http://vtk.1045678.n5.nabble.com/vtk-8-2-0-rc2-problem-building-wheels-td5749301.html.
This process creates a wheel that I can install with pip, but the resulting directory structure is very weird. When I directly install from source I get a bunch of libvtk*.so files in PREFIX/lib, and vtk.py and vtkmodules in PREFIX/lib/python3.8/site-packages. If I add those to DYLD_LIBRARY_PATH and PYTHONPATH things works fine. However, the wheel (which is installing to the Mac OS user directory PREFIX=~/Library/Python/3.8), creates a vtk/ directory in PREFIX/lib/python/site-packages. In that vtk/ directory it creates an empty stub __init__.py, a bunch of libvtk*dylib files, and a python3.8/site-packages directory which containts vtk.py and vtkmodules/. With this setup I can import vtk, but import vtk.util fails with a ModuleNotFound error. I’m pretty sure that if I manually move the files into the a directory structure identical to that created by the build-from-source process, it works, so the actual library and python files are probably OK. It just looks like the wheel building process appends an extra vtk/python3.8/site-packages to the path, or something like that.
Is anyone still interested in creating a wheel file for vtk with python 3.8 (as that old mailing list thread suggested people were at one point)? Does anyone know how the os x wheel-creating-script (VTKPythonPackage/scripts/macos_build_wheels.py) decides on the directory structure?
VTK now contains wheel-building logic in its own master branch now (9.0 will have it too). This is done so that you can build any VTK you want with whichever modules you want. I need to write docs for how to use it, but I sketched the outlines of what is necessary in this post: https://discourse.vtk.org/t/vtkxdmf3writer-not-in-python-pypi-distribution/2470/2.
Great, thanks. I hadn’t realized that make it into master. It took me a moment to find the VTK_WHEEL_BUILD option, but eventually I discovered advanced mode in the ccmake interface.
No, it’s a CMake option. It’s not in the cache because it’s not really something meant to be turned on and off once the initial configure goes through since it’s effectively a completely separate build mode/structure (i.e., make install will not make anything useful).
I’ve now tested master, and it almost works. The compilation and installation process appears to work fine, but the resulting installation in ~/Library seems not quite right. If I try to just run python and import vtk, I get
>>> import vtk
Traceback (most recent call last):
File "/Users/bernstei/Library/Python/3.8/lib/python/site-packages/vtkmodules/__init__.py", line 13, in <module>
from . import vtkCommonCore
ImportError: dlopen(/Users/bernstei/Library/Python/3.8/lib/python/site-packages/vtkmodules/vtkCommonCore.cpython-38-darwin.so, 2): Library not loaded: @rpath/libvtkWrappingPythonCore.1.dylib
Referenced from: /Users/bernstei/Library/Python/3.8/lib/python/site-packages/vtkmodules/vtkCommonCore.cpython-38-darwin.so
Reason: image not found
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/bernstei/Library/Python/3.8/lib/python/site-packages/vtk.py", line 30, in <module>
all_m = importlib.import_module('vtkmodules.all')
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/Users/bernstei/Library/Python/3.8/lib/python/site-packages/vtkmodules/__init__.py", line 15, in <module>
import _vtkmodules_static
ModuleNotFoundError: No module named '_vtkmodules_static'
If I add ~/Library/Python/3.8/lib/python/site-packages/vtkmodules to DYLD_LIBRARY_PATH, it works well enough to run my application. I get the impression that ths isn’t supposed to be necessary. Should I open a github issue?
A GitLab issue is better. Could you send the output of otool -l output on one of the Python modules? I’m interested in the value of the LC_RPATH load command. It should have an entry for something like @loader_path/../some/relative/path which ends up resolving to where the real libraries live.
Here’s the output of otool -l (sorry, it won’t let me upload a file). Note that in the python 3.7 vtk 8.1 wheel the libraries were placed in site-packages/vtk along side __init__.py and vtkmodules/, while now (python 3.8m, vtk master) they’re placed in site-packages/vtkmodules/, one directory below vtk.py. I’ll also open a gitlab issue.
_SOME_HOME_DIR_/Library/Python/3.8/lib/python/site-packages/vtkmodules/libvtkDomainsChemistry.dylib:
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
0xfeedfacf 16777223 3 0x00 6 29 2640 0x00118085
Load command 0
cmd LC_SEGMENT_64
cmdsize 632
segname __TEXT
vmaddr 0x0000000000000000
vmsize 0x000000000003c000
fileoff 0
filesize 245760
maxprot 0x00000005
initprot 0x00000005
nsects 7
flags 0x0
Section
sectname __text
segname __TEXT
addr 0x0000000000005e50
size 0x000000000002ccc2
offset 24144
align 2^4 (16)
reloff 0
nreloc 0
flags 0x80000400
reserved1 0
reserved2 0
Section
sectname __stubs
segname __TEXT
addr 0x0000000000032b12
size 0x00000000000006b4
offset 207634
align 2^1 (2)
reloff 0
nreloc 0
flags 0x80000408
reserved1 0 (index into indirect symbol table)
reserved2 6 (size of stubs)
Section
sectname __stub_helper
segname __TEXT
addr 0x00000000000331c8
size 0x0000000000000b00
offset 209352
align 2^2 (4)
reloff 0
nreloc 0
flags 0x80000400
reserved1 0
reserved2 0
Section
sectname __gcc_except_tab
segname __TEXT
addr 0x0000000000033cc8
size 0x00000000000029c0
offset 212168
align 2^2 (4)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Section
sectname __const
segname __TEXT
addr 0x0000000000036690
size 0x0000000000001eff
offset 222864
align 2^4 (16)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Section
sectname __cstring
segname __TEXT
addr 0x000000000003858f
size 0x0000000000002ff1
offset 230799
align 2^0 (1)
reloff 0
nreloc 0
flags 0x00000002
reserved1 0
reserved2 0
Section
sectname __unwind_info
segname __TEXT
addr 0x000000000003b580
size 0x0000000000000a80
offset 243072
align 2^2 (4)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Load command 1
cmd LC_SEGMENT_64
cmdsize 632
segname __DATA
vmaddr 0x000000000003c000
vmsize 0x0000000000006000
fileoff 245760
filesize 24576
maxprot 0x00000003
initprot 0x00000003
nsects 7
flags 0x0
Section
sectname __nl_symbol_ptr
segname __DATA
addr 0x000000000003c000
size 0x0000000000000008
offset 245760
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000006
reserved1 286 (index into indirect symbol table)
reserved2 0
Section
sectname __got
segname __DATA
addr 0x000000000003c008
size 0x00000000000000c0
offset 245768
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000006
reserved1 287 (index into indirect symbol table)
reserved2 0
Section
sectname __la_symbol_ptr
segname __DATA
addr 0x000000000003c0c8
size 0x00000000000008f0
offset 245960
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000007
reserved1 311 (index into indirect symbol table)
reserved2 0
Section
sectname __mod_init_func
segname __DATA
addr 0x000000000003c9b8
size 0x0000000000000088
offset 248248
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000009
reserved1 0
reserved2 0
Section
sectname __const
segname __DATA
addr 0x000000000003ca40
size 0x0000000000004c60
offset 248384
align 2^4 (16)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Section
sectname __bss
segname __DATA
addr 0x00000000000416a0
size 0x0000000000000040
offset 0
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000001
reserved1 0
reserved2 0
Section
sectname __common
segname __DATA
addr 0x00000000000416e0
size 0x0000000000000008
offset 0
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000001
reserved1 0
reserved2 0
Load command 2
cmd LC_SEGMENT_64
cmdsize 72
segname __LINKEDIT
vmaddr 0x0000000000042000
vmsize 0x000000000001d000
fileoff 270336
filesize 115676
maxprot 0x00000001
initprot 0x00000001
nsects 0
flags 0x0
Load command 3
cmd LC_ID_DYLIB
cmdsize 64
name @rpath/libvtkDomainsChemistry.1.dylib (offset 24)
time stamp 1 Wed Dec 31 19:00:01 1969
current version 9.0.0
compatibility version 1.0.0
Load command 4
cmd LC_DYLD_INFO_ONLY
cmdsize 48
rebase_off 270336
rebase_size 512
bind_off 270848
bind_size 11336
weak_bind_off 282184
weak_bind_size 1008
lazy_bind_off 283192
lazy_bind_size 13544
export_off 296736
export_size 10128
Load command 5
cmd LC_SYMTAB
cmdsize 24
symoff 307672
nsyms 1305
stroff 330940
strsize 55072
Load command 6
cmd LC_DYSYMTAB
cmdsize 80
ilocalsym 0
nlocalsym 602
iextdefsym 602
nextdefsym 271
iundefsym 873
nundefsym 432
tocoff 0
ntoc 0
modtaboff 0
nmodtab 0
extrefsymoff 0
nextrefsyms 0
indirectsymoff 328552
nindirectsyms 597
extreloff 0
nextrel 0
locreloff 0
nlocrel 0
Load command 7
cmd LC_UUID
cmdsize 24
uuid 45B2B59A-956C-3BF0-B6A5-F64069AC6F07
Load command 8
cmd LC_BUILD_VERSION
cmdsize 32
platform macos
sdk 10.15
minos 10.14
ntools 1
tool ld
version 530.0
Load command 9
cmd LC_SOURCE_VERSION
cmdsize 16
version 0.0
Load command 10
cmd LC_LOAD_DYLIB
cmdsize 56
name @rpath/libvtkIOLegacy.1.dylib (offset 24)
time stamp 2 Wed Dec 31 19:00:02 1969
current version 9.0.0
compatibility version 1.0.0
Load command 11
cmd LC_LOAD_DYLIB
cmdsize 64
name @rpath/libvtkIOXMLParser.1.dylib (offset 24)
time stamp 2 Wed Dec 31 19:00:02 1969
current version 9.0.0
compatibility version 1.0.0
Load command 12
cmd LC_LOAD_DYLIB
cmdsize 64
name @rpath/libvtkRenderingCore.1.dylib (offset 24)
time stamp 2 Wed Dec 31 19:00:02 1969
current version 9.0.0
compatibility version 1.0.0
Load command 13
cmd LC_LOAD_DYLIB
cmdsize 64
name @rpath/libvtkFiltersSources.1.dylib (offset 24)
time stamp 2 Wed Dec 31 19:00:02 1969
current version 9.0.0
compatibility version 1.0.0
Load command 14
cmd LC_LOAD_DYLIB
cmdsize 56
name @rpath/libvtkIOCore.1.dylib (offset 24)
time stamp 2 Wed Dec 31 19:00:02 1969
current version 9.0.0
compatibility version 1.0.0
Load command 15
cmd LC_LOAD_DYLIB
cmdsize 64
name @rpath/libvtkFiltersGeneral.1.dylib (offset 24)
time stamp 2 Wed Dec 31 19:00:02 1969
current version 9.0.0
compatibility version 1.0.0
Load command 16
cmd LC_LOAD_DYLIB
cmdsize 64
name @rpath/libvtkFiltersCore.1.dylib (offset 24)
time stamp 2 Wed Dec 31 19:00:02 1969
current version 9.0.0
compatibility version 1.0.0
Load command 17
cmd LC_LOAD_DYLIB
cmdsize 72
name @rpath/libvtkCommonExecutionModel.1.dylib (offset 24)
time stamp 2 Wed Dec 31 19:00:02 1969
current version 9.0.0
compatibility version 1.0.0
Load command 18
cmd LC_LOAD_DYLIB
cmdsize 64
name @rpath/libvtkCommonDataModel.1.dylib (offset 24)
time stamp 2 Wed Dec 31 19:00:02 1969
current version 9.0.0
compatibility version 1.0.0
Load command 19
cmd LC_LOAD_DYLIB
cmdsize 64
name @rpath/libvtkCommonTransforms.1.dylib (offset 24)
time stamp 2 Wed Dec 31 19:00:02 1969
current version 9.0.0
compatibility version 1.0.0
Load command 20
cmd LC_LOAD_DYLIB
cmdsize 56
name @rpath/libvtkCommonMisc.1.dylib (offset 24)
time stamp 2 Wed Dec 31 19:00:02 1969
current version 9.0.0
compatibility version 1.0.0
Load command 21
cmd LC_LOAD_DYLIB
cmdsize 56
name @rpath/libvtkCommonMath.1.dylib (offset 24)
time stamp 2 Wed Dec 31 19:00:02 1969
current version 9.0.0
compatibility version 1.0.0
Load command 22
cmd LC_LOAD_DYLIB
cmdsize 56
name @rpath/libvtkCommonCore.1.dylib (offset 24)
time stamp 2 Wed Dec 31 19:00:02 1969
current version 9.0.0
compatibility version 1.0.0
Load command 23
cmd LC_LOAD_DYLIB
cmdsize 56
name @rpath/libvtksys.1.dylib (offset 24)
time stamp 2 Wed Dec 31 19:00:02 1969
current version 9.0.0
compatibility version 1.0.0
Load command 24
cmd LC_LOAD_DYLIB
cmdsize 48
name /usr/lib/libc++.1.dylib (offset 24)
time stamp 2 Wed Dec 31 19:00:02 1969
current version 800.7.0
compatibility version 1.0.0
Load command 25
cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libSystem.B.dylib (offset 24)
time stamp 2 Wed Dec 31 19:00:02 1969
current version 1281.0.0
compatibility version 1.0.0
Load command 26
cmd LC_RPATH
cmdsize 24
path $ORIGIN (offset 12)
Load command 27
cmd LC_FUNCTION_STARTS
cmdsize 16
dataoff 306864
datasize 792
Load command 28
cmd LC_DATA_IN_CODE
cmdsize 16
dataoff 307656
datasize 16
Well, this indicates that something is doing rpath logic, but not special casing macOS rpath logic and instead using typical ELF rpath placeholders instead.