Python IDE autocompletion for upcoming VTK 9.1

Hi all, just today I merged a patch into VTK-master to improve the vtk-python method descriptors and docstrings. This should help autocompletion and hinting for ipython and jupyter. Anyone who wants to try it out, please do so, because VTK 9.1 is coming soon and I won’t have time to do additional testing myself.

if you have an IDE that uses .pyi files, there is a script that you can run to generate them. Note that this script will only work with the current master branch, it will not work with yesterday’s master or with VTK 9.0.3.

@amaclean since you’re a PyCharm user, is there a chance that you can try this script to see if it makes any difference to the user experience? When run with no arguments, the script will simply deposit the .pyi files next to the .pyd files.

3 Likes

Used Linux and it seems to work nicely. I ran it with no parameters. One suggestion, it might be good to mention that if the environment is not set up properly call it like this:

vtkpython ./vtk_generate_pyi.py

As regards PyCharm, it doesn seem to make any difference. PyCharm seems to do this when it loads the modules. However PyCharm is aware of the files because when I run vtk_generate_pyi.py it reindexes and also when these files are deleted.

Thanks for giving it a spin, Andrew. If the .pyi files cause PyCharm to reindex, then at least I know that it’s pulling information from them. And some of that information (e.g. @overload hints) is only available if the .pyi files are present.

@phcerdan you mentioned issues with vtk-python autocompletion in an earlier thread, If you can try the current master (or vtk-9.1-rc1 when it comes out in a few days), any feedback you can give would be greatly appreciated.

1 Like

Thanks for the work @dgobbi!

Fast check: I am using pyright (language server) in vim trough the plugin coc.vim.
I installed vtk in a python virtual environment from a wheel generated from the build tree of current master. And then I used the script to generate the stub pyi files.

I am able to get the info from .pyi stubs files in the editor. However I have to specify
the vtkmodules path.

If I import the whole vtk package import vtk I can only access inner python methods

image

However if I import using the vtkmodules path it works flawlessly.

image

It works really well in windows and PyCharm.

It even resolves this one (only happens in Windows):

  • from vtkmodules.vtkInteractionWidgets import vtkCameraOrientationWidget
    Cannot find reference 'vtkCameraOrientationWidget' in '__init__.py'

This is the cleanest code inspection for VTK in PyCharm that I have seen!

Next question:
Will we have to run this every time we build VTK?

Thanks, Andrew. One of the features I’m hoping people will find useful is the type hints, particularly for overloaded methods:

overloads
This is the feature that I’m most interested in people testing on the various IDEs and Python consoles.

I strongly suspect that the ‘vtkCameraOrientationWidget’ error is due to leftovers on your system from an old build or something like that. The fact that it mentions an __init__.py makes me think your system might have a vtkInteractionWidgets folder somewhere that it doesn’t belong.

I do not have any immediate plans to add the .pyi files to the VTK build, but wouldn’t object if someone else did so. I’ve found that ipython, jupyter, and the default Python console actually do fairly well just with the updated docstrings in the wrappers, and for them the .pyi files don’t seem to add much benefit (though I would be interested to hear from others on the subject).

@phcerdan thanks for giving it a spin. Were you able to get any hinting for method argument types?

The script only generates vtkmodules .pyi files, it would need to be modified to generate a vtk.pyi (and the file would be very large). It’s something I could do, but I’m not convinced that it would be a good idea.

Thanks a lot @dgobbi! This is awesome already. If at some point this could be available directly after pip installing vtk it would be great. /PyCharm user

For those of you using import vtk you can easily start using the vtkmodules files. I have a script that looks through your code and provides you with a list of the necessary from ... import ... statements.

Please see VTKImportsForPython, and ignore the comment about sorting. It needs testing as I am not sure that I capture all of the VTK classes since pattern matching is used.

Also there is a script VTKModulesForCxx that will generate a find_package(VTK COMPONENTS ...) command for CMake, useful for Cxx development.

Both these scripts have some minor changes coming through (tomorrow?).

2 Likes

Yep!

I would prefer to add the stubs directly to the vtk package. But as an alternative, another optional package vtk-stubs, vtk-pyi or similar name, can be created. The only inconvenient is that vtk and vtk-stubs might get out of sync with each other when updating vtk but not vtk-stubs.

The setup.py.in file seems the place to trigger the stubs script after the build? I am not familiar with vtk python packaging.

My inclination would be to trigger the script with cmake during the build, rather than with setup.py after the build. Putting the .pyi files in the build tree would allow us to add a test for them (e.g. the test could load them into Python to check the syntax, and could check for unresolved types, etc). But I don’t think we’re quite there yet.

1 Like

Also, for people who are using “import vtk”, you might have better luck with this instead:

import vtkmodules.all as vtk

That should be sufficient for PyCharm to take the “vtk” hints from “vtkmodules”. I don’t know whether it will work for other IDEs, however, since every IDE does hints differently.

1 Like

Thanks @dgobbi, it works really well in PyCharm, as soon as you type vtk. you get a dropdown list of all the classes etc. and the modules they live in! It’s really fast too. Love it, thanks for the tip!

I’ve been crazy busy and haven’t followed what’s going on with Python, sorry about that.

This morning I did a clean master pull / VTK build to test a new filter. I found that the way I’ve been testing/debugging Python test scripts for the last many years no longer works. For example:

~/Develop/Bisect/debug/bin/vtkpython TestExtractPoints.py

Gives me a bunch of import errors. As a member of the uninformed, unwashed masses, let me ask: what do I need to do to get this working again?

Hi Will, as long as your version of Python is somewhere between 3.6 and 3.9, running vtkpython as you have indicated should still work just fine. That’s how I usually run my test scripts, too. Most of the vtk-python tests haven’t changed in 5 years or more, so backwards compatibility should be pretty good.

What was the text of the error?

My python version is 3.6 on Ubuntu.

Load errors are as follows:

Traceback (most recent call last):
File “TestExtractPoints.py”, line 2, in
import vtk
File “/home/will/Develop/Bisect/release/lib/python3.6/site-packages/vtk.py”, line 31, in
all_m = importlib.import_module(‘vtkmodules.all’)
File “/usr/local/lib/python3.6/importlib/init.py”, line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “/home/will/Develop/Bisect/release/lib/python3.6/site-packages/vtkmodules/all.py”, line 7, in
from .vtkCommonCore import *
ImportError: /home/will/Develop/Bisect/release/lib/python3.6/site-packages/vtkmodules/vtkCommonCore.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PyExc_IndexError

This error means that libpython (libpython3.6.so) is not being linked, or perhaps there is a symbol visibility problem.
@ben.boeckel do you have ideas about what might cause this?

I suspect the platform is not working with VTK_PYTHON_OPTIONAL_LINK in some way. Changing that option might help?

I unchecked that option (VTK_PYTHON_OPTIONAL_LINK) and rebuilt. VTK then failed to build:

[1/122] Linking CXX shared library lib/libvtkWrappingPythonCore3.6-9.0.so.9.0.0
FAILED: lib/libvtkWrappingPythonCore3.6-9.0.so.9.0.0
: && /usr/bin/c++ -fPIC -O3 -DNDEBUG -Wl,-lc -shared -Wl,-soname,libvtkWrappingPythonCore3.6-9.0.so.1 -o lib/libvtkWrappingPythonCore3.6-9.0.so.9.0.0 Wrapping/PythonCore/CMakeFiles/WrappingPythonCore.dir/PyVTKEnum.cxx.o Wrapping/PythonCore/CMakeFiles/WrappingPythonCore.dir/PyVTKExtras.cxx.o Wrapping/PythonCore/CMakeFiles/WrappingPythonCore.dir/PyVTKNamespace.cxx.o Wrapping/PythonCore/CMakeFiles/WrappingPythonCore.dir/PyVTKObject.cxx.o Wrapping/PythonCore/CMakeFiles/WrappingPythonCore.dir/PyVTKSpecialObject.cxx.o Wrapping/PythonCore/CMakeFiles/WrappingPythonCore.dir/PyVTKTemplate.cxx.o Wrapping/PythonCore/CMakeFiles/WrappingPythonCore.dir/vtkPythonArgs.cxx.o Wrapping/PythonCore/CMakeFiles/WrappingPythonCore.dir/vtkPythonCommand.cxx.o Wrapping/PythonCore/CMakeFiles/WrappingPythonCore.dir/vtkPythonOverload.cxx.o Wrapping/PythonCore/CMakeFiles/WrappingPythonCore.dir/vtkPythonUtil.cxx.o Wrapping/PythonCore/CMakeFiles/WrappingPythonCore.dir/vtkSmartPyObject.cxx.o Wrapping/PythonCore/CMakeFiles/WrappingPythonCore.dir/PyVTKMethodDescriptor.cxx.o Wrapping/PythonCore/CMakeFiles/WrappingPythonCore.dir/PyVTKReference.cxx.o -Wl,-rpath,/home/will/Develop/Bisect/release/lib: lib/libvtkCommonCore-9.0.so.9.0.0 lib/libvtksys-9.0.so.9.0.0 -ldl /usr/lib/x86_64-linux-gnu/libtbb.so.2 /usr/local/lib/libpython3.6m.a -lpthread -ldl -lutil -Wl,-rpath-link,/home/will/Develop/Bisect/release/lib && :
/usr/bin/ld: /usr/local/lib/libpython3.6m.a(pystate.o): warning: relocation against _PyGILState_check_enabled' in read-only section .text’
/usr/bin/ld: /usr/local/lib/libpython3.6m.a(compile.o): relocation R_X86_64_PC32 against symbol `PyExc_SyntaxError’ can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.