What do you mean by “a custom C++ vtk function”? VTK is mainly a library of C++ classes and those classes are wrapped in Python using a builtin mechanism (not SWIG) which is driven by a set of CMake macros. If you develop some new VTK-derived classes, you should be able to easily wrap them as well if your classes are organised as a custom VTK module. Does it make sense to you?