# Remote modules vs external modules
**URL:** https://discourse.vtk.org/t/remote-modules-vs-external-modules/2003
**Category:** Development
**Created:** [October 29, 2019, 6:21pm UTC](https://discourse.vtk.org/t/remote-modules-vs-external-modules/2003 "2019-10-29T18:21:30Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@dgobbi](https://discourse.vtk.org/u/dgobbi)
#### Post date: [October 29, 2019, 6:21pm UTC](https://discourse.vtk.org/t/remote-modules-vs-external-modules/2003/1 "2019-10-29T18:21:30Z")
Over the weekend, I started updating vtk-dicom to the new VTK 9 build system.
For previous versions of VTK, it was possible to build vtk-dicom either as part of the VTK build (i.e. as a remote module, thanks to Bill Lorensen), or as an external package against an existing VTK build. External builds have always been challenging due to the fact that, historically, the VTK cmake commands for wrapping are neither streamlined nor stable between VTK versions.
I plan to continue supporting both remote and external builds of vtk-dicom for VTK 9. But there seems to be a hitch. For remote modules, `vtk.module` must be in the base directory of the repository, but for external packages, `vtk.module` must be in a subdirectory (so that it can be found by the main `CMakeLists.txt` for the package). I can solve this problem by having two identical copies of `vtk.module`, one in the base directory and another in the directory that contains the source code for the module. But it’s a rather ugly solution.
The best solution would probably be to expand the idea of “remote module” to also allow for “remote packages”, where a package can contain one or more modules (e.g. as subdirectories). For vtk-dicom, at least, I think it would make it easier for me to maintain things moving forward.
Does anyone have opinions on this?
---
### Author:  [@lassoan](https://discourse.vtk.org/u/lassoan)
#### Post date: [October 29, 2019, 6:36pm UTC](https://discourse.vtk.org/t/remote-modules-vs-external-modules/2003/2 "2019-10-29T18:36:14Z")
I don’t know what is the best approach to implement this, but I can confirm that building VTK remote modules after VTK was already built, in a separate build tree is a very important use case for 3D Slicer extensions, too.
For example, in SlicerVirtualReality extension builds VTKRenderingOpenVR VTK remote module after VTK has been already built, in a separate build tree.
This is the main reason we keep backporting essential VTK patches instead of switching to a recent VTK master.
---
### Author:  [@dgobbi](https://discourse.vtk.org/u/dgobbi)
#### Post date: [October 29, 2019, 7:48pm UTC](https://discourse.vtk.org/t/remote-modules-vs-external-modules/2003/3 "2019-10-29T19:48:15Z")
Hi Andas, I’m looking at the cmake files for SlicerVirtualReality, and I can’t figure out where it gets VTKRenderingOpenVR from. It’s marked as an external project, but it doesn’t seem to be downloaded, nor is it inside the SlicerVirtualReality repository. Where is it located?
---
### Author:  [@ben.boeckel](https://discourse.vtk.org/u/ben.boeckel)
#### Post date: [October 29, 2019, 8:23pm UTC](https://discourse.vtk.org/t/remote-modules-vs-external-modules/2003/4 "2019-10-29T20:23:15Z")
> [@dgobbi](#):
>
> For remote modules, `vtk.module` must be in the base directory of the repository
I don’t think that’s true? We recursively glob for the file.
> [@lassoan](#):
>
> I can confirm that building VTK remote modules after VTK was already built, in a separate build tree is a very important use case for 3D Slicer extensions, too.
I’d prefer to see these stay as external modules (possibly with contract testing in VTK).
> [@lassoan](#):
>
> For example, in SlicerVirtualReality extension builds VTKRenderingOpenVR VTK remote module after VTK has been already built, in a separate build tree.
I consider this an unsupported configuration; you’ll need to mangle your `VTK::RenderingOpenVR` module if VTK isn’t building it (or accept the one VTK came with).
---
### Author:  [@dgobbi](https://discourse.vtk.org/u/dgobbi)
#### Post date: [October 29, 2019, 8:41pm UTC](https://discourse.vtk.org/t/remote-modules-vs-external-modules/2003/5 "2019-10-29T20:41:42Z")
> [@ben.boeckel](#):
>
> I don’t think that’s true? We recursively glob for the file.
If so, that makes things easier. Then I can arrange things so that the CMakeLists.txt in my base directory is only used when building as an external project.
---
### Author:  [@lassoan](https://discourse.vtk.org/u/lassoan)
#### Post date: [October 29, 2019, 10:16pm UTC](https://discourse.vtk.org/t/remote-modules-vs-external-modules/2003/6 "2019-10-29T22:16:15Z")
> [@ben.boeckel](#):
>
> I consider this an unsupported configuration; you’ll need to mangle your `VTK::RenderingOpenVR` module if VTK isn’t building it (or accept the one VTK came with).
At the time VTK is built, we don’t know what remote modules various extensions will need.
How do you mangle a module?
---
### Author:  [@ben.boeckel](https://discourse.vtk.org/u/ben.boeckel)
#### Post date: [October 30, 2019, 1:09pm UTC](https://discourse.vtk.org/t/remote-modules-vs-external-modules/2003/7 "2019-10-30T13:09:34Z")
If HDF5 is built without MPI, do you go and build its MPI bits if you need them? No, you error saying “the HDF5 you gave me is not suitable for what I need”. Why are we trying to make VTK so different?
As for proper mangling, things that need changed:
- module name (something like `ITK::VTKRenderingOpenVR`)
- symbol names (a mechanism will need to be added for this)
- header locations, library name (controllable through `vtk_module_build` already)
---
### Author:  [@lassoan](https://discourse.vtk.org/u/lassoan)
#### Post date: [October 30, 2019, 2:32pm UTC](https://discourse.vtk.org/t/remote-modules-vs-external-modules/2003/8 "2019-10-30T14:32:36Z")
I agree that it is not reasonable to change VTK build options externally (e.g., enable Qt or change the rendering backend). What we need is to be able to leverage VTK’s build system to build VTK-based external libraries both as part of VTK build; and after VTK build as a separate library.
It is of course possible to implement handling of different build modes in each remote module, as it is done in [vtkDICOM](https://github.com/dgobbi/vtk-dicom/blob/master/CMakeLists.txt#L32-L171). However, that would mean copy-pasting, slightly modifying, and maintaining significant amount of non-trival CMake code between all remote modules.
Instead, it would be much better if VTK would contain common CMake files that all remote modules could include. It would simplify the work of both remote module developers (they would need to develop and maintain less CMake code) and VTK developers (as build system of remote modules would be more uniform and they could be tuned all at once by modifying scripts in the main VTK repository).
OpenVR may be a special case, because it is a remote module stored in the VTK source tree (at least this was the status before the large VTK build system rework). OpenVR may have to be converted to a proper remote module or a regular local VTK module.
---
### Author:  [@ben.boeckel](https://discourse.vtk.org/u/ben.boeckel)
#### Post date: [October 30, 2019, 2:36pm UTC](https://discourse.vtk.org/t/remote-modules-vs-external-modules/2003/9 "2019-10-30T14:36:46Z")
> [@lassoan](#):
>
> What we need is to be able to leverage VTK’s build system to build VTK-based external libraries both as part of VTK build; and after VTK build as a separate library.
Why does it need to support being built inside VTK? Why not always just do `find_package(VTK)` just like any other dependency? I’d prefer these specialized modules become separate repositories that rely on VTK just like VTK relies on Qt: an external dependency.
---
### Author:  [@lassoan](https://discourse.vtk.org/u/lassoan)
#### Post date: [October 30, 2019, 3:56pm UTC](https://discourse.vtk.org/t/remote-modules-vs-external-modules/2003/10 "2019-10-30T15:56:08Z")
> [@ben.boeckel](#):
>
> Why does it need to support being built inside VTK?
By building inside VTK I mean that developers can add remote modules to VTK by changing CMake options when configuring VTK. There are many reasons why it is useful:
- convenience feature for developers: many small projects don’t even use superbuild to build dependencies, so cutting down number of libraries they have to configure manually is a big deal
- flexibility for VTK maintainers: if VTK maintainers decide that they want to reduce the VTK tree size or don’t want to maintain some VTK modules anymore then modules can be moved to external repositories without impacting projects that rely on that VTK module
- dependency between modules can be automatically handled (e.g., if remote modules depend on other remote modules)
> [@ben.boeckel](#):
>
> I’d prefer these specialized modules become separate repositories that rely on VTK just like VTK relies on Qt: an external dependency
Do you mean that you would prefer if remote modules infrastructure would go away entirely? That would not be good for anyone, as it would mean less flexibility and increased maintenance burden for both VTK and remote module developers.
---
### Author:  [@ben.boeckel](https://discourse.vtk.org/u/ben.boeckel)
#### Post date: [October 30, 2019, 6:16pm UTC](https://discourse.vtk.org/t/remote-modules-vs-external-modules/2003/11 "2019-10-30T18:16:37Z")
> [@lassoan](#):
>
> By building inside VTK I mean that developers can add remote modules to VTK by changing CMake options when configuring VTK.
With the new system, VTK’s options have documented behaviors. I _really_ don’t want to even allow remote modules to interfere with such things. Being inside the build is too close; it’s basically an implicit `friend` declaration for VTK’s own build system. It was not designed with such access in mind.
> [@lassoan](#):
>
> convenience feature for developers: many small projects don’t even use superbuild to build dependencies, so cutting down number of libraries they have to configure manually is a big deal
We’re planning on providing packages built by VTK more often starting with 9.0. It should be as easy as `pip install https://vtk.org/...` once that happens for Python. Not sure about C++ yet; maybe Anaconda, `.dpkg`, and `.rpm` packages is good enough there.
> [@lassoan](#):
>
> flexibility for VTK maintainers: if VTK maintainers decide that they want to reduce the VTK tree size or don’t want to maintain some VTK modules anymore then modules can be moved to external repositories without impacting projects that rely on that VTK module
If that were happening, I’d do (for example):
```cmake
find_package(vtkio REQUIRED)
add_library(VTK::IOXML INTERFACE)
target_link_libraries(VTK::IOXML INTERFACE vtkio::xml)
```
Not build it as part of VTK.
> [@lassoan](#):
>
> dependency between modules can be automatically handled (e.g., if remote modules depend on other remote modules)
More `find_package`. Just like every other project out there. You’re already going to have to support that otherwise you’re forcing _everyone_ to build a custom VTK to build your project (since it’s not going into any of the packages that are part of the packaging mentioned above). That’s not nice either.
> [@lassoan](#):
>
> That would not be good for anyone, as it would mean less flexibility and increased maintenance burden for both VTK and remote module developers.
Yes, I, personally, want it to go away. I’ve yet to hear an argument for why it should be possible that isn’t basically “that sounds like a lot of work”. And before, I could see why. But now that the API VTK uses is just as usable outside of it, that sounds fine to me. If you use the module system, it even helps you write your package configuration file for your dependencies! It can’t write your top-level config file, but it does what it can. Plus, what VTK doesn’t export into its install tree for is explicitly not part of any API guarantee and I won’t care if it ends up breaking a remote module.
As for burdens, it’d mean _less_ for VTK. `find_package(VTK)` has to work _anyways_, so that’s not a problem. It also means we don’t have to halt development when remote modules don’t follow VTK closely enough (new testing machines being broken with remote modules, having remote modules rely on accidentally-public stuff that is only accessible in a build, etc.). Yes, it’d mean more for remote module developers, but why is VTK so different than HDF5 or any of your other dependencies?
---
### Author:  [@dgobbi](https://discourse.vtk.org/u/dgobbi)
#### Post date: [October 30, 2019, 7:50pm UTC](https://discourse.vtk.org/t/remote-modules-vs-external-modules/2003/12 "2019-10-30T19:50:18Z")
For vtkDICOM, if I was forced to choose between supporting external builds vs. building as a remote module, then I would definitely go for external build. However, I can’t deny that I like the convenience of remote modules and don’t want them to go away.
One concern that I have is packaging (pypi). Are the remote modules going to be included in the main vtk package? I kinda think they shouldn’t, even though it means I’ll have to make my own vtkDICOM pypi package. Neither does it make sense for them to be built by default, except for by the dashboards for testing.
Also, for Python, I don’t know if the package structure is resolved yet. The python modules for external packages obviously cannot accessed by “`import vtk.`” or “`import vtkmodules.`”. So should things be set up to that people use “`import vtk.`” or simply "`import vtk"`?
---
### Author:  [@ben.boeckel](https://discourse.vtk.org/u/ben.boeckel)
#### Post date: [October 30, 2019, 8:17pm UTC](https://discourse.vtk.org/t/remote-modules-vs-external-modules/2003/13 "2019-10-30T20:17:29Z")
> [@dgobbi](#):
>
> However, I can’t deny that I like the convenience of remote modules and don’t want them to go away.
What I want to do is convert them to contract tests.
> [@dgobbi](#):
>
> One concern that I have is packaging (pypi). Are the remote modules going to be included in the main vtk package?
I _really_ don’t want to do that. You’ll be subject to VTK’s version scheme at that point. And since development for them happens all over the place, respinning releases because a bug was fixed in `vtkDICOM` seems a bit much to me.
> [@dgobbi](#):
>
> Also, for Python, I don’t know if the package structure is resolved yet. The python modules for external packages obviously cannot accessed by “ `import vtk.` ” or “ `import vtkmodules.` ”. So should things be set up to that people use “ `import vtk.` ” or simply " `import vtk"` ?
Python is where it gets really hairy. There’s no real way to distribute a package that can live in one of two places which doesn’t force everyone to do:
```python
try:
import vtkdicom
except ImportError:
import vtkmodules.dicom as vtkdicom
```
without making a _third_ package/module to do it for you and having everyone use _that_ one. The `vtk` and `vtkmodules` bits built as part of ParaView has this issue; it cannot be vendored in any sensible way without affecting everyone who wants to be flexible over how VTK is provided. (A big reason I really want ParaView to rely on VTK releases rather than a submodule, but that’s a long road.)