# Wrapping VTK in an inline namespace

**URL:** <https://discourse.vtk.org/t/wrapping-vtk-in-an-inline-namespace/8072>\
**Category:** Development\
**Created:** [March 16, 2022, 7:02pm UTC](https://discourse.vtk.org/t/wrapping-vtk-in-an-inline-namespace/8072 "2022-03-16T19:02:47Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ryan\_Krattiger](https://discourse.vtk.org/user_avatar/discourse.vtk.org/ryan_krattiger/32/4723_2.png) [@Ryan\_Krattiger](https://discourse.vtk.org/u/Ryan_Krattiger)\
**Post date:** [March 16, 2022, 7:02pm UTC](https://discourse.vtk.org/t/wrapping-vtk-in-an-inline-namespace/8072/1 "2022-03-16T19:02:47Z")

</div>

# Motivation

The motivation for this change is to allow applications to link multiple major versions of VTK into the same application.

A simple example of this could be an application A depends on library B and VTK 9, and library B depends on VTK 8. At runtime, both VTK 9 and VTK 8 will need to be loaded to satisfy the dependencies for both application A and library B. The problem with this is the symbols from VTK 9 will conflict with the symbols for VTK 8.

# Objective

The goal of this change is NOT to support applications building against multiple version of VTK in the same project. The goal is to only support linking external project(s) that support a different version(s) of VTK than the main application/each other.

# Proposal

The current proposal of how this implementation would work is as follows.

Add a CMake option for naming the ABI namespace `VTK_ABI_NAMESPACE_NAME`. Default is `vtk${VTK_MAJOR_VERSION}`

Insert an inline namespace around VTK symbols.

```c++
inline namespace VTK_ABI_NAMESPACE_NAME {
...
} // inline namespace VTK_ABI_NAMESPACE_NAME

```

# Current Work

This work has been preliminarily done [here](https://gitlab.kitware.com/vtk/vtk/-/merge_requests/8993).

## Issues

- vtkWrap does not recognize namespaces. A brief explanation from @dgobbi of why this is the case can be be found [here](https://discourse.vtk.org/t/adding-and-wrapping-a-custom-namespace-within-vtk/1099/7).
- Some VTK headers may be included in C source, so `inline namespace` cannot be used directly
  - Fix: Use wrappers `VTK_ABI_NAMESPACE_{BEGIN,END}`

---

<div class="post-metadata">

**Author:** ![Ryan\_Krattiger](https://discourse.vtk.org/user_avatar/discourse.vtk.org/ryan_krattiger/32/4723_2.png) [@Ryan\_Krattiger](https://discourse.vtk.org/u/Ryan_Krattiger)\
**Post date:** [March 16, 2022, 7:03pm UTC](https://discourse.vtk.org/t/wrapping-vtk-in-an-inline-namespace/8072/2 "2022-03-16T19:03:25Z")

</div>

@bill-hoffman @ben.boeckel

---

<div class="post-metadata">

**Author:** ![bill-hoffman](https://discourse.vtk.org/user_avatar/discourse.vtk.org/bill-hoffman/32/3750_2.png) [@bill-hoffman](https://discourse.vtk.org/u/bill-hoffman)\
**Post date:** [March 16, 2022, 7:32pm UTC](https://discourse.vtk.org/t/wrapping-vtk-in-an-inline-namespace/8072/3 "2022-03-16T19:32:47Z")

</div>

> [@Ryan\_Krattiger](#):
>
> vtkWrap does not recognize namespaces. A brief explanation from @dgobbi of why this is the case can be be found [here](https://discourse.vtk.org/t/adding-and-wrapping-a-custom-namespace-within-vtk/1099/7).

That is not exactly the same thing. That is adding namespaces to the wrapping code so that if you have something in a namespace that shows up in a namespace in python. In this case with an inline namespace, we don’t want to change the resulting python wrapped code at all. We just need the parser to ignore the inline namespace when doing the wrapping. @dgobbi what is the best way to do that?

---

<div class="post-metadata">

**Author:** ![dgobbi](https://discourse.vtk.org/user_avatar/discourse.vtk.org/dgobbi/32/18_2.png) [@dgobbi](https://discourse.vtk.org/u/dgobbi)\
**Post date:** [March 16, 2022, 8:53pm UTC](https://discourse.vtk.org/t/wrapping-vtk-in-an-inline-namespace/8072/4 "2022-03-16T20:53:40Z")

</div>

> [@bill-hoffman](#):
>
> we don’t want to change the resulting python wrapped code at all.

All that’s needed is a rule in vtkParse.y to ignore the “inline namespace” declaration, it’s a one-line change. But vtkParse.tab.c will also need to be regenerated. I’ll put together an MR later today.

---

<div class="post-metadata">

**Author:** ![Ryan\_Krattiger](https://discourse.vtk.org/user_avatar/discourse.vtk.org/ryan_krattiger/32/4723_2.png) [@Ryan\_Krattiger](https://discourse.vtk.org/u/Ryan_Krattiger)\
**Post date:** [March 16, 2022, 9:04pm UTC](https://discourse.vtk.org/t/wrapping-vtk-in-an-inline-namespace/8072/5 "2022-03-16T21:04:09Z")

</div>

> By only defining the namespace macro to expand when ` __VTK_WRAP__ ` is not defined seems to work as expected without any changes to the vtkParser. The current problem with this implementation is not all files include `vtkABI.h` which is where I have the macros defined at the moment.

Edit: Scratch that, it still gets stuck on the macro.

If adding something to ignore `inline namespace` is not too hard though, I think that would be preferred.

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.vtk.org/u/ben.boeckel)\
**Post date:** [March 17, 2022, 5:25am UTC](https://discourse.vtk.org/t/wrapping-vtk-in-an-inline-namespace/8072/6 "2022-03-17T05:25:20Z")

</div>

> [@Ryan\_Krattiger](#):
>
> If adding something to ignore `inline namespace` is not too hard though, I think that would be preferred.

This is best because any VTK-using project wanting to do the same thing will have its own inline namespace; the literal names used here will not be usable generally.

> [@Ryan\_Krattiger](#):
>
> Add a CMake option for naming the ABI namespace `VTK_ABI_NAMESPACE_NAME`. Default is `vtk${VTK_MAJOR_VERSION}`

The default is to not have a namespace at all, correct?

---

<div class="post-metadata">

**Author:** ![Ryan\_Krattiger](https://discourse.vtk.org/user_avatar/discourse.vtk.org/ryan_krattiger/32/4723_2.png) [@Ryan\_Krattiger](https://discourse.vtk.org/u/Ryan_Krattiger)\
**Post date:** [March 17, 2022, 6:09pm UTC](https://discourse.vtk.org/t/wrapping-vtk-in-an-inline-namespace/8072/7 "2022-03-17T18:09:39Z")

</div>

> [@ben.boeckel](#):
>
> The default is to not have a namespace at all, correct?

That could be done. What is the disadvantage of having it on by defautl?

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.vtk.org/u/ben.boeckel)\
**Post date:** [March 19, 2022, 5:15am UTC](https://discourse.vtk.org/t/wrapping-vtk-in-an-inline-namespace/8072/8 "2022-03-19T05:15:22Z")

</div>

> [@Ryan\_Krattiger](#):
>
> What is the disadvantage of having it on by defautl?

I’ll have to peek at the MR, but CMake defaults are difficult. It should at least definitely be:

```cmake
set(VTK_NAMESPACE "<DEFAULT>"
  CACHE STRING "docs")
if (VTK_NAMESPACE STREQUAL "<DEFAULT>")
  set(VTK_NAMESPACE "actual default")
endif ()

```

because otherwise a build tree that configures with VTK9 but later checks out VTK10 will continue to be mangled with `vtk9`. This is not good, so the above is preferred. However, with this behavior, switching between branches is now a rebuild of the _entire tree_ because the namespace changed. In that sense, I think the core default should be an empty namespace.

---

<div class="post-metadata">

**Author:** ![Ryan\_Krattiger](https://discourse.vtk.org/user_avatar/discourse.vtk.org/ryan_krattiger/32/4723_2.png) [@Ryan\_Krattiger](https://discourse.vtk.org/u/Ryan_Krattiger)\
**Post date:** [March 29, 2022, 3:47pm UTC](https://discourse.vtk.org/t/wrapping-vtk-in-an-inline-namespace/8072/9 "2022-03-29T15:47:41Z")

</div>

So I ran into an issue with forward declarations that will cause issues with existing VTK codes after adding the namespace.

In a lot of VTK code there are forward declarations of classes…

```auto
class vtkObject;

```

To the compiler, this creates a new declaration of `vtkObject` that is not namespaced and becomes ambiguous between `::vtk9::vtkObject` and `::vtkObject`. Right now the quick fix in the examples is to just add the `VTK_ABI_NAMESPACE_{BEGIN,END}` macro around the forward declarations.

```auto
VTK_ABI_NAMESPACE_BEGIN
class vtkObject;
VTK_ABI_NAMESPACE_END

```

This is rather verbose and ideally the downstream user shouldn’t have to think about these macros.

Two possible solutions to this problem immediately come to mind. This first solution is to just add another layer of abstraction, similar to do what I did for the AutoInit `DECLARE` macro, and hide the namespace inside of a macro.

```auto
vtkForwardDeclare(class vtkObject);

```

Another possible option is to inject forward declarations of VTK module classes into the `<ModuleName>Module.h` headers or a `<ModuleName>Fwd.h` header that just forward declares everything in the module. In that case, users would remove all of the VTK forward declarations from their source and just include headers.

@ben.boeckel @bill-hoffman Do you guys, or anyone else, have better ideas on how to solve this or a preference/strong objection to either/both of the ideas here? Personally I think the former (macro) is ugly and verbose and the latter (forward header) is a more C+±style solution. But both are going to require changes by the downstream user.

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.vtk.org/u/ben.boeckel)\
**Post date:** [March 29, 2022, 4:26pm UTC](https://discourse.vtk.org/t/wrapping-vtk-in-an-inline-namespace/8072/10 "2022-03-29T16:26:51Z")

</div>

I agree that the former is ugly. The latter requires some tooling to ensure that it matches up with the set of public classes (basically just more `HeaderTest` work). Without some way that shows up in CI that flags when the `Fwd` header is out of sync, I don’t think the latter is any better in the long run.

---

<div class="post-metadata">

**Author:** ![lassoan](https://discourse.vtk.org/user_avatar/discourse.vtk.org/lassoan/32/50_2.png) [@lassoan](https://discourse.vtk.org/u/lassoan)\
**Post date:** [April 8, 2022, 6:57pm UTC](https://discourse.vtk.org/t/wrapping-vtk-in-an-inline-namespace/8072/11 "2022-04-08T18:57:40Z")

</div>

Have you seen any major library that isolates different library versions using namespaces? If yes, we may get good ideas how they solved the problems you have run into. If there are no (or very few examples) then using namespaces for this may not be a good idea.

> [@Ryan\_Krattiger](#):
>
> The goal is to only support linking external project(s) that support a different version(s) of VTK than the main application/each other.

We have come across this problem when building applications that worked with external hardware that came with SDKs that used VTK and Qt internally. We did not want the SDK to dictate what VTK+Qt versions our application should use, so we asked the SDK developers to fix this. They changed their build system to build VTK statically and mangled Qt DLL names and it was all good. There was no need to change VTK or our application in any way. A similar example is how an application can use many different C runtime versions - there is no messing with namespaces there either.

If you cannot ask the third-party developers to change how they build VTK then you may be able to use manifests or rpaths to ensure each executable loads the correct VTK version.

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.vtk.org/u/ben.boeckel)\
**Post date:** [April 8, 2022, 10:27pm UTC](https://discourse.vtk.org/t/wrapping-vtk-in-an-inline-namespace/8072/12 "2022-04-08T22:27:41Z")

</div>

> [@lassoan](#):
>
> Have you seen any major library that isolates different library versions using namespaces?

Not via a per-version mechanism, no. Qt and Boost allow one to mangle using some pre-build process, but that’s about as far as I’ve seen C++ libraries go. It’s basically what VTK is doing.

> [@lassoan](#):
>
> They changed their build system to build VTK statically and mangled Qt DLL names and it was all good.

This works on Windows and macOS where symbols are looked up in a per-library namespace. It doesn’t work at all on Linux where all public symbols get dumped into a single lookup table. One could make the VTK symbols non-exported, but that isn’t something we support either.

> [@lassoan](#):
>
> A similar example is how an application can use many different C runtime versions - there is no messing with namespaces there either.

This is due to the symbol lookup mechanisms; it is a platform feature that isn’t everywhere.

> [@lassoan](#):
>
> you may be able to use manifests or rpaths to ensure each executable loads the correct VTK version.

Again, this leaves Linux out to dry as the loader will see duplicate sonames and just say “nope, already did that” before driving right off a cliff.

---

<div class="post-metadata">

**Author:** ![lassoan](https://discourse.vtk.org/user_avatar/discourse.vtk.org/lassoan/32/50_2.png) [@lassoan](https://discourse.vtk.org/u/lassoan)\
**Post date:** [April 8, 2022, 11:21pm UTC](https://discourse.vtk.org/t/wrapping-vtk-in-an-inline-namespace/8072/13 "2022-04-08T23:21:09Z")

</div>

> [@ben.boeckel](#):
>
> Not via a per-version mechanism, no. Qt and Boost allow one to mangle using some pre-build process, but that’s about as far as I’ve seen C++ libraries go. It’s basically what VTK is doing.

It seems that the motivation for the proposed change is not very strong:

- no other libraries are doing this (if I understood your comment correctly)
- VTK has been fine without this mechanism until now (for almost 30 years and thousands of VTK-based projects)
- there are not-too-complicated alternative solutions for Windows and macOS
- you can find solutions for Linux, too (e.g., work with the third-party library developer to support more recent VTK versions)

If you wanted to do a small, non-invasive change then even a weak motivation can be sufficient.

However, if you want to introduce a new special VTK mechanism (such as `VTK_ABI_NAMESPACE_BEGIN` or `vtkForwardDeclare`) that impacts every class in VTK proper (and potentially all VTK-based projects) then you would need to demonstrate high value for the entire VTK community. Apart from solving VTK upgrade problem in your project, would there be other useful applications of this new namespace mechanism?

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.vtk.org/u/ben.boeckel)\
**Post date:** [April 8, 2022, 11:55pm UTC](https://discourse.vtk.org/t/wrapping-vtk-in-an-inline-namespace/8072/14 "2022-04-08T23:55:31Z")

</div>

> [@lassoan](#):
>
> work with the third-party library developer to support more recent VTK versions

That’s the push for this effort. The other project just cannot update (because VTK _still_ keeps breaking API despite my efforts to try and push back on such things).

> [@lassoan](#):
>
> and potentially all VTK-based projects

ParaView 100% doesn’t care about this, so I don’t think it should care at all. It should just be limited to VTK (that is, every project wanting to do this can make their own decision about it and how to do it). Note that VTK also doesn’t provide mechanisms intended to work for anything not VTK either (that is, there is no exported framework/API for doing this to an arbitrary project).

---

<div class="post-metadata">

**Author:** ![Ryan\_Krattiger](https://discourse.vtk.org/user_avatar/discourse.vtk.org/ryan_krattiger/32/4723_2.png) [@Ryan\_Krattiger](https://discourse.vtk.org/u/Ryan_Krattiger)\
**Post date:** [April 9, 2022, 3:41pm UTC](https://discourse.vtk.org/t/wrapping-vtk-in-an-inline-namespace/8072/15 "2022-04-09T15:41:53Z")

</div>

> [@lassoan](#):
>
> Have you seen any major library that isolates different library versions using namespaces?

Yes, there are a couple of libraries in Boost, notably Boost.Log and Boost.Filesystem, that utilize this mechanism for versioning. That said, that is not what this change is for, it was just convenient to use the major version as a default example. It seems both Qt and Boost also provide tools for mangling all symbols .

> [@lassoan](#):
>
> It seems that the motivation for the proposed change is not very strong.

Just a note, the proposed changes here are 100% opt-in for the foreseeable future. The purpose is the make handling VTK updates easier in the cases where up/down-stream projects also depend on different major versions of VTK.

> [@lassoan](#):
>
> then you would need to demonstrate high value for the entire VTK community

While it does touch most of the files in the main VTK repo, it is non-intrusive to all downstream code except in the case of forward declarations. So what that means for VTK-based projects is _if_ they want to use a mangled VTK, they will need to do something different to handle the forward declarations. Otherwise it is business as usual.

---

<div class="post-metadata">

**Author:** ![lassoan](https://discourse.vtk.org/user_avatar/discourse.vtk.org/lassoan/32/50_2.png) [@lassoan](https://discourse.vtk.org/u/lassoan)\
**Post date:** [April 10, 2022, 5:50pm UTC](https://discourse.vtk.org/t/wrapping-vtk-in-an-inline-namespace/8072/16 "2022-04-10T17:50:25Z")

</div>

> [@Ryan\_Krattiger](#):
>
> It seems both Qt and Boost also provide tools for mangling all symbols .

Thanks, it was a good pointer. I remembered that Qt supports custom infix for DLL names, but not that it also supports placing Qt into a custom namespace - see [Qt in namespace](https://wiki.qt.io/Qt_In_Namespace). It makes the proposed change much more acceptable, because:

- It is much easier to justify introduction of this complex mechanism if we can show that another well-known libraries do the same.
- There is a proven implementation that we can learn from: the risks of introducing unforeseen complications is much lower and there is no need to invent new things (for example, I would use the same macros that Qt uses for forward declarations).
- Qt implementation has been proven to not affect external Qt code (if Qt is built without namespace then external code does not need to bother with the namespace macros).

There could still be some VTK-specific issues, for example how to handle symbols in third-party libraries (can they be placed in the namespace?). It is also hard to imagine that it is really impossible to hide internal use of an old VTK version with a static build and hiding/stripping of private symbols (which would be a much less invasive change).

---

<div class="post-metadata">

**Author:** ![ben.boeckel](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.vtk.org/u/ben.boeckel)\
**Post date:** [April 11, 2022, 5:07pm UTC](https://discourse.vtk.org/t/wrapping-vtk-in-an-inline-namespace/8072/17 "2022-04-11T17:07:18Z")

</div>

> [@lassoan](#):
>
> There could still be some VTK-specific issues, for example how to handle symbols in third-party libraries (can they be placed in the namespace?).

Yes, this is still a problem that needs tackled with this approach.

> [@lassoan](#):
>
> It is also hard to imagine that it is really impossible to hide internal use of an old VTK version with a static build and hiding/stripping of private symbols (which would be a much less invasive change).

Yes, but the (my) confidence in any such solution actually working in a production-ready way is also near the “good luck, have fun storming the castle” level. And the crashes from these kinds of things are not typically obvious or easy to reason about.