VTK Array doesn't work correctly if it's used under Unreal Engine

Development
,
1

Hello Everyone.
I’ve been trying to integrate VTK into Unreal Engine for months.
I wrote a plugin to wrap VTK using as a thirdparty library.
The plugin module built with RTTI enabled due to compiling VTK on Linux.
But everything else follows Unreal Rules by turning off RTTI.
Almost VTK works pretty, but there is critical problems.
That’s what vtk array doesn’t correctly map internal memory layout for component and tuple.
Here is simple c++ codes,


float colorMap[8][4] = 
{ 
    { 0.0, 0.0, 0.0, 1.0 },
    { 0.3, 0.0, 0.0, 1.0 },
    { 0.6, 0.0, 0.0, 1.0 },
    { 0.9, 0.0, 0.0, 1.0 },
    { 0.9, 0.3, 0.3, 1.0 },
    { 0.9, 0.6, 0.6, 1.0 },
    { 0.9, 0.9, 0.9, 1.0 },
    { 1.0, 1.0, 1.0, 1.0 }
};

vtkNew < vtkFloatArray > array;  // (e.g almost arrays is derived from vtkAOSDataArrayTemplate)
array->SetNumberOfComponents(4);
for (int i=0;i<8;i++) {
    array->InsertNextTuple(colorMap[i]);
}

int numComps = array->GetNumberOfComponents();
int numTuples = array->GetNumberOfTuples();

At results, console program print 4 number of components and 8’s tuples, that works correctly as expect.
At Unreal side, that always print 31 number of components and only 1’s tuple.
It’s definitely incorrect.
[EDIT] There is a little bit mistake. number of components is constantly 31 not 33 for the colorMap.

I have frequently cross checked in some scenarios.
First, checked out if there is no stdc++ runtime compatible issues,
because both implement each diffrent memory allocators, vtk use standard c++'s malloc, new, delete but Unreal use FMalloc which implements mimalloc internaly.
Resulty both use the same version’s stdc++ runtime in Linux so there is no problems.

Second, I tested with two VTK build versions, one is built with gcc 14 or later in Linux, other is built by compiler clang++18 Unreal Engine provide byself.
Unfortunately that result same thing.

Finally, there is another issue,VTK Polydata Reader get failed to read cell data such normals, scalars via vtkCellData in Unreal but works in the console app.
I’m not sure this issue coming from array issue first said.

I use VTK 9.4.1 or 9.4.2, and with Unreal 5.5.4.
Have anyone same experience?
Welcome anyone help me.

2

When you compile VTK for your plugin, are the VTK libraries static or are they shared?

Also, when Unreal Engine loads the plugin, do you know whether it calls dlopen() with RTLD_GLOBAL or not?

As an old-timer with VTK, I know that in VTK 5 (around 10 years ago), VTK did not depend on RTTI at all. It implemented its own SafeDownCast() and IsA() methods that work fine even if compiler RTTI was disabled. However, very few developers are aware of the problems that RTTI can cause with plugins on Linux, so it’s likely that VTK 9 uses RTTI in many places.

I’m curious, what happens if you try to build VTK with RTTI disabled? Does it fail to build? Or does it build, but then fail when you try to use it (e.g. segfault)?

Edit: For anyone reading this, the issues with RTTI and dlopen() are described here:
https://gcc.gnu.org/faq.html#dso

3

Thank your reply. I’ve compiled VTK only as shared library. Also, I tried both dlopen or not.
For dlopen, I just use GetDllHandle which independent to platform in Unreal to load VTK at any time. it’s exactly same as dlopen.
For not dlopen, I don’t have to do anything. As you know, VTK is loaded when the app started up by the ld.
In most of case, I use second one.
For RTTI , you’re right. Also, the VTK team explain that VTK’s array model has been re-designed after 7.1. here [vtkArrayDispatch and Related Tools - VTK documentation](https://vtkArray Dispatch and Related Tools)
For everything, it’s successful in while build VTK and integrate to Unreal no failed.
It’s problem that VTK array holds incorrect memory layout at runtime.
Although, App cause segfault when I try to read cell base data via vtkCellData only in Unreal.
When I trace the callstack, the final broken point is in VTK Array’s data buffer meet to null.

4

Do you know whether unreal’s GetDllHandle uses the RTLD_GLOBAL flag, or whether it can be forced to use it? Because if VTK is built with shared libraries and if dlopen() is called with RTLD_GLOBAL, then it should work (in theory, at least).

You are correct about the VTK array redesign, the new design uses typeid() in many places. Therefore if RTTI is not working, then VTK arrays will not work.

5

On closer inspection, the data arrays don’t actually use typeid(), they only mention it in the code comments in the sense of “this operation is like typeid(), but doesn’t actually use typeid()”.

So if the data arrays are failing, it might not actually be due to RTTI. It might be due to:
a) symbols for static variables that aren’t resolved correctly
b) symbols for template instantiations that aren’t resolved correctly
c) static initialization of translation units not occurring in the correct order

6

Right, I think RTLD_GLOBAL will be used. I can see that in GetDllHandle defintion below.
the function use RTLD_GLOBAL if unreal 4 module is not.
Also, I don’t have any VTK installed on system global.
VTK libraryies I’ll load are in a path such as MyPlugin/Binaries/ThirdParty/MyVtkThirdpartyLibrary/Linux.

void* FUnixPlatformProcess::GetDllHandle( const TCHAR* Filename )
{
	check( Filename );
	FString AbsolutePath = FPaths::ConvertRelativePathToFull(Filename);

	// first of all open the lib in LOCAL mode (we will eventually move to GLOBAL if required)
	int DlOpenMode = RTLD_LAZY;
	void *Handle = dlopen( TCHAR_TO_UTF8(*AbsolutePath), DlOpenMode | RTLD_LOCAL );
	if (Handle)
	{
		bool UpgradeToGlobal = false;
		// check for the "ue4_module_options" symbol
		const char **ue4_module_options = (const char **)dlsym(Handle, "ue4_module_options");
		if (ue4_module_options)
		{
			// split by ','
			TArray<FString> Options;
			FString UE4ModuleOptions = FString(ANSI_TO_TCHAR(*ue4_module_options));
			int32 OptionsNum = UE4ModuleOptions.ParseIntoArray(Options, ANSI_TO_TCHAR(","), true);
			for(FString Option : Options)
			{
				if (Option.Equals(FString(ANSI_TO_TCHAR("linux_global_symbols")), ESearchCase::IgnoreCase))
				{
					UpgradeToGlobal = true;
				}
			}
		}
		else
		{
			// is it ia ue4 module ? if not, move it to GLOBAL
			void *IsUE4Module = dlsym(Handle, "ThisIsAnUnrealEngineModule");
			if (!IsUE4Module)
			{
				IsUE4Module = dlsym(Handle, "InitializeModule");
			}

			if (!IsUE4Module)
			{
				UpgradeToGlobal = true;
			}
		}

		if (UpgradeToGlobal)
		{
			dlclose( Handle );
			Handle = dlopen( TCHAR_TO_UTF8(*AbsolutePath), DlOpenMode | RTLD_GLOBAL );
		}
	} 
	else if (!FString(Filename).Contains(TEXT("/")))
	{
		// if not found and the filename did not contain a path we search for it in the global path
		Handle = dlopen( TCHAR_TO_UTF8(Filename), DlOpenMode | RTLD_GLOBAL );
	}

	if (!Handle)
	{
		UE_LOG(LogCore, Warning, TEXT("dlopen failed: %s"), UTF8_TO_TCHAR(dlerror()) );
	}

	return Handle;
}
7

Sure, I get failed to build if RTTI is disabled. I have to enable RTTI in Build.cs only for the module that require to use directly VTK.

8

The “global” in RTLD_GLOBAL is not about where the libraries are installed. It is about whether dlopen() hides their symbols from other shared objects used by the same process.

RTLD_GLOBAL
The object’s symbols are made available for the relocation processing of any other object.

RTLD_LOCAL
The object’s symbols are not made available for the relocation processing of any other object.

9

Okay, I got it. On first tracking, I have thought this would have something to ABI.
But I thought it’s not simple for ABI. So, I just tried to find the clue in more simple case, I just had insterested in that definite same codes work each differently on simple console app and Unreal with the same VTK shared libs. In both, they run with a loading VTK shared library at start up instead of way using dlopen.
I don’t think this have a something with RTTI.
Do you think there is wrong for shared library?

a) symbols for static variables that aren’t resolved correctly?
I don’t use static variables for VTK. At sample codes use variables on stack and heap.

b) symbols for template instantiations that aren’t resolved correctly
I think it would be very rare because simple console app works correctly for same codes which use VTK template array API.

c) static initialization of translation units not occurring in the correct order
I’m not sure but I don’t have a lot idea about this.
When do this things go? what does it going to be related with Shared Library?

10

a) VTK uses static variables internally. Even if you don’t use them in your own code, they will still exist and they can cause problems.
b) The reason I mentioned templates is because https://gcc.gnu.org/faq.html#dso explicitly states that templates can cause problems for dlopen()
c) You can read more about static initialization here: https://en.cppreference.com/w/cpp/language/siof.html. It’s a tricky subject, and it’s probably not the problem…

In any case, it has been a long time since I’ve built any apps that use dlopen. I’ve already given you all the information that I have.

11

Thank you for your time and information.
It was helpful. I’ll try to figure out your mentions in order to solve it.

12

One final comment. I read through the GetDllHandle() code that you posted:

// check for the ue4_module_options symbol|
const char **ue4_module_options = (const char **)dlsym(Handle, ue4_module_options);|
...
	if (Option.Equals(FString(ANSI_TO_TCHAR("linux_global_symbols")), ESearchCase::IgnoreCase))
	{
		UpgradeToGlobal = true;
	}

The above code suggests that you can declare the following global variable in your module:

__attribute__((visibility("default")))
const char *ue4_module_options = "linux_global_symbols";

According to the code you posted, GetDllHandle() will read this variable using dlsym(), and if the variable is set to “linux_global_symbols”, then GetDllHandle() will use RTLD_GLOBAL.

13

Thank you. I was able to see Unreal seems to load symbols for its own modules on RTLD_LOCAL anything else on RTLD_GLOBAL by debugging. So I can see VTK shared library is loading in RTLD_GLOBAL.
I’m understanding what you would like to say is that symbols in shared library should works if they is loaded in RTLD_GLOBAL.
I’ll try to use way by dlopen by changing call timing.