How to integrate VTK with Unity

Development
1

Hello there, pretty new to the forum and VTK

I’m working on a project that need advance rendering in Unity. I’ve thought this would be a achievable through VTK and i still think it can’t be that hard but i’m stuck.

There is the VTK External Rendering module and Unity provides a low level plugin interface where C++ can be used. There’s an example that uses OpenGL calls to manipulate vertex buffers and textures in Unity’s OpenGL context. I’m following the approach described in this paper Virtual Interaction and Visualisation of 3D Medical Imaging Data with VTK and Unity

So, I made a Unity script that uses a command buffer that calls a vtk rendering function every frame at (I think) the correct state of OpenGL rendering. This function is imported from a dynamic library which has the VTK code.

Unity code:

[DllImport(“RenderingPlugin”)]
private static extern IntPtr GetRenderVTKFunc ();

CommandBuffer buf = null;

// Start is called before the first frame update

IEnumerator Start() {

    buf = new CommandBuffer();
    buf.name = "VTK CommandBuffer";

    Camera.main.AddCommandBuffer(CameraEvent.BeforeImageEffects, buf);

    yield return StartCoroutine("callPluginEndOfFrame");
}

private IEnumerator callPluginEndOfFrame() {

    while (true) {
        yield return new WaitForEndOfFrame();

        //SetTimeFromUnity(Time.timeSinceLevelLoad);
        buf.IssuePluginEvent(GetRenderVTKFunc(), 1);         
    }
}

Fragment of VTK code:

//Definitions

vtkNew cs;
vtkNew mapper;

mapper->SetInputConnection(cs->GetOutputPort());
m_CubeActor->SetMapper(mapper.GetPointer());
m_CubeActor->GetProperty()->SetColor(1, 1, 0.2);
m_CubeActor->GetProperty()->SetOpacity(0.5);
m_CubeActor->SetPosition(0.5, 0, 0);

// create the VTK external renderer
vtkNew renWin;
m_ExternalVTKWidget->SetRenderWindow(renWin.GetPointer());
m_ExternalVTKWidget->GetRenderWindow()->AddRenderer(m_renderer.GetPointer());
m_renderer->AddActor(m_CubeActor.GetPointer());
m_renderer->ResetCamera();
m_externalVtkCamera = vtkSmartPointer(
dynamic_cast<vtkExternalOpenGLCamera*>(m_renderer->GetActiveCamera()));

//Function that is called by Unity

void RenderAPI_OpenGLCoreES::UpdateVtk(const double viewMatrix[16], const double projectionMatrix[16]) //std::string &debugStr)
{
m_externalVtkCamera->SetViewTransformMatrix(viewMatrix);
m_externalVtkCamera->SetProjectionTransformMatrix(projectionMatrix);

m_CubeActor->RotateX(1.0f);
m_CubeActor->RotateY(1.0f);

m_ExternalVTKWidget->GetRenderWindow()->Render();
}

When executing this the following OpenGL error pop up

OPENGL NATIVE PLUG-IN ERROR: GL_INVALID_OPERATION: Operation illegal in current state

Any thoughts would be appreciated

Thanks in advance!

2 Likes
2

My guess is that Unity is expecting the external plugin to create the OpenGL context, but I don’t see that happening here in the VTK code. Could you please link the relevant Unity documentation?

Cc: @ken-martin

3

Unity low level interface plugin doc: Low-Level interface plugin

Thanks for your response! It’s a possibility but I don´t think thats the case because in the example provided by Unity in the documentation the c++ code only draws (through glew) on the OpenGL context created by Unity

Low-level interface plugin example (non VTK rendering)

4

I have a little bit more info about this. In case someone can help me.

I’ve been doing some debugging and i get this error:

glDrawBuffer failed because ‘0x405’ is not a compatible or supported buffer type under the current framebuffer configuration (GL_INVALID_OPERATION)

This error is triggered by the glDrawBuffer in the render function where doublebuffer is active in vtkExternalOpenGLCamera. There:

if (ren->GetRenderWindow()->GetDoubleBuffer())
{
  glDrawBuffer(static_cast<GLenum>(win->GetBackBuffer()));

  // Reading back buffer means back left. see OpenGL spec.
  // because one can write to two buffers at a time but can only read from
  // one buffer at a time.
  glReadBuffer(static_cast<GLenum>(win->GetBackBuffer()));
}

I’ve doing some researching about framebuffers, back and front buffer but i’m struggling to understand it properly.

Thanks in advance

5

I’m out of my depth at this at this point too. @ken-martin or @utkarshayachit?

6

Hi Oscar!
I am integrating the VTK with Unity, any luck you have already solve this issue?

7

Hi,

I am one of the authors of the paper on the first post -just wanted to inform that we have released part of the code open source and can be found here:
https://gitlab.com/3dheart_public/vtktounity
If you have any problems please let us know through the issue system in gitlab, or here and we will try to help.

Alberto

8

Hi Alberto, I just wanted to ask, with this framework, is it possible to load .vtp (poly data) files in Unity?

9

Yes it is.

If you have further question, please open your own topic.