# How to integrate VTK with Unity

**URL:** https://discourse.vtk.org/t/how-to-integrate-vtk-with-unity/1108
**Category:** Development
**Created:** [June 10, 2019, 8:27am UTC](https://discourse.vtk.org/t/how-to-integrate-vtk-with-unity/1108 "2019-06-10T08:27:42Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![Oscar](https://discourse.vtk.org/user_avatar/discourse.vtk.org/oscar/32/1498_2.png) [@Oscar](https://discourse.vtk.org/u/Oscar)
#### Post date: [June 10, 2019, 8:27am UTC](https://discourse.vtk.org/t/how-to-integrate-vtk-with-unity/1108/1 "2019-06-10T08:27:42Z")

</div>

Hello there, pretty new to the forum and VTK

I’m working on a project that need advance rendering in Unity. I’ve thought this would be a achievable through VTK and i still think it can’t be that hard but i’m stuck.

There is the VTK External Rendering module and Unity provides a low level plugin interface where C++ can be used. There’s an example that uses OpenGL calls to manipulate vertex buffers and textures in Unity’s OpenGL context. I’m following the approach described in this paper [Virtual Interaction and Visualisation of 3D Medical Imaging Data with VTK and Unity](https://www.researchgate.net/publication/327182589_Virtual_Interaction_and_Visualisation_of_3D_Medical_Imaging_Data_with_VTK_and_Unity)

So, I made a Unity script that uses a command buffer that calls a vtk rendering function every frame at (I think) the correct state of OpenGL rendering. This function is imported from a dynamic library which has the VTK code.

Unity code:

> [DllImport(“RenderingPlugin”)]  
> private static extern IntPtr GetRenderVTKFunc ();
> 
> CommandBuffer buf = null;
> 
> ```
> // Start is called before the first frame update
> 
> ```
> 
> IEnumerator Start() {
> 
> ```
> buf = new CommandBuffer();
> buf.name = "VTK CommandBuffer";
> 
> Camera.main.AddCommandBuffer(CameraEvent.BeforeImageEffects, buf);
> 
> yield return StartCoroutine("callPluginEndOfFrame");
> }
> 
> private IEnumerator callPluginEndOfFrame() {
> 
> while (true) {
> yield return new WaitForEndOfFrame();
> 
> //SetTimeFromUnity(Time.timeSinceLevelLoad);
> buf.IssuePluginEvent(GetRenderVTKFunc(), 1);         
> }
> }
> 
> ```

Fragment of VTK code:

> //Definitions
> 
> vtkNew cs;  
> vtkNew mapper;
> 
> mapper-\>SetInputConnection(cs-\>GetOutputPort());  
> m\_CubeActor-\>SetMapper(mapper.GetPointer());  
> m\_CubeActor-\>GetProperty()-\>SetColor(1, 1, 0.2);  
> m\_CubeActor-\>GetProperty()-\>SetOpacity(0.5);  
> m\_CubeActor-\>SetPosition(0.5, 0, 0);
> 
> // create the VTK external renderer  
> vtkNew renWin;  
> m\_ExternalVTKWidget-\>SetRenderWindow(renWin.GetPointer());  
> m\_ExternalVTKWidget-\>GetRenderWindow()-\>AddRenderer(m\_renderer.GetPointer());  
> m\_renderer-\>AddActor(m\_CubeActor.GetPointer());  
> m\_renderer-\>ResetCamera();  
> m\_externalVtkCamera = vtkSmartPointer(  
> dynamic\_cast\<vtkExternalOpenGLCamera\*\>(m\_renderer-\>GetActiveCamera()));
> 
> //Function that is called by Unity
> 
> void RenderAPI\_OpenGLCoreES::UpdateVtk(const double viewMatrix[16], const double projectionMatrix[16]) //std::string &debugStr)  
> {  
> m\_externalVtkCamera-\>SetViewTransformMatrix(viewMatrix);  
> m\_externalVtkCamera-\>SetProjectionTransformMatrix(projectionMatrix);
> 
> m\_CubeActor-\>RotateX(1.0f);  
> m\_CubeActor-\>RotateY(1.0f);
> 
> m\_ExternalVTKWidget-\>GetRenderWindow()-\>Render();  
> }

When executing this the following OpenGL error pop up

> OPENGL NATIVE PLUG-IN ERROR: GL\_INVALID\_OPERATION: Operation illegal in current state

Any thoughts would be appreciated

Thanks in advance!

---

<div class="post-metadata">

### Author: ![ben.boeckel](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.vtk.org/u/ben.boeckel)
#### Post date: [June 10, 2019, 1:36pm UTC](https://discourse.vtk.org/t/how-to-integrate-vtk-with-unity/1108/2 "2019-06-10T13:36:28Z")

</div>

My guess is that Unity is expecting the external plugin to create the OpenGL context, but I don’t see that happening here in the VTK code. Could you please link the relevant Unity documentation?

Cc: @ken-martin

---

<div class="post-metadata">

### Author: ![Oscar](https://discourse.vtk.org/user_avatar/discourse.vtk.org/oscar/32/1498_2.png) [@Oscar](https://discourse.vtk.org/u/Oscar)
#### Post date: [June 10, 2019, 1:48pm UTC](https://discourse.vtk.org/t/how-to-integrate-vtk-with-unity/1108/3 "2019-06-10T13:48:38Z")

</div>

Unity low level interface plugin doc: [Low-Level interface plugin](https://docs.unity3d.com/Manual/NativePluginInterface.html)

Thanks for your response! It’s a possibility but I don´t think thats the case because in the example provided by Unity in the documentation the c++ code only draws (through glew) on the OpenGL context created by Unity

[Low-level interface plugin example (non VTK rendering)](https://bitbucket.org/Unity-Technologies/graphicsdemos/src/default/)

---

<div class="post-metadata">

### Author: ![Oscar](https://discourse.vtk.org/user_avatar/discourse.vtk.org/oscar/32/1498_2.png) [@Oscar](https://discourse.vtk.org/u/Oscar)
#### Post date: [June 18, 2019, 9:24am UTC](https://discourse.vtk.org/t/how-to-integrate-vtk-with-unity/1108/4 "2019-06-18T09:24:54Z")

</div>

I have a little bit more info about this. In case someone can help me.

I’ve been doing some debugging and i get this error:

> glDrawBuffer failed because ‘0x405’ is not a compatible or supported buffer type under the current framebuffer configuration (GL\_INVALID\_OPERATION)

This error is triggered by the glDrawBuffer in the render function where doublebuffer is active in vtkExternalOpenGLCamera. There:

> ```
> if (ren->GetRenderWindow()->GetDoubleBuffer())
> {
> glDrawBuffer(static_cast<GLenum>(win->GetBackBuffer()));
> 
> // Reading back buffer means back left. see OpenGL spec.
> // because one can write to two buffers at a time but can only read from
> // one buffer at a time.
> glReadBuffer(static_cast<GLenum>(win->GetBackBuffer()));
> }
> 
> ```

I’ve doing some researching about framebuffers, back and front buffer but i’m struggling to understand it properly.

Thanks in advance

---

<div class="post-metadata">

### Author: ![ben.boeckel](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/b/ea5d25/32.png) [@ben.boeckel](https://discourse.vtk.org/u/ben.boeckel)
#### Post date: [June 18, 2019, 11:55am UTC](https://discourse.vtk.org/t/how-to-integrate-vtk-with-unity/1108/5 "2019-06-18T11:55:35Z")

</div>

I’m out of my depth at this at this point too. @ken-martin or @utkarshayachit?

---

<div class="post-metadata">

### Author: ![haochengd](https://discourse.vtk.org/letter_avatar_proxy/v4/letter/h/82dd89/32.png) [@haochengd](https://discourse.vtk.org/u/haochengd)
#### Post date: [September 6, 2019, 5:55am UTC](https://discourse.vtk.org/t/how-to-integrate-vtk-with-unity/1108/6 "2019-09-06T05:55:04Z")

</div>

Hi Oscar!  
I am integrating the VTK with Unity, any luck you have already solve this issue?

---

<div class="post-metadata">

### Author: ![Alberto\_Gomez](https://discourse.vtk.org/user_avatar/discourse.vtk.org/alberto_gomez/32/1156_2.png) [@Alberto\_Gomez](https://discourse.vtk.org/u/Alberto_Gomez)
#### Post date: [December 3, 2019, 11:39am UTC](https://discourse.vtk.org/t/how-to-integrate-vtk-with-unity/1108/7 "2019-12-03T11:39:20Z")

</div>

Hi,

I am one of the authors of the paper on the first post -just wanted to inform that we have released part of the code open source and can be found here:  
[https://gitlab.com/3dheart\_public/vtktounity](https://gitlab.com/3dheart_public/vtktounity)  
If you have any problems please let us know through the issue system in gitlab, or here and we will try to help.

Alberto

---

<div class="post-metadata">

### Author: ![Vikram\_Apilla](https://discourse.vtk.org/user_avatar/discourse.vtk.org/vikram_apilla/32/2142_2.png) [@Vikram\_Apilla](https://discourse.vtk.org/u/Vikram_Apilla)
#### Post date: [July 28, 2020, 9:47am UTC](https://discourse.vtk.org/t/how-to-integrate-vtk-with-unity/1108/8 "2020-07-28T09:47:29Z")

</div>

Hi Alberto, I just wanted to ask, with this framework, is it possible to load .vtp (poly data) files in Unity?

---

<div class="post-metadata">

### Author: ![mwestphal](https://discourse.vtk.org/user_avatar/discourse.vtk.org/mwestphal/32/19_2.png) [@mwestphal](https://discourse.vtk.org/u/mwestphal)
#### Post date: [August 4, 2020, 9:26am UTC](https://discourse.vtk.org/t/how-to-integrate-vtk-with-unity/1108/9 "2020-08-04T09:26:40Z")

</div>

Yes it is.

If you have further question, please open your own topic.

---

<div class="post-metadata">

### Author: ![mwestphal](https://discourse.vtk.org/user_avatar/discourse.vtk.org/mwestphal/32/19_2.png) [@mwestphal](https://discourse.vtk.org/u/mwestphal)
#### Post date: [August 4, 2020, 9:26am UTC](https://discourse.vtk.org/t/how-to-integrate-vtk-with-unity/1108/10 "2020-08-04T09:26:45Z")

</div>