# vtk+DearImGui integration through vtkGenericOpenGLRenderWindow
**URL:** https://discourse.vtk.org/t/vtk-dearimgui-integration-through-vtkgenericopenglrenderwindow/4424
**Category:** Development
**Tags:** proposal
**Created:** [October 15, 2020, 6:14am UTC](https://discourse.vtk.org/t/vtk-dearimgui-integration-through-vtkgenericopenglrenderwindow/4424 "2020-10-15T06:14:03Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@jaswantp](https://discourse.vtk.org/u/jaswantp)
#### Post date: [October 15, 2020, 6:14am UTC](https://discourse.vtk.org/t/vtk-dearimgui-integration-through-vtkgenericopenglrenderwindow/4424/1 "2020-10-15T06:14:04Z")
I made this post to help ImGui/3rd party UI users who wish to display VTK content in an externally provided context(glad/glbindings/…) and (or) windowing system (glfw/…) or DearImGUI, Nuklear, etc.
So far, there is only one [post](http://vtk.1045678.n5.nabble.com/how-to-render-VTK-into-user-provided-FBO-td5749371.html) which discusses procedures to render VTK into user provided frame buffer. It is quite outdated and does not function for VTK \>= 9.0, probably since much of VTK’s OpenGL code base has changed.
I’ve updated a [fork](https://github.com/jaswantp/imgui-vtk) of the repo to conform with imgui\_impl\_xxx.h files and vtkGenericOpenGLRenderWindow. It is a straight forward usage for DearImGUI users, but not the case for VTK users.
Alternative:
It would be easier to have a class vtkDearImGuiVport; sub-class [`vtkGenericOpenGLRenderWindow`](https://vtk.org/doc/nightly/html/classvtkGenericOpenGLRenderWindow.html). In an override to [`vtkGenericOpenGLRenderWindow::Frame()`](https://vtk.org/doc/nightly/html/classvtkGenericOpenGLRenderWindow.html#a571ceb6d2d816ad0178350c01194fd26), access texture id from [`vtkOpenGLRenderWindow::DrawPixelsTextureObject`](https://github.com/Kitware/VTK/blob/master/Rendering/OpenGL2/vtkOpenGLRenderWindow.cxx#L144) and pipe it to ImGui::Image(…). This would remove the additional render FBO to texture done in the current repo. But a huge problem arises when StereoRender is off since the internal texture object is not filled in [`vtkRenderWindow::StereoRenderComplete()`](https://github.com/Kitware/VTK/blob/master/Rendering/Core/vtkRenderWindow.cxx#L497).
Maybe in a default path inside [`vtkRenderWindow::StereoRenderComplete()`](https://github.com/Kitware/VTK/blob/master/Rendering/Core/vtkRenderWindow.cxx#L497), call [`vtkOpenGLRenderWindow::GetPixelData(..)`](https://github.com/Kitware/VTK/blob/master/Rendering/OpenGL2/vtkOpenGLRenderWindow.cxx#L701) to fill in [`vtkOpenGLRenderWindow::ResultFrame`](https://github.com/Kitware/VTK/blob/master/Rendering/Core/vtkRenderWindow.cxx#L65) and break away. This will ensure [`vtkOpenGLRenderWindow::ResultFrame`](https://github.com/Kitware/VTK/blob/master/Rendering/Core/vtkRenderWindow.cxx#L65) is populated. From then on, [vtkOpenGLRenderWindow::DrawPixels](https://github.com/Kitware/VTK/blob/master/Rendering/OpenGL2/vtkOpenGLRenderWindow.cxx#L1233) will initialize [`vtkOpenGLRenderWindow::DrawPixelsTextureObject`](https://github.com/Kitware/VTK/blob/master/Rendering/OpenGL2/vtkOpenGLRenderWindow.cxx#L144).
DearImGUI Gallery [thread](https://github.com/ocornut/imgui/issues/3488#issuecomment-709008332)
Edit: update links
Edit: Add a link to DearImGui gallery post
---
### Author:  [@phcerdan](https://discourse.vtk.org/u/phcerdan)
#### Post date: [March 22, 2021, 4:03pm UTC](https://discourse.vtk.org/t/vtk-dearimgui-integration-through-vtkgenericopenglrenderwindow/4424/2 "2021-03-22T16:03:42Z")
Thanks for sharing @jaswantp!
I had a go on integrating vtk+ImGui: [GitHub - phcerdan/vtkImGuiAdapter: Adapter to add imgui to an existing vtkRenderWindow using SDL2.](https://github.com/phcerdan/vtkImGuiAdapter/)
vtkImGuiAdapter provides two classes: `vtkImGuiSDL2OpenGLRenderWindow` (subclass of the `vtkSDL2OpenGLRendeWindow` available since 9.0) and `vtkImGuiSDL2RenderWindowInteractor`.
vtkImGuiAdapter takes the path to override `Frame` and let ImGui loop control the re-rendering. The render window is owned by VTK, and ImGui writes on it. I am sure optimizations can be done by someone with more OpenGL knowledge than myself to avoid some unnecessary re-renderings.
It has worked for me pretty well to put up some demos really quick.
Here is the [code example](https://github.com/phcerdan/vtkImGuiAdapter/blob/6552628028d4fddd36485e41ca787453f27e679c/example-imgui-vtk-sdl.cpp) that generates that gif.
[Link to the discourse post](https://discourse.vtk.org/t/interactive-text-widget-search-box/4009/5) where I first introduced it
---
### Author:  [@jaswantp](https://discourse.vtk.org/u/jaswantp)
#### Post date: [March 22, 2021, 4:32pm UTC](https://discourse.vtk.org/t/vtk-dearimgui-integration-through-vtkgenericopenglrenderwindow/4424/3 "2021-03-22T16:32:47Z")
Thanks!
I noticed your post when I first searched vtk + imgui on here and sort of borrowed some parts of your SDL2RenderWindow to implement [vtkGlfwRenderWindow](https://github.com/jaswantp/vtkGlfw) 🙂
> The render window is owned by VTK, and ImGui writes on it.
Right. My implementation does the opposite thing. The window manager(provided by the user!) owns the windows and VTK renders its stuff onto a DearImGUI window.
- The render window is generic, think of it as a dummy window.
- The render window interactor is also generic, defaults to platform-native interactors. (vtkXRenderWindowInteractor on Linux, vtkWin32OpenGLRenderWindowInteractor on win, cocoa on mac)
- The opengl context is user-provided.
This would be suitable for folks who’d like to embed a VTK visualization into an existing DearImGUI app as opposed to embedding DearImGUI windows into a VTK visualization/app.
I originally had to set this up for prototyping purposes. Now, I moved on to using ParaView for everything so I figured it’d be a good idea to push and update that out-dated imgui-vtk repo.
This specific style of embedding OpenGL scenes into DearImGUI is done in almost all indie game-engines. (see Dear ImGUI [render to viewport](https://gamedev.stackexchange.com/questions/140693/how-can-i-render-an-opengl-scene-into-an-imgui-window)).
---
### Author:  [@phcerdan](https://discourse.vtk.org/u/phcerdan)
#### Post date: [March 22, 2021, 8:07pm UTC](https://discourse.vtk.org/t/vtk-dearimgui-integration-through-vtkgenericopenglrenderwindow/4424/4 "2021-03-22T20:07:28Z")
Hi @jaswantp, I have failed to run your fork using VTK version: `9.0.20210114`. Basically nothing gets rendered inside the VTK sub-window. On the bright side, there are no crashes.
Have you played with it recently? Should I try another VTK version?
---
### Author:  [@jaswantp](https://discourse.vtk.org/u/jaswantp)
#### Post date: [March 23, 2021, 1:16am UTC](https://discourse.vtk.org/t/vtk-dearimgui-integration-through-vtkgenericopenglrenderwindow/4424/5 "2021-03-23T01:16:20Z")
Hello, This happens when build version \>= 2. This is my output for ParaView 5.9.0’s VTK which is `9.0.20201030`
I’m surprised you did not see a spam of such OpenGL errors. Perhaps you built VTK to not display OpenGL errors.
If you turned on `VTK_REPORT_OPENGL_ERRORS` and `VTK_REPORT_OPENGL_ERRORS_IN_RELEASE_BUILDS`, your console would be spammed with the errors in the image above.
Anyway, it was displaying a blank window because of a stupid typo where I used `GL_DRAW_BUFFER` instead of `GL_DRAW_FRAMEBUFFER` in the bind and unbind calls around `Render()`. This is the invalid enum that OpenGL complains about.
With [this](https://github.com/jaswantp/imgui-vtk/commit/4b0eb0b46dc6ca30c2a19c782a7e48ee89984e9a) fix commit, it should work! 🙂 Let me know if it doesn’t.
There are three things you could do to get it working.
1. Retry from my repo.
2. vtk 9.0.1
3. vtk 8.2.0. Refer to [upstream](https://github.com/trlsmax/imgui-vtk)? It has a [fix](https://github.com/trlsmax/imgui-vtk/commit/6bd8b7541416182372def064e5b142495e6ed69c#diff-27df39e680c4c670b084f136944951ba3e680ee6e16c353177d511a4b3e4af3bR244) for backward compatibility up to 8.2.0 that binds the application fbo (`g_FBOHdl`) to `GL_FRAMEBUFFER` instead of `GL_DRAW_FRAMEBUFFER`.
Thanks for bringing this up. I’ll have to keep the source up to date with the constant updates being made to the `vtkOpenGLRenderWindow::Frame()` since 9.0.0
I’m also interested in your build setup. Did you link with ParaView’s VTK? Or is it bare VTK? Do you use the demo provided context loader i.e, glad? Or does it default to system identified OpenGL loader like GLEW/etc…
---
### Author:  [@phcerdan](https://discourse.vtk.org/u/phcerdan)
#### Post date: [March 23, 2021, 9:12am UTC](https://discourse.vtk.org/t/vtk-dearimgui-integration-through-vtkgenericopenglrenderwindow/4424/6 "2021-03-23T09:12:19Z")
That fixes it, thanks! 🥳
I didn’t know about `VTK_REPORT_OPENGL_ERRORS`, really useful 🙂.
> Do you use the demo provided context loader i.e, glad? Or does it default to system identified OpenGL loader like GLEW/etc
I used bare VTK 9, with SDL2, and GLEW as loader.
By the way, the VTK\_BUILD\_VERSION in development is a date (i.e `20210114`), I suspect
`#if VTK_BUILD_VERSION >= 2` isn’t filtering anything.
---
### Author:  [@jaswantp](https://discourse.vtk.org/u/jaswantp)
#### Post date: [March 23, 2021, 9:41am UTC](https://discourse.vtk.org/t/vtk-dearimgui-integration-through-vtkgenericopenglrenderwindow/4424/7 "2021-03-23T09:41:32Z")
Hi @phcerdan
> I didn’t know about `VTK_REPORT_OPENGL_ERRORS` , really useful 🙂.
Right :), it’s the only way to find errors in OpenGL with vtk.
> I used bare VTK 9, with SDL2, and GLEW as loader.
Good to know that DearImGUI from demo app was able to render itself into a context provided by GLEW.
> By the way, the VTK\_BUILD\_VERSION in development is a date (i.e `20210114` ), I suspect
> `#if VTK_BUILD_VERSION >= 2` isn’t filtering anything.
I initially used it as a hack to separate logic for anything \<= 9.0.1. This [commit](https://gitlab.kitware.com/vtk/vtk/-/commit/588247020e455b9c3f7bf37a64525c31c5071f52), post v9.0.1 introduces a render frame buffer. From what I understood in the source, it required changing the enum argument of glBindFrameBuffer in my code.
So, GL\_FRAMEBUFFER(for \<= 9.0.1) → GL\_DRAW\_FRAMEBUFFER (for anything \> 9.0.1)).
This will let VTK blit its render frame buffer into our frame buffer which will be rendered onto a texture eventually displayed inside a DearImGUI window.
---
### Author:  [@jaswantp](https://discourse.vtk.org/u/jaswantp)
#### Post date: [May 29, 2021, 6:25am UTC](https://discourse.vtk.org/t/vtk-dearimgui-integration-through-vtkgenericopenglrenderwindow/4424/8 "2021-05-29T06:25:56Z")
@phcerdan , if you’re interested, do you mind giving [this](https://github.com/jaswantp/vtkDearImGUIInjector) a spin with your VTK build?
Note: Clone the imgui submodule too.
I made a rapid prototype that intercepts interactive events from VTK and injects the overlay right before Frame is called.
The important change is that
1. requires no 3rd party windowing system!
2. No need to render to texture! It uses the same render buffer that VTK will use as a read-buffer and blit into display buffer. This blit occurs in `Frame` call.
For me it works on windows with VTK master branch.
If you see the test\_imgui\_vtk.cxx file, injection is primarily done with one line.
```auto
dearImGUIinjector->Inject(interactor)
```
---
### Author:  [@phcerdan](https://discourse.vtk.org/u/phcerdan)
#### Post date: [June 1, 2021, 12:45pm UTC](https://discourse.vtk.org/t/vtk-dearimgui-integration-through-vtkgenericopenglrenderwindow/4424/9 "2021-06-01T12:45:02Z")
Hi @jaswantp,
I just tested it in Linux, It compiles fine.
Tested using two versions of VTK 9:
- commit 48f8d11f2b79e552fc25d1ab407f9881b51dce0a (15 April 2021)
- VTK 9 master (1 Jun 2021).
In both cases, the executable `test_imgui_vtk` starts fine, but when interacting with the VTK rendering, an assertion from imgui pops up:
```cpp
test_imgui_vtk: vtkDearImGUIInjector/src/imgui/imgui.cpp:7154: void ImGui::ErrorCheckNewFrameSanityChecks(): Assertion `(g.IO.DeltaTime > 0.0f || g.FrameCount == 0) && "Need a positive DeltaTime!"' failed.
[1] 3015218 abort (core dumped) ./test_imgui_vtk
```
When building in Release mode, once I click and start interacting with the VTK scene, I am not able to keep interacting with it (see gif).
There are no OpenGL errors reported with `VTK_REPORT_OPENGL_ERRORS` and `VTK_REPORT_OPENGL_ERRORS_IN_RELEASE_BUILD` both `ON`.
> [@jaswantp](#):
>
> - requires no 3rd party windowing system!
> - No need to render to texture! It uses the same render buffer that VTK will use as a read-buffer and blit into display buffer. This blit occurs in `Frame` call.
Looking good though!
---
### Author:  [@jaswantp](https://discourse.vtk.org/u/jaswantp)
#### Post date: [June 1, 2021, 3:48pm UTC](https://discourse.vtk.org/t/vtk-dearimgui-integration-through-vtkgenericopenglrenderwindow/4424/10 "2021-06-01T15:48:41Z")
I haven’t yet tested it on Linux though. So yes, there might be some bugs there. I’m developing it on Windows using VS2019 community. I prefer this IDE for its simple and usable debugger.
I wish to develop this into a state explorer for VTK based apps. Essentially, traverse the vtk pipeline from renderer \> actors \> mappers \> algorithms/sources \> inputs.
I’d imagine ImGui tree widgets/tables that expand to show the properties available with vtkGet/vtkSet macros in real time and possibly small textures of the output before and after a filter is applied. For the renderer and all actors, I want to display and set the transform matrices, possibly with sliders. I hope this would improve my understanding of matrix transformations.
On windows, one could technically inject a line of code `dearImGUIinjector->Inject(interactor)` into paraview.exe process and use the interactor object of ParaView’s active RenderView, possibly a QVTKInteractor. Since ParaView is open source, it makes it a little bit easier to do dll injection.
I don’t know how long it’s gonna take, but I’d like to see this happen eventually.
---
### Author:  [@jaswantp](https://discourse.vtk.org/u/jaswantp)
#### Post date: [June 5, 2021, 10:23am UTC](https://discourse.vtk.org/t/vtk-dearimgui-integration-through-vtkgenericopenglrenderwindow/4424/11 "2021-06-05T10:23:22Z")
> [@phcerdan](#):
>
> When building in Release mode, once I click and start interacting with the VTK scene, I am not able to keep interacting with it (see gif).
This is by design. ImGUI stops forwarding mouse events to VTK when the mouse is inside imgui window. But I think this is not expected of an overlay! Thanks for mentioning that! I will add a checkbox to grab mouse input.
About the imgui assertion error, I’ll debug it on a linux box and see what I can do… maybe it has to do with the implementation of `chrono::system_clock` on nix vs windows c++ libraries.
---
### Author:  [@jaswantp](https://discourse.vtk.org/u/jaswantp)
#### Post date: [June 6, 2021, 10:21am UTC](https://discourse.vtk.org/t/vtk-dearimgui-integration-through-vtkgenericopenglrenderwindow/4424/12 "2021-06-06T10:21:26Z")
Alright, I debugged it over the weekend and my assumptions were partly right.
_Bug_: On nix platform, `Render` function was called inconsistently, so io.DeltaTime was \<= 0ms
_Cause_: Since VTK doesn’t call Render in every iteration of the event loop, I had to create a one-shot timer for 0ms so that the event queue will always have a TimerEvent that will call my render function. This is to make sure the overlay is drawn on every frame. Apparently, the x11 Render window interactor doesn’t behave well with 0ms timers, as a matter of fact, it doesn’t even fire the timer.
I’ll have to delegate vtkDearImGuiInjector to call ProcessEvents effectively taking over the event loop. This could be achieved if interactor framework exposed [HandleEventLoop](https://vtk.org/doc/nightly/html/classvtkRenderWindowInteractor.html#a26f99ef93b03a2935d14c22ceb96b00d) via vtkBooleanMacro. I want to do this without deriving `vtkRenderWindowInteractor` for simplicity at the injection site. Since it’s default value is false, its okay for now!
---
### Author:  [@jaswantp](https://discourse.vtk.org/u/jaswantp)
#### Post date: [July 23, 2021, 10:09am UTC](https://discourse.vtk.org/t/vtk-dearimgui-integration-through-vtkgenericopenglrenderwindow/4424/13 "2021-07-23T10:09:39Z")
The link to the repo has [changed](https://github.com/jspanchu/vtkDearImGUIInjector)
---
### Author:  [@jaswantp](https://discourse.vtk.org/u/jaswantp)
#### Post date: [October 10, 2021, 11:22pm UTC](https://discourse.vtk.org/t/vtk-dearimgui-integration-through-vtkgenericopenglrenderwindow/4424/14 "2021-10-10T23:22:27Z")
I spent a good weekend debugging the Emscripten side of things in VTK. In the end, I got this to compile with Emscripten for WebAssembly. The simple [cone example](https://code.thepanchu.in/vtkDearImGUIInjector/) with DearImGUI. This uses GLES 3.0, some VTK functionality will not work…
---
### Author:  [@vvttkk](https://discourse.vtk.org/u/vvttkk)
#### Post date: [May 6, 2022, 4:06am UTC](https://discourse.vtk.org/t/vtk-dearimgui-integration-through-vtkgenericopenglrenderwindow/4424/15 "2022-05-06T04:06:58Z")
Hi,
I am using my c++14 project in vs2019 to run vtkImGuiAdapter-project, there is #include \ specified in example-imgui-vtk-sdl
In order to use gl3.h, I have downloaded windows 64bit gles3.0.4 and 3.0.2 and 32bit 3.0.2 from [OpenGL ES Emulator – Arm Developer](https://developer.arm.com/downloads/-/opengl-es-emulator-downloads) and make setting in the example-imgui-vtk-sdl to use the headers, lib, and dll files. However when I ALL\_BUILD -\>build the project, there is this same error in libEGL.dll for all gles downloaded, the error is
```
'invalid or corrupt file: cannot read at 0x328'
```
The same error encountered when I use libEGL.dll (ver 2.1, 64bit) downloaded from [libegl.dll free download | DLL‑files.com](https://www.dll-files.com/libegl.dll.html)
I am using Windows SDK version 10.0.20348.0 for Windows 10.0.19044, would like to find out how GLES3 is set up in windows 10 environment .