vtk+DearImGui integration through vtkGenericOpenGLRenderWindow

[jaswantp]

I made this post to help ImGui/3rd party UI users who wish to display VTK content in an externally provided context(glad/glbindings/…) and (or) windowing system (glfw/…) or DearImGUI, Nuklear, etc.

So far, there is only one post which discusses procedures to render VTK into user provided frame buffer. It is quite outdated and does not function for VTK >= 9.0, probably since much of VTK’s OpenGL code base has changed.

I’ve updated a fork of the repo to conform with imgui_impl_xxx.h files and vtkGenericOpenGLRenderWindow. It is a straight forward usage for DearImGUI users, but not the case for VTK users.

Alternative:
It would be easier to have a class vtkDearImGuiVport; sub-class vtkGenericOpenGLRenderWindow. In an override to vtkGenericOpenGLRenderWindow::Frame(), access texture id from vtkOpenGLRenderWindow::DrawPixelsTextureObject and pipe it to ImGui::Image(…). This would remove the additional render FBO to texture done in the current repo. But a huge problem arises when StereoRender is off since the internal texture object is not filled in vtkRenderWindow::StereoRenderComplete().

Maybe in a default path inside vtkRenderWindow::StereoRenderComplete(), call vtkOpenGLRenderWindow::GetPixelData(..) to fill in vtkOpenGLRenderWindow::ResultFrame and break away. This will ensure vtkOpenGLRenderWindow::ResultFrame is populated. From then on, vtkOpenGLRenderWindow::DrawPixels will initialize vtkOpenGLRenderWindow::DrawPixelsTextureObject.

DearImGUI Gallery thread
Edit: update links
Edit: Add a link to DearImGui gallery post

[phcerdan (Pablo Hernandez Cerdan)]

Thanks for sharing @jaswantp!

I had a go on integrating vtk+ImGui: GitHub - phcerdan/vtkImGuiAdapter: Adapter to add imgui to an existing vtkRenderWindow using SDL2.

vtkImGuiAdapter provides two classes: vtkImGuiSDL2OpenGLRenderWindow (subclass of the vtkSDL2OpenGLRendeWindow available since 9.0) and vtkImGuiSDL2RenderWindowInteractor.

vtkImGuiAdapter takes the path to override Frame and let ImGui loop control the re-rendering. The render window is owned by VTK, and ImGui writes on it. I am sure optimizations can be done by someone with more OpenGL knowledge than myself to avoid some unnecessary re-renderings.

It has worked for me pretty well to put up some demos really quick.

Here is the code example that generates that gif.

Link to the discourse post where I first introduced it

[jaswantp]

Thanks!

I noticed your post when I first searched vtk + imgui on here and sort of borrowed some parts of your SDL2RenderWindow to implement vtkGlfwRenderWindow

The render window is owned by VTK, and ImGui writes on it.

Right. My implementation does the opposite thing. The window manager(provided by the user!) owns the windows and VTK renders its stuff onto a DearImGUI window.

• The render window is generic, think of it as a dummy window.
• The render window interactor is also generic, defaults to platform-native interactors. (vtkXRenderWindowInteractor on Linux, vtkWin32OpenGLRenderWindowInteractor on win, cocoa on mac)
• The opengl context is user-provided.

This would be suitable for folks who’d like to embed a VTK visualization into an existing DearImGUI app as opposed to embedding DearImGUI windows into a VTK visualization/app.

I originally had to set this up for prototyping purposes. Now, I moved on to using ParaView for everything so I figured it’d be a good idea to push and update that out-dated imgui-vtk repo.

This specific style of embedding OpenGL scenes into DearImGUI is done in almost all indie game-engines. (see Dear ImGUI render to viewport).

[phcerdan (Pablo Hernandez Cerdan)]

Hi @jaswantp, I have failed to run your fork using VTK version: 9.0.20210114. Basically nothing gets rendered inside the VTK sub-window. On the bright side, there are no crashes.

Have you played with it recently? Should I try another VTK version?

[jaswantp]

Hello, This happens when build version >= 2. This is my output for ParaView 5.9.0’s VTK which is 9.0.20201030

I’m surprised you did not see a spam of such OpenGL errors. Perhaps you built VTK to not display OpenGL errors.

If you turned on VTK_REPORT_OPENGL_ERRORS and VTK_REPORT_OPENGL_ERRORS_IN_RELEASE_BUILDS, your console would be spammed with the errors in the image above.

Anyway, it was displaying a blank window because of a stupid typo where I used GL_DRAW_BUFFER instead of GL_DRAW_FRAMEBUFFER in the bind and unbind calls around Render(). This is the invalid enum that OpenGL complains about.

With this fix commit, it should work! Let me know if it doesn’t.

There are three things you could do to get it working.

1. Retry from my repo.
2. vtk 9.0.1
3. vtk 8.2.0. Refer to upstream? It has a fix for backward compatibility up to 8.2.0 that binds the application fbo (g_FBOHdl) to GL_FRAMEBUFFER instead of GL_DRAW_FRAMEBUFFER.

Thanks for bringing this up. I’ll have to keep the source up to date with the constant updates being made to the vtkOpenGLRenderWindow::Frame() since 9.0.0

I’m also interested in your build setup. Did you link with ParaView’s VTK? Or is it bare VTK? Do you use the demo provided context loader i.e, glad? Or does it default to system identified OpenGL loader like GLEW/etc…

[phcerdan (Pablo Hernandez Cerdan)]

That fixes it, thanks!
I didn’t know about VTK_REPORT_OPENGL_ERRORS, really useful .

Do you use the demo provided context loader i.e, glad? Or does it default to system identified OpenGL loader like GLEW/etc

I used bare VTK 9, with SDL2, and GLEW as loader.

By the way, the VTK_BUILD_VERSION in development is a date (i.e 20210114), I suspect
#if VTK_BUILD_VERSION >= 2 isn’t filtering anything.

[jaswantp]

Hi @phcerdan

I didn’t know about VTK_REPORT_OPENGL_ERRORS , really useful .

Right :), it’s the only way to find errors in OpenGL with vtk.

I used bare VTK 9, with SDL2, and GLEW as loader.

Good to know that DearImGUI from demo app was able to render itself into a context provided by GLEW.

By the way, the VTK_BUILD_VERSION in development is a date (i.e 20210114 ), I suspect
#if VTK_BUILD_VERSION >= 2 isn’t filtering anything.

I initially used it as a hack to separate logic for anything <= 9.0.1. This commit, post v9.0.1 introduces a render frame buffer. From what I understood in the source, it required changing the enum argument of glBindFrameBuffer in my code.

So, GL_FRAMEBUFFER(for <= 9.0.1) → GL_DRAW_FRAMEBUFFER (for anything > 9.0.1)).

This will let VTK blit its render frame buffer into our frame buffer which will be rendered onto a texture eventually displayed inside a DearImGUI window.

[jaswantp]

@phcerdan , if you’re interested, do you mind giving this a spin with your VTK build?

Note: Clone the imgui submodule too.

I made a rapid prototype that intercepts interactive events from VTK and injects the overlay right before Frame is called.

The important change is that

1. requires no 3rd party windowing system!
2. No need to render to texture! It uses the same render buffer that VTK will use as a read-buffer and blit into display buffer. This blit occurs in Frame call.

For me it works on windows with VTK master branch.

If you see the test_imgui_vtk.cxx file, injection is primarily done with one line.

dearImGUIinjector->Inject(interactor)

[phcerdan (Pablo Hernandez Cerdan)]

Hi @jaswantp,
I just tested it in Linux, It compiles fine.
Tested using two versions of VTK 9:

• commit 48f8d11f2b79e552fc25d1ab407f9881b51dce0a (15 April 2021)
• VTK 9 master (1 Jun 2021).

In both cases, the executable test_imgui_vtk starts fine, but when interacting with the VTK rendering, an assertion from imgui pops up:

test_imgui_vtk: vtkDearImGUIInjector/src/imgui/imgui.cpp:7154: void ImGui::ErrorCheckNewFrameSanityChecks(): Assertion `(g.IO.DeltaTime > 0.0f || g.FrameCount == 0) && "Need a positive DeltaTime!"' failed.
[1]    3015218 abort (core dumped)  ./test_imgui_vtk

When building in Release mode, once I click and start interacting with the VTK scene, I am not able to keep interacting with it (see gif).

There are no OpenGL errors reported with VTK_REPORT_OPENGL_ERRORS and VTK_REPORT_OPENGL_ERRORS_IN_RELEASE_BUILD both ON.

@jaswantp

• requires no 3rd party windowing system!
• No need to render to texture! It uses the same render buffer that VTK will use as a read-buffer and blit into display buffer. This blit occurs in Frame call.

Looking good though!

[jaswantp]

I haven’t yet tested it on Linux though. So yes, there might be some bugs there. I’m developing it on Windows using VS2019 community. I prefer this IDE for its simple and usable debugger.

I wish to develop this into a state explorer for VTK based apps. Essentially, traverse the vtk pipeline from renderer > actors > mappers > algorithms/sources > inputs.

I’d imagine ImGui tree widgets/tables that expand to show the properties available with vtkGet/vtkSet macros in real time and possibly small textures of the output before and after a filter is applied. For the renderer and all actors, I want to display and set the transform matrices, possibly with sliders. I hope this would improve my understanding of matrix transformations.

On windows, one could technically inject a line of code dearImGUIinjector->Inject(interactor) into paraview.exe process and use the interactor object of ParaView’s active RenderView, possibly a QVTKInteractor. Since ParaView is open source, it makes it a little bit easier to do dll injection.

I don’t know how long it’s gonna take, but I’d like to see this happen eventually.

[jaswantp]

@phcerdan

When building in Release mode, once I click and start interacting with the VTK scene, I am not able to keep interacting with it (see gif).

This is by design. ImGUI stops forwarding mouse events to VTK when the mouse is inside imgui window. But I think this is not expected of an overlay! Thanks for mentioning that! I will add a checkbox to grab mouse input.

About the imgui assertion error, I’ll debug it on a linux box and see what I can do… maybe it has to do with the implementation of chrono::system_clock on nix vs windows c++ libraries.

[jaswantp]

Alright, I debugged it over the weekend and my assumptions were partly right.

Bug: On nix platform, Render function was called inconsistently, so io.DeltaTime was <= 0ms

Cause: Since VTK doesn’t call Render in every iteration of the event loop, I had to create a one-shot timer for 0ms so that the event queue will always have a TimerEvent that will call my render function. This is to make sure the overlay is drawn on every frame. Apparently, the x11 Render window interactor doesn’t behave well with 0ms timers, as a matter of fact, it doesn’t even fire the timer.

I’ll have to delegate vtkDearImGuiInjector to call ProcessEvents effectively taking over the event loop. This could be achieved if interactor framework exposed HandleEventLoop via vtkBooleanMacro. I want to do this without deriving vtkRenderWindowInteractor for simplicity at the injection site. Since it’s default value is false, its okay for now!

[jaswantp]

The link to the repo has changed

[jaswantp]

I spent a good weekend debugging the Emscripten side of things in VTK. In the end, I got this to compile with Emscripten for WebAssembly. The simple cone example with DearImGUI. This uses GLES 3.0, some VTK functionality will not work…