Rendering/interaction issues with VTK 8.2 (Java) + JOGL on Windows

Hi all.
I am using VTK 8.2 from a Java application.

I am trying to use JOGL as a mechanism to have VTK render within my Java application, in the effort of having a cross-platform rendering pipeline working also under MacOS (MacOS build will be my next effort ;-)), in addition to under Windows and maybe Linux.

However, as of now, I am testing on Windows. So, my current setup is:

  • Windows 10 64-bit
  • Java 8
  • VTK 8.2, built from source with Java wrapping and JOGL
  • JOGAMP v2.3.2

In my application’s code (medical domain), I have a vtkJoglCanvasComponent-derived panel which takes care of rendering some vtkImagePlaneWidgets which reslice a volume coming from a CT or MR scan.

I am successful in having the widgets showing on screen:

However, interaction with this panel seems to be very “slow”, or at least to have a very low refresh rate. In particular, if I drag with my left mouse button on the view, which shall move/rotate the camera, the view updates once every some seconds… Actually, it seems to update as soon as I stop moving the mouse cursor (while still keeping mouse button down). Is there some kind of “lazy repaint timer” or something similar which can be tweaked?
Or what else could cause this “lazy” update/repaint behavior?

There are also some events in my application which causes some “Render()” calls to be invoked on my vtkJoglCanvasComponent-derived panel, and even in this case the repaint is done in an extremely lazy / delayed way. Can you guess why? Any suggestions on what I could try?

A second (and even more “radical”) issue. If on my scene I add an orientation marker widget, then the whole application crashes as soon as I start interacting (i.e., left-dragging) with the scene. Here is the code I use to add my orientation marker widget on my vtkJoglCanvasComponent-derived panel:

protected void buildOrientationMarkerWidget()
{
// Orientation marker
orientMarkerWidget = new vtkOrientationMarkerWidget();

orientMarkerCubeProp = new vtkAnnotatedCubeActor();
orientMarkerCubeProp.SetXMinusFaceText("R");
orientMarkerCubeProp.SetXPlusFaceText("L");
orientMarkerCubeProp.SetYMinusFaceText("A");
orientMarkerCubeProp.SetYPlusFaceText("P");
orientMarkerCubeProp.SetZMinusFaceText("I");
orientMarkerCubeProp.SetZPlusFaceText("S");
double[] darkGray = {0.2, 0.2, 0.2};
double[] almostWhite = {0.9, 0.9, 0.9};
orientMarkerCubeProp.GetCubeProperty().SetOpacity(0); // The "embedded" cube shall be transparente
orientMarkerCubeProp.GetXPlusFaceProperty().SetColor(darkGray);
orientMarkerCubeProp.GetXPlusFaceProperty().SetEdgeColor(darkGray);
orientMarkerCubeProp.GetXMinusFaceProperty().SetColor(darkGray);
orientMarkerCubeProp.GetXMinusFaceProperty().SetEdgeColor(darkGray);
orientMarkerCubeProp.GetYPlusFaceProperty().SetColor(darkGray);
orientMarkerCubeProp.GetYPlusFaceProperty().SetEdgeColor(darkGray);
orientMarkerCubeProp.GetYMinusFaceProperty().SetColor(darkGray);
orientMarkerCubeProp.GetYMinusFaceProperty().SetEdgeColor(darkGray);
orientMarkerCubeProp.GetZPlusFaceProperty().SetColor(almostWhite);
orientMarkerCubeProp.GetZPlusFaceProperty().SetEdgeColor(almostWhite);
orientMarkerCubeProp.GetZPlusFaceProperty().SetEdgeVisibility(0);
orientMarkerCubeProp.GetZMinusFaceProperty().SetColor(almostWhite);
orientMarkerCubeProp.GetZMinusFaceProperty().SetEdgeColor(almostWhite);
orientMarkerCubeProp.GetZMinusFaceProperty().SetEdgeVisibility(0);

vtkCubeSource cubeSource = new vtkCubeSource();
cubeSource.Update();

double[] white = {255, 255, 255};
double[] grayCyan = {145, 205, 225};
double[] darkRed = {200, 2, 2};
vtkUnsignedCharArray faceColors = new vtkUnsignedCharArray();
faceColors.SetNumberOfComponents(3);
faceColors.InsertNextTuple3(white[0], white[1], white[2]);
faceColors.InsertNextTuple3(white[0], white[1], white[2]);
faceColors.InsertNextTuple3(grayCyan[0], grayCyan[1], grayCyan[2]);
faceColors.InsertNextTuple3(grayCyan[0], grayCyan[1], grayCyan[2]);
faceColors.InsertNextTuple3(darkRed[0], darkRed[1], darkRed[2]);
faceColors.InsertNextTuple3(darkRed[0], darkRed[1], darkRed[2]);

cubeSource.GetOutput().GetCellData().SetScalars(faceColors);
cubeSource.Update();

vtkPolyDataMapper cubeMapper = new vtkPolyDataMapper();
cubeMapper.SetInputData(cubeSource.GetOutput());
cubeMapper.Update();

vtkActor cubeActor = new vtkActor();
cubeActor.SetMapper(cubeMapper);

vtkPropAssembly propAssembly = new vtkPropAssembly();
propAssembly.AddPart(orientMarkerCubeProp);
propAssembly.AddPart(cubeActor);

orientMarkerWidget.SetOutlineColor( darkGray[0], darkGray[1], darkGray[2]/*0.9300, 0.5700, 0.1300*/ );
orientMarkerWidget.SetOrientationMarker(propAssembly/*orientMarkerCubeProp*/);
orientMarkerWidget.SetInteractor(getRenderWindowInteractor());
orientMarkerWidget.SetViewport(0, 0.4, 0.2, 0.6);
orientMarkerWidget.SetEnabled(1);
orientMarkerWidget.InteractiveOff();

}

In particular, the initial scene appears correctly. But then, as soon as I start left-dragging on the scene, I get this “debug assertion” (I have built VTK 8.2 in debug mode for now), and the application crashes:

Any idea on what could be causing this? Am I missing to setup something in my code?

An important remark: before switching to JOGL-based rendering, I had implemented exactly the same panel with “direct” rendering from VTK to my AWT panel (through the JAWT interface), and it is working perfectly under Windows. However, I would like to switch to JOGL due to the need to make it work also under MacOS (JAWT-based rendering from VTK won’t work with the current version of JRE / MacOS).

Any feedback will be appreciated.

Thanks and regards,

Marco Sambin

Your texture object is not bound to the active texture image unit.
image
Scaning the vtk source and adding some break points might shed some light on why.

Hi Todd,

thank you for your reply.
Tomorrow I will try to put my Visual Studio Debugger at work. I am not an OpenGL expert, but I will try to understand what happens, and report back.

And what about the interaction / repaint slowness with my vtkJoglCanvasComponent-derived panel? Any ideas?

Thanks and best regards,

Marco

Hi all.
I have some more data to share.
First of all, here is the “native calls” stack trace of the call which causes the debug assertion:

=================

|>|vtkRenderingOpenGL2-8.2.dll!vtkTextureObject::SendParameters() Line 502|C++|
|---|---|---|
| |vtkRenderingOpenGL2-8.2.dll!vtkTextureObject::Bind() Line 449|C++|
| |vtkRenderingOpenGL2-8.2.dll!vtkTextureObject::Activate() Line 389|C++|
| |vtkRenderingOpenGL2-8.2.dll!vtkOpenGLPolyDataMapper::RenderPieceStart(vtkRenderer * ren, vtkActor * actor) Line 2473|C++|
| |vtkRenderingOpenGL2-8.2.dll!vtkOpenGLPolyDataMapper::RenderPiece(vtkRenderer * ren, vtkActor * actor) Line 2630|C++|
| |vtkRenderingCore-8.2.dll!vtkPolyDataMapper::Render(vtkRenderer * ren, vtkActor * act) Line 69|C++|
| |vtkRenderingOpenGL2-8.2.dll!vtkOpenGLActor::Render(vtkRenderer * ren, vtkMapper * mapper) Line 107|C++|
| |vtkRenderingCore-8.2.dll!vtkActor::RenderOpaqueGeometry(vtkViewport * vp) Line 196|C++|
| |vtkRenderingCore-8.2.dll!vtkPropAssembly::RenderOpaqueGeometry(vtkViewport * ren) Line 191|C++|
| |vtkRenderingCore-8.2.dll!vtkRenderer::UpdateOpaquePolygonalGeometry() Line 742|C++|
| |vtkRenderingCore-8.2.dll!vtkRenderer::DeviceRenderOpaqueGeometry() Line 432|C++|
| |vtkRenderingOpenGL2-8.2.dll!vtkOpenGLRenderer::DeviceRenderOpaqueGeometry() Line 435|C++|
| |vtkRenderingOpenGL2-8.2.dll!vtkOpenGLRenderer::UpdateGeometry() Line 323|C++|
| |vtkRenderingOpenGL2-8.2.dll!vtkOpenGLRenderer::DeviceRender() Line 236|C++|
| |vtkRenderingCore-8.2.dll!vtkRenderer::Render() Line 372|C++|
| |vtkRenderingCore-8.2.dll!vtkRendererCollection::Render() Line 53|C++|
| |vtkRenderingCore-8.2.dll!vtkRenderWindow::DoStereoRender() Line 333|C++|
| |vtkRenderingCore-8.2.dll!vtkRenderWindow::Render() Line 292|C++|
| |vtkRenderingOpenGL2-8.2.dll!vtkOpenGLRenderWindow::Render() Line 2565|C++|
| |vtkRenderingOpenGL2-8.2.dll!vtkGenericOpenGLRenderWindow::Render() Line 245|C++|
| |vtkRenderingCore-8.2.dll!vtkRenderWindowInteractor::Render() Line 189|C++|
| |vtkInteractionStyle-8.2.dll!vtkInteractorStyleTrackballCamera::Rotate() Line 265|C++|
| |vtkInteractionStyle-8.2.dll!vtkInteractorStyleTrackballCamera::OnMouseMove() Line 47|C++|
| |vtkRenderingCore-8.2.dll!vtkInteractorStyle::ProcessEvents(vtkObject * __formal, unsigned long event, void * clientdata, void * calldata) Line 1083|C++|
| |vtkCommonCore-8.2.dll!vtkCallbackCommand::Execute(vtkObject * caller, unsigned long event, void * callData) Line 43|C++|
| |vtkCommonCore-8.2.dll!vtkSubjectHelper::InvokeEvent(unsigned long event, void * callData, vtkObject * self) Line 572|C++|
| |vtkCommonCore-8.2.dll!vtkObject::InvokeEvent(unsigned long event, void * callData) Line 785|C++|
| |vtkRenderingCore-8.2.dll!vtkRenderWindowInteractor::MouseMoveEvent() Line 1039|C++|
| |vtkRenderingCoreJava.dll!Java_vtk_vtkRenderWindowInteractor_MouseMoveEvent_1123(JNIEnv_ * env, _jobject * obj) Line 1392|C++|

=================

As obvious, the assertion fails due to the fact vtkTextureObject::IsBound() returns false. A single step inside the IsBound() method reveals that in this specific case this method is called with this->Target==GL_TEXTURE_BUFFER.
However, inside the IsBound() method, the final call to “glGetIntegerv(target,&objectId);” seems to fail, and in fact, the value assigned to the “result” variable is “false”:

GLint objectId;
    glGetIntegerv(target,&objectId);
    result=static_cast<GLuint>(objectId)==this->Handle;

The net effect is that IsBound() returns false, the debug assertion fails, and the application final crashes.

Do all this means something to you, OpenGL experts? Unfortunately, I am not one of those.

Thanks for your cooperation and best regards,

Marco Sambin

You probably don’t have an initialised OpenGL context.

Maybe something like this
GLU.createGLU(glu);
gLCanvas.getContext().makeCurrent();

Hi Todd,

thanks a lot for your feedback, I appreciate, and I will try to apply your suggestion.

However, my question is: considering that I only obtain this issue (debug assertion + crash) when I start interacting with my scene through a left mouse drag, while everything works fine if I trigger new “Render()” from my application, e.g., following to a move / rotation of the cut planes made on other non-VTK panels of my application, where would you put the code that you suggest above?

Also, can you guess why this “binding” issue between the texture object and the active texture image unit comes out only when I drag the scene, while it doesn’t come out when I modify and re-render the scene programmatically, as the effect of other operations done on other panels of my application?

Thank you very much for your help.
Best regards,

Marco Sambin

Ok. So it only crashes on rotation WITH the orientation marker. Presumably that’s because the marker uses a texture and it tries to bind when no OpenGL context is set. Did you check whether the Bind() method is called when there is no marker?

Hello Todd,

yes, it is correct, I get the debug assertion + crash ONLY if I have the orientation marker widget in my scene.

With further debugging, I discovered that, after I start my “left dragging” operation by pressing my left mouse button, the vtkTextureObject::Bind() method gets called multiple times:

  • vtkTextureObject::Bind() gets called multiple times with (this->Target == GL_TEXTURE_2D). All these calls succeed.
  • Then, the first time vtkTextureObject::Bind() gets called with (this->Target == GL_TEXTURE_BUFFER), the debug assertion is triggered and the crash happens.

Then, I removed the orientation marker widget from my scene, I rebuilt and restarted my Java application, I attached again my native debugger to the Java process and I put a breakpoint on vtkTextureObject::Bind() method.
Well, vtkTextureObject::Bind() does get called multiple times in this case as well, but it is ALWAYS called with (this->Target == GL_TEXTURE_2D), and it always succeeds.

Do you think the real issue when the orientation marker widget is present in my scene is that vtkTextureObject::Bind() at some point gets called with (this->Target == GL_TEXTURE_BUFFER)? Is this abnormal “per se”?

Thanks in advance for your feedback and insight.

Best regards,

Marco

Perhaps you have a multi-threading problem which could also explain the micro-pauses during user interaction. Check GLContext.getCurrent() at the point of binding the texture. Is the context assigned for that thread?

These might help
http://forum.jogamp.org/OpenGL-amp-multi-threading-td2665744.html
http://forum.jogamp.org/transfer-images-from-TextureData-to-a-Texture-off-of-the-EDT-in-JOGL-td4032678.html

Hello Todd,

thank you for your reply.

All interactions with the scene and all painting operations are done in the EDT thread in my application.
Also, I have checked the Thread ID in the native debugger, and the calls to vtkTextureObject::Bind() are always done on the same thread, both in the cases when Bind() succeeds and in the cases when Bind() crashes.

However, I’ve made an interesting discovery. If I switch to a more “basic” orientation marker, then my program does not crash and works normally (although with a too-low update rate) even when I interact with my scene. In particular, I modified my buildOrientationMarkerWidget() method like this:

> ======================
> 
> protected void buildOrientationMarkerWidgetSimpler()
>   {
>     // Orientation marker
>     orientMarkerWidget = new vtkOrientationMarkerWidget();
> 
>     vtkAnnotatedCubeActor orientMarkerCubeProp = new vtkAnnotatedCubeActor();
>     orientMarkerCubeProp.SetXMinusFaceText("R");
>     orientMarkerCubeProp.SetXPlusFaceText("L");
>     orientMarkerCubeProp.SetYMinusFaceText("A");
>     orientMarkerCubeProp.SetYPlusFaceText("P");
>     orientMarkerCubeProp.SetZMinusFaceText("I");
>     orientMarkerCubeProp.SetZPlusFaceText("S");
> 
>     orientMarkerWidget.SetOutlineColor( 0.9300, 0.5700, 0.1300 );
>     orientMarkerWidget.SetOrientationMarker(orientMarkerCubeProp);
>     orientMarkerWidget.SetInteractor(getRenderWindowInteractor());
>     orientMarkerWidget.SetViewport(0, 0.4, 0.2, 0.6);
>     orientMarkerWidget.SetEnabled(1);
>     orientMarkerWidget.InteractiveOff();
>   }
> 
> ======================

So, maybe in my original version there is something wrong with my vtkPropAssembly, which causes issues to JOGL?

Also, the slowness / low update rate issue remains. Basically, it looks like my JOGL view only gets updated when the EDT thread is not busy. For instance, while I drag my scene, the JOGL canvas only gets updated when I move my mouse very slowly. On the other side, if I continue moving/dragging my mouse very quickly without pauses, my JOGL canvas never gets updated.

Do you have suggestions about this?

Thanks in advance for all your feedback.
Best regards,

Marco Sambin

Why did you pass a vtkPropAssembly to the orientation marker widget anyway?

Hello Todd,

using the vtkPropAssembly was a trick to be able to assign different colors to the faces of the orientation marker widget. It was a smart suggestion by a user on this list.

Do you see something wrong in how this vtkPropAssembly is built in the code?

In any case, right now my main concern is the update performance / delays. Digging in the code, I saw that the implementation of the Render() method of vtkJoglCanvasComponent finally calls repaint() on the inner GLCanvas uiComponent. As it seems, calling repaint() while dragging the mouse is actually not calling paint() until the EDT is load-free.

This is quite strange, because according to my experience calling repaint() in a mouse event handler normally results in a quite immediate redraw of the component. But this experience of mine is mostly with lightweight components, not sure if this is different with heavyweight components like GLCanvas. Or, if the lightweight + heavyweight mixture I have on my UI may still influence this behavior.

I’ve made some tests overriding the Render() call of my vtkJoglCanvasComponent-derived panel, and I replaced the call to repaint() with a direct call to paint() of uiComponent (i.e., the GLCanvas). In my tests, this has greatly improved the canvas’ refresh rate while left-dragging my scene (even though we are still far from the performance that I can obtain with vtkCanvas under Windows - is this penalty expected?), but has other side effects. I am not sure this is the right way to go with my vtkJoglCanvasComponent-derived panel…

Any comment will be greatly appreciated.

Thanks and best regards,

Marco Sambin

I think this is a threading issue.

Since you have it working with JAWT your best bet is to understand the difference between how it handles threads versus JOGAMP; particularly when updating the UI.

Hello Todd,

thanks a lot for your feedback, I appreciate.

I thought that doing repaints in the EDT was correct for both JAWT and JOGL, isn’t it?
I only manipulate my scene in the EDT, and also interaction with the scene (left-dragging with the mouse) is done on the EDT. Moreover, this isn’t something upon which I have much control. I simply use a vtkJoglCanvasComponent-derived panel, and I inherit management of mouse-based interaction from it (and my scene’s interactor).

I will try to get a more in-depth knowledge about threading best-practices with JOGL. I still need to find a good source of documentation about it, though.

Thanks and best regards,

Marco Sambin

There may well be a bug in VTK which JOGL exposes, but unless you can pinpoint the reason it isn’t encountered in JAWT I don’t think it will be resolved.

Is this relevant? https://stackoverflow.com/questions/22976815/java-opengl-draw-offscreen-buffer-to-image

Hello.
Today I’ve discovered a very interesting thing while trying to debug the issue of the low repaint/refresh rate on my vtkJoglCanvasComponent-derived panel, especially when interacting with the scene (e.g., left-dragging with my mouse).

In the initialization of my panel, I had the following method (called on the EDT):

> ======================
> public synchronized void initializeView()
>   {
>     if (! viewInited)
>     {
>       final vtkGenericRenderWindowInteractor interactor = getRenderWindowInteractor();
>       interactor.EnableRenderOff();
> 
>       ...
> 
>       // Schedule a delayed repaint of the 3D scene
>       ActionListener taskPerformer = new ActionListener() 
>       {
>         public void actionPerformed(ActionEvent evt) 
>         {
>           interactor.EnableRenderOn();
>           Render();
>         }
>       };
>       javax.swing.Timer repaintTimer = new javax.swing.Timer(250, taskPerformer);
>       repaintTimer.setRepeats(false);
>       repaintTimer.start();
>     }
> ======================

This EnableRenderOff() / EnableRenderOn() sequence was a legacy of the JAWT version of this implementation. With this code, left-dragging with my mouse on the scene results in very slow and choppy repaints of my scene.

On the other side, if I comment both those calls (EnableRenderOff() and later EnableRenderOn()) in my code, and then execute again my application, updates / repaints of my scene are very smooth and quick!

Can you guess why? Is something incorrect in those calls (wrong threads, etc.)? Both calls are done on the EDT as far as I understand.

Another little step forward today… :wink:

Thanks and best regards,

Marco

Since the timer runs only once, I cannot see how it would have an impact on the interactor. Unless you are repeatedly resetting viewInited and calling initializeView().

Hello.
New discovery: I’ve reviewed the source of vtkAbstractJoglComponent (which is the base class for my JOGL-based panel), and here is a fragment of the source code of its constructor:

================

[…]
// Make sure when VTK internally request a Render, the Render get
// properly triggered
renderWindowToUse.AddObserver(“WindowFrameEvent”, this, “Render”);
renderWindowToUse.GetInteractor().AddObserver(“RenderEvent”, this, “Render”);
renderWindowToUse.GetInteractor().SetEnableRender(false);
[…]
================

So, it basically looks like these JOGL-based components are designed to work with interactor.SetEnableRender(false).

So, my sequence of calls EnableRenderOff() / EnableRenderOn() had the final effect of changing the way interactor repainting worked with JOGL components, and probably set it to an undesirable value.

So, at least the issue related to slow repainting upon interaction with the scene with JOGL components seems to have an explanation now, and can be considered as fixed.

Regards,

Marco Sambin

Thanks Marco for your investigation and dedication. I’ll keep that in mind if someone else run into something similar.

Hello Sebastien,

now I will move on to building VTK 8.2 + Java + JOGL on macOS.
To this end, I was following this thread:

However, I miss the conclusion of this thread, i.e., which modification should be done on VTK 8.2 in order to obtain a successful and functioning build with JOGL + Java on Mac.

Have you maybe received some updates from the original poster?
If someone has already spent days in fixing some issues, I would be very happy if I could avoid spending again a lot of time in figuring out what should be modified.

Using VTK v8.2 is a must for me.

Thanks for your feedback and best regards,

Marco Sambin