the speed is too slow for vtkWindowToImageFilter, how to speed it?

Support
1

Currently, we need to use vtk in my project. We want to obtain the off sreen rendering picture and show it. We use vtkWindowToImageFilter to obtain the rendering picture. However, we find it’s too slow. My code is:

from PyQt5.QtWidgets import *
from PyQt5.QtGui import *
from PyQt5.QtCore import *
import sys, os
import numpy as np
import vtk, qimage2ndarray

class vtkLabel(QLabel):
    def __init__(self):
        super(vtkLabel, self).__init__()
        cone = vtk.vtkConeSource()
        mapper = vtk.vtkPolyDataMapper()
        mapper.SetInputConnection(cone.GetOutputPort())
        actor = vtk.vtkActor()
        actor.SetMapper(mapper)

        ren = vtk.vtkRenderer()
        ren.AddActor(actor)
        renWin = vtk.vtkRenderWindow()
        renWin.AddRenderer(ren)
        renWin.SetOffScreenRendering(1)
        imageFilter = vtk.vtkWindowToImageFilter()
        imageFilter.SetInput(renWin)

        self.ren = ren
        self.renWin = renWin
        self.imageFilter = imageFilter

    def resizeEvent(self, QMouseEvent):
        super(vtkLabel, self).resizeEvent(QMouseEvent)
        print(self.width(), self.height())
        self.renWin.SetSize(self.width(), self.height())
        self.renWin.Render()
        self.imageFilter.Modified()
        self.imageFilter.Update()
        displayImg = self.imageFilter.GetOutput()

        dims = displayImg.GetDimensions()
        from vtk.util.numpy_support import vtk_to_numpy
        numImg = vtk_to_numpy(displayImg.GetPointData().GetScalars())
        numImg = numImg.reshape(dims[1], dims[0], 3)
        numImg = numImg.transpose(0, 1, 2)
        numImg = np.flipud(numImg)

        displayQImg = qimage2ndarray.array2qimage(numImg)
        pixmap = QPixmap.fromImage(displayQImg)
        self.pixmap = pixmap

    def paintEvent(self, QPaintEvent):
        super(vtkLabel, self).paintEvent(QPaintEvent)
        painter = QPainter(self)
        width = self.width()
        height = self.height()
        painter.drawPixmap(QPoint(0, 0), self.pixmap, QRect(0, 0, width, height))

class MainWindow(QMainWindow):
    def __init__(self, parent=None):
        super(MainWindow, self).__init__(parent)
        self.resize(500,500)
        self.imageLabel = vtkLabel()
        self.setCentralWidget(self.imageLabel)

app = QApplication(sys.argv)
window = MainWindow()
window.show()
app.exec_()

In this code, we find the speed is slow when we change the window size. And the weird thing is that the speed would be fast if I delete/comment the code “renWin.SetOffScreenRendering(1)”.

How can I speed up this code? Any suggestion is appreciated.

2

As far as I know OffScreenRendering is very slow because it renders without hardware acceleration only on the CPU and not on the GPU. So far I have not seen a way to make OffScreenRendering faster.

3

What are you trying to achieve?
It looks that you are copying the result of VTK to a QT widget.
You might be interested by QVTKOpenGLNativeWidget

4

But why “renWin.SetOffScreenRendering(1)” make so much difference? Have you tried my code? The code without renWin.SetOffScreenRendering(1) would be much faster than that with renWin.SetOffScreenRendering(1).

5

Yes, I want to embed VTK into QT. However, I don’t want to a QVTK class, because QT is more convenient to add more complicate operation than VTK

6

Using OffScreen framebuffers is not necessarily slower. But using vtkWindowToImageFilter is slow because you will copy VTK result from GPU memory to CPU memory and it will affect badly the performance.
Running complicated operation to the framebuffer as a post processing pass can be done very efficiently on the GPU using vtkRenderPass but you will need basic OpenGL knowledge.

7

Could you please provide an example to use vtkRenderPass?

8

vtkToneMappingPass
You can also read the test to understand how to enable render passes: TestToneMappingPass

9

Thank you very much. I can understand the render pass, but I don’t know how to get the picture in render window by using vtkRenderPass. Could you please provide a demo?

10 
vtkNew<vtkRenderer> renderer;

// custom passes
vtkNew<vtkCameraPass> cameraP;
vtkNew<vtkSequencePass> seq;
vtkNew<vtkOpaquePass> opaque;
vtkNew<vtkLightsPass> lights;

vtkNew<vtkRenderPassCollection> passes;
passes->AddItem(lights);
passes->AddItem(opaque);
seq->SetPasses(passes);
cameraP->SetDelegatePass(seq);

vtkNew<vtkToneMappingPass> toneMappingP;
toneMappingP->SetDelegatePass(cameraP);

// The following code set the render pass to the renderer
vtkOpenGLRenderer::SafeDownCast(renderer)->SetPass(toneMappingP);

renWin->AddRenderer(renderer);
11

The code is clear. But how can I capture the picture from render window?

12

What do you mean by capture the picture?

13

Just like those code in the resizeEvent, when the size of window is change, I want to obtain the picture in render window and then show it in the QT window.
In summary, I want to use the VTK for render, and use QT for all operation.

14

If you want to keep using QT to modify the VTK image, you will have to accept the performance issues.
If the performances are a real issue, consider using render passes GPU shaders instead of QT to modify the image.

15

Have you run the posted code? Could you please explain why renWin.SetOffScreenRendering(1) will make so large difference?

16

Hi, Michael, it sounds like I am trying to do something similar. I need to get the RGBA data from the renderer to use in other code downstream. It sounds like the performance is limited by the copy performance from GPU to CPU memory. The performance I’m currently getting is too slow. Are there any alternatives? I’ve tried using GetRGBACharPixelData() on the window but the data is not yielding the upsampling that would be applied by the WindowToImageFilter. Is there an easy way to tell VTK to use CPU only for certain things? Thanks for your help.