Which way to handle large data (2000x2560x1440 uint8)

I need to visualize 2000 images each 2560x1440 and 8 bit color using Python3. The images are stored in a properietary 3d-printing format (RLE encoded). I have code to extract each image as an numpy array. Using vtk.vtkImageImport().CopyImportVoidPointer I could read this into vtk. Problem is that reading all images in one numpy-array causes memory problems (6GB of data).

I see a few possible solutions:

  1. use numpy.memmap but this is slow on linux, I could investigate this further;
  2. save all images to disk and use a vtk file reader, but saving to disk is slow, so also not optimal;
  3. write my own vtk file reader which directly reads the 3d-printing file, however I have found a way to do this in python.
  4. read each slice into a vtk.vtkVolume() and add it to the renderer using vtk.vtkRenderer(). I encountered the problem that the Z-order is not adjusted when rotating the scene.

Am I missing another way to handle this much data? If not which solution is the best to investigate further?

Below the code which shows my Z-order problem in solution 4:

import numpy as np
import vtk
from tempfile import mkdtemp
import os.path as path
import psutil

def init():
    import sys,os
    global installpath
    if getattr(sys, 'frozen', False):# frozen
        installpath = os.path.dirname(sys.executable)
    else: # unfrozen
        installpath = os.path.dirname(os.path.realpath(__file__))
    print ("Installed at: ",installpath)

def addCube(offset,colorIndex):

    data_matrix[0+dd:35+dd,0+dh:35+dh,0+dw:35+dw] = colorIndex

    # For VTK to be able to use the data, it must be stored as a VTK-    image.
    #  This can be done by the vtkImageImport-class which
    # imports raw data and stores it.
    dataImporter = vtk.vtkImageImport()
    # The previously created array is converted to a string of chars and     imported.

    data_string = data_matrix.tostring()
    del (data_matrix)
    dataImporter.CopyImportVoidPointer(data_string, len(data_string))
    del (data_string)

    # The type of the newly imported data is set to unsigned char     (uint8)
    # Because the data that is imported only contains an intensity value
    #  (it isnt RGB-coded or someting similar), the importer must be     told this is the case.
    # The following two functions describe how the data is stored and     the dimensions of the array it is stored in.
    #  For this simple case, all axes are of length 75 and begins with the first element.
    #  For other data, this is probably not the case.
    # I have to admit however, that I honestly dont know the difference between SetDataExtent()
    #  and SetWholeExtent() although VTK complains if not both are used.
    dataImporter.SetDataExtent(0, w-1, 0, h-1, 0, d-1)
    dataImporter.SetWholeExtent(0, w-1, 0, h-1, 0, d-1)

    # The following class is used to store transparency-values for later retrival.
    #  In our case, we want the value 0 to be
    # completely opaque whereas the three different cubes are given different transparency-values to show how it works.
    alphaChannelFunc = vtk.vtkPiecewiseFunction()
    alphaChannelFunc.AddPoint(0, 0.0)
    alphaChannelFunc.AddPoint(colorIndex, 1.0)#0.05)

    # This class stores color data and can create color tables from a     few color points.
    #  For this demo, we want the three cubes to be of the colors red green and blue.
    colorFunc = vtk.vtkColorTransferFunction()
    #colorFunc.AddRGBPoint(0, 0.0, 0.0, 0.0)
    colorFunc.AddRGBPoint(1, 1.0, 0.0, 0.0)
    colorFunc.AddRGBPoint(2, 0.0, 1.0, 0.0)
    colorFunc.AddRGBPoint(3, 0.0, 0.0, 1.0)
    colorFunc.AddRGBPoint(4, 1.0, 1.0, 0.0)
    colorFunc.AddRGBPoint(5, 1.0, 0.0, 1.0)
    colorFunc.AddRGBPoint(6, 0.0, 1.0, 1.0)
    colorFunc.AddRGBPoint(7, 1.0, 1.0, 1.0)

    # The previous two classes stored properties.
    #  Because we want to apply these properties to the volume we want to render,
    # we have to store them in a class that stores volume properties.
    volumeProperty = vtk.vtkVolumeProperty()

    volumeMapper = vtk.vtkFixedPointVolumeRayCastMapper()

    # The class vtkVolume is used to pair the previously declared volume as well as the properties
    #  to be used when rendering that volume.
    volume = vtk.vtkVolume()

    return volume

def main():
    global installpath
    colors = vtk.vtkNamedColors()

    # We begin by creating the data we want to render.

    # With almost everything else ready, its time to initialize the     renderer and window, as well as
    #  creating a method for exiting the application
    renderer = vtk.vtkRenderer()
    renderWin = vtk.vtkRenderWindow()
    renderInteractor = vtk.vtkRenderWindowInteractor()

    # We add the volume to the renderer ...

    # ... and set window size.
    renderWin.SetSize(400, 400)

    # A simple function to be called when the user decides to quit the     application.
    def exitCheck(obj, event):
        if obj.GetEventPending() != 0:

    # Tell the application to use the function as an exit check.
    renderWin.AddObserver("AbortCheckEvent", exitCheck)

    # Because nothing will be rendered without any input, we order the     first render manually
    #  before control is handed over to the main-loop.

if __name__ == '__main__':

If you let VTK to allocate and manage the array (create a vtkImageData object, set its extent to the final volume size, allocate it, and get the voxels as a numpy array) then there should be no problem setting the voxels slice by slice using numpy.

As a rule of thumb, have 10x more virtual memory space (preferably physical RAM) as your input data size to avoid memory issues and get acceptable performance.

Thx for your reply. Does this mean that displaying 6GB of data with a pc which has 4GB physical RAM is not possible with vtk?

It is possible to process a 6GB data set on a machine with 4GB RAM (you just have to allocate enough virtual memory), but it can make your program several magnitudes slower. You need to get a better computer or downsample your data set.