Problems and questions using vtkSmartVolumeMapper for large volume streaming and rendering

I need to volume render datasets with a wide range of volume sizes: 5-120GBs is common and 120GB+ are rare but still happen. This will mostly be for non-interactive scenarios (predefined animations or stills), so I can afford to wait for a high-quality result. My goal is to volume render at the highest quality possible with respect to volume size, available system memory, and available VRAM. These volumes are stored in 16-bit, grayscale TIFs (one per slice). I’ve been running into multiple questions and problems and was hoping someone could clarify these things for me. For my testing, I’m working on a 2017 iMac (64GB of RAM, Radeon Pro 575 4GB VRAM), macOS 10.14.4 and VTK 8.1.0.

I’m working in a project that uses C++14, so I started by modifying the GPURenderDemo.cxx example, swapping out the image reading sections with a call to vtkTIFFReader. Based on this test file, I also added a vtkMemoryLimitImageDataStreamer between the resampler and the volume mapper. This all works and I can render small volumes quite easily:

    // Load the volume data
    auto volReader = vtkSmartPointer<vtkTIFFReader>::New();
    volReader->SetFileNames(slicePaths);
    volReader->SetOrientationType(1);
    volReader->ReleaseDataFlagOn();

    // Create the renderer, render window and interactor
    auto colors = vtkSmartPointer<vtkNamedColors>::New();
    auto renderer = vtkSmartPointer<vtkRenderer>::New();
    auto renWin = vtkSmartPointer<vtkRenderWindow>::New();
    renWin->AddRenderer(renderer);

    // Connect it all. Note that funny arithematic on the
    // SetDesiredUpdateRate - the vtkRenderWindow divides it
    // allocated time across all renderers, and the renderer
    // divides it time across all props. If clip is
    // true then there are two props
    auto iren = vtkSmartPointer<vtkRenderWindowInteractor>::New();
    iren->SetRenderWindow(renWin);
    iren->SetDesiredUpdateRate(frameRate / (1 + clip));
    iren->GetInteractorStyle()->SetDefaultRenderer(renderer);

    auto resample = vtkSmartPointer<vtkImageResample>::New();
    if (scaleFactor < 1.0) {
        resample->SetInputConnection(volReader->GetOutputPort());
        resample->SetAxisMagnificationFactor(0, scaleFactor);
        resample->SetAxisMagnificationFactor(1, scaleFactor);
        resample->SetAxisMagnificationFactor(2, scaleFactor);
    }

    // Volume Streamer
    // memLimitBytes is precalculated or user-provided
    auto streamer = vtkSmartPointer<vtkMemoryLimitImageDataStreamer>::New();
    streamer->SetMemoryLimit(memLimitBytes / 1024);
    if (scaleFactor < 1.0) {
        streamer->SetInputConnection(resample->GetOutputPort());
    } else {
        streamer->SetInputConnection(volReader->GetOutputPort());
    }
    streamer->UpdateWholeExtent();

    // Create our volume and mapper
    auto volume = vtkSmartPointer<vtkVolume>::New();
    auto mapper = vtkSmartPointer<vtkSmartVolumeMapper>::New();
    mapper->SetInputConnection(streamer->GetOutputPort());

    // Set the sample distance on the ray to be 1/2 the average spacing
    double spacing[3];
    if (scaleFactor < 1.0) {
        resample->GetOutput()->GetSpacing(spacing);
    } else {
        volReader->GetOutput()->GetSpacing(spacing);
    }
    mapper->SetSampleDistance((spacing[0] + spacing[1] + spacing[2]) / 6.0);
    
    // Rest of example code continues
    ...

My first problem is that my program uses more than the memory limit set on the streamer. As an example, if I load a 17GB dataset, set the scale factor to 1.0, and set the memory limit to 4GB, my program still uses ~18GBs of RAM. I know that the streamer is doing something, because if I set the scale factor to 0.6, the program only uses 4-7GBs of RAM (it varies as the volume loads). My working theory is that this is because the volume mapper must still hold all of the volume in memory and therefore only the reader and resampler pipeline are memory limited. This would make my total potential memory usage be the streamer memory limit plus the resampled volume size. Can someone confirm that this is the case? If so, is there a better way to limit the total RAM usage of my program?

My next problem is with the vtkSmartMapper GPU Ray Caster. I consistently get OpenGL Out of Memory Errors when I don’t manually scale the volume to fit within VRAM. My confusion here is that I have seen hints in other posts that seem to indicate that the GPU Ray Caster will do some sort of interpolation and downscaling in order to fit the volume in VRAM. Is this correct or have I misread? Is there a way to make the mapper fallback to the CPU renderer if the volume won’t fit in VRAM? This also wouldn’t be too big of an issue if I could programmatically determine how much VRAM is available and set the scale factor accordingly. However, I can’t seem to find the correct way to use vtkGPUInfoList for this purpose.

My final problem is with the CPU Ray Caster. When I use it, my first frame renders at full resolution, but later frames render at low resolution. This is very similar to this other thread, but sort of the opposite effect. This happens regardless of the volume size:

Typing on my phone on the move so must be brief: Yes I think the whole volume must be in RAM to be rendered.

There is a SetPartitions(…) on vtkOpenGLGPUVolumeRayCastMapper to do partitioned rendering, but I think it’s opt-in and you have to set it up yourself.

Probably others who have more insight than I do.

Cheers,

Elvis

Den mån 1 apr. 2019 17:53Seth via VTK noreply@discourse.vtk.org skrev:

Thanks for the suggestion. If I’m understanding you correctly, I would take my in-memory volume and setup the GPU renderer with total partitions = VolSize / VRAM. I can see how that would work to improve my GPU rendering reliability. It’s a little less than ideal in that I still need an option to fallback to CPU rendering if there’s not a GPU available at runtime, and the vtkSmartMapper makes that relatively easy.

As a side note, I’ve also started getting this error with the GPU renderers:

vtkVolumeTexture (0x7fdbf66283a0): Capabilities check via proxy texture 3D allocation failed!

It seems to happen when the volume is larger than, but relatively near, the amount of VRAM. The render window appears, but nothing renders into it.

If the volume is dramatically larger than VRAM, program memory use will skyrocket (multiples of the actual volume size), the volume will render incorrectly with wraparound, and I’ll get this error:

Generic Warning: In Rendering/OpenGL2/vtkOpenGLError.h, line 207
Failed to allocate 3D texture.1 OpenGL errors detected
  0 : (1285) Out of memory


ERROR: In [VTK source dir]/src/vtk/Rendering/VolumeOpenGL2/vtkVolumeTexture.cxx, line 763
vtkVolumeTexture (0x7fc7dd90a000): Texture 3D allocation failed! 

Have you not considered using the OSPRay rendering mode? if VTK is compiled with it, the SmartMapper will let you switch to it:

volumeMapper.SetRequestedRenderModeToOSPRay()

The OSPRay renderer is extremely fast, fully multi-threaded and keeps the volume on the CPU side, so you can use your full 64GB of RAM. One caveat: The vtkOSPRayVolumeMapperNode used to initialize OSPRay with shared buffers, i.e. zero-copy buffers. In the OSPRay documentation, this is referred to as shared_structured_volume. See https://www.ospray.org/documentation.html

This mode was however turned off by a recent commit 5e381f81 to enable caching. So watch for a duplication of your memory cost if you leave it as the default. I use VTK with the shared buffer forced to ON, and I am able for example to volume render a 3.5 Gvoxels (float32) dataset on my laptop.

I confirm that the same volume can also be rendered on my GPU (I only have 6 GB of VRAM) by a call to
volumeMapper.SetPartitions(2,2,1). Note however that the vtkSmartMapper does not expose that function. You would have to explicitly create a vtkOpenGLGPUVolumeRayCastMapper(). Note also that even though I can make one nice image on the first Render() call, interaction therafter is near impossible.

Finally, you asked about the CPU Ray Caster. You must mean the vtkFixedPointVolumeRayCastMapper(). I can eliminate the problem you mentioned by changing the Sampling Distance. By default, I see it is set to 1.0. Calling volumeMapper.SetLockSampleDistanceToInputSpacing(1) is what it takes to get a nicely resolved volume render. Once again, I cannot use the vtkSmartMapper() to do that since it does not expose that option. I explicitly create a vtkFixedPointVolumeRayCastMapper() and then I am good to go.

I did not talk about streaming bigger volumes yet. I don’t have any tips.
HTH.

1 Like

I have seen the OSPRay mode and it looks very nice, but I haven’t tried to get it up and running yet. I’m currently developing for myself, but as part of larger project, so I’m trying to avoid adding more dependencies.

Sorry for the confusion around the mappers. I did mean vtkGPUVolumeRayCastMapper and vtkFixedPointVolumeRayCastMapper. I have tried each of these on their own and as part of vtkSmartVolumeMapper and have had the issues I’ve described. I’ve also tried vtkOpenGLGPUVolumeRayCastMapper and had the same issues, but I honestly can’t tell if it’s actually a different class than vtkGPUVolumeRayCastMapper.

Anyway, thank you for the note about the sampling distance! That was indeed my issue with the CPU ray caster. Knowing that it was the sampling distance at fault, I was also able to get the vtkSmartVolumeMapper working by setting these two flags:

mapper->SetInteractiveAdjustSampleDistances(false);
mapper->SetAutoAdjustSampleDistances(true);

For streaming a large volume to the GPU, consider the approach suggested by @estan: Using vtkOpenGLGPUVolumeRayCastMapper::SetParitions()

As far as preferring the vtkSmartVolumeMapper over vtkGPUVolumeRayCastMapper - note that even though the intention behind the smart volume mapper was to switch between GPU and CPU rendering in a user-agnostic way (among other things), it does not do so anymore. Post the recent OpenGL2 rendering framework overhaul, the smart volume mapper always uses the GPU mapper unless explicitly set to use the CPU via vtkSmartVolumeMapper::SetRequestedRenderMode.

That’s very good to know. In that case, I guess I need to just pick a renderer OR implement some sort of manual fallback code. Is there a good way to programmatically check for GPU support and resources? I haven’t had any luck with vtkGPUInfoList. Since the documentation for that class tells you to use platform-specific implementations, and yet those implementations have been removed, my intuition is that all of these things are related and that isn’t supported anymore.

In a nutshell, no. And that’s why the info list is not implemented. Note that vtkGPUInfoList is meant to be overriden by platform + driver specific API to be of any use. For now, the idea is that most modern machines provide some form of graphics, either dedicated or on-board and support OpenGL. You could use vtkTextureObject::GetMaximumTextureSize() to query available graphics memory.

1 Like

That’s great! I’ll look into using that. I was already thinking that VRAM might be the primary limitation on the GPU side and that my selection mechanism would basically be to select the mapper with the best volsize/RAM ratio.

Sorry, my question is about how to compile VTK with OSPRay,

I have already checked Module_vtkRenderingOSPRay in cmake-gui and recompile VTK, but my program got error as:

Warning VTK is not linked to OSPRay so can not VolumeRender with it

What did I miss?

The error is fixed after I add below two lines:

vtkOSPRayPass* osprayPass = vtkOSPRayPass::New();
render->SetPass(vtkOSPRayPass);

thanks!

Hello Edward, I got the same error as:
Warning VTK is not linked to OSPRay so can not VolumeRender with it.
But I can’t fix it for I can’t include head file “<vtkOSPRayPass.h>”.
I reconfigure my project but it still doesn’t work.

Here is my part of CMakeLists.txt content:
find_package(VTK REQUIRED);

Could you tell me how to find the required modules for I could use vtkOSPRayPass in my project?

Thanks!

If you have built with VTK_ENABLE_OSPRAY:BOOL=ON, then the module you should link to is VTK::RenderingRayTracing

Hope that helps.

Hi,

It seems you don’t compile OSPRay module correctly.

Don’t compile the module as remote module, I have tried this way serval times but failed. Finally I compiled OSP separately and then reconfigured VTK with OSPRay checked.

OSP depends some libs, you should install or compile them first.

good luck!

Hi,

hi,
I am trying to install OSPRay module, and I got a problem when using the CMake GUI to configure OSPRay,
I got this:
CMake Error at components/ospcommon/cmake/ispc.cmake:66 (message):
Need at least version 1.9.1 of Intel SPMD Compiler (ISPC).
Call Stack (most recent call first):
CMakeLists.txt:52 (include)

here is what I did during install OSPRay module:

  1. reconfigure vtk as Sankhesh Jhaveri sugessted.
  2. rebuild vtk and got error"fatal error C1083: Cannot open include file: ‘ospray/ospray.h’: No such file or directory"
  3. compiling OSPRay on Windows “http://sdvis.org/ospray/download/OSPRay_readme_release-1.8.1.pdf
  4. download some libs such as : ISPC, TBB, Embree;

I think I am not successfully compiled ISPC and TBB for I set TBB_ROOT and ISPC_EXECUTABLE manually in CMake.

Hi,

The error message is clear, you should install the lastest ISPC compiler.

I have install ISPC v1.11.0. I download ISPC from https://sourceforge.net/projects/ispcmirror/files/v1.11.0/, and then copy it into Path. Did I miss any step?

Sorry, it seems just copy the file ispc to /usr/local/bin (I do my work on Ubuntu), I’m not very sure, it’s some time ago.

Thank you! I restart my computer and now it worked well.