Hi, I’m trying to make an adaptive volume loader, first trying to load super big volume that will probably not fit into the memory and then decrease the volume dimensions and try again until the volume actually loads.
I’ve used ErrorObserver example to intersept error messages, then I rebuild the volume data. This generally works.
The problem I’ve seen happen is when I try to load the volume of size 23gb, while my free memory is 5gb, I catch “Out of memory”, then try to load 3gb volume, it still gives “Out of memory” even though I have 5gb free, and only on the third try with 0.5gb it loads without error.
This doesn’t happen if I load the same 3gb volume from the start, what could be the problem, opengl not releasing resources fast enough, error hanging inside the buffer for too long? how do I manually free the opengl resources?
It is generally approached from the other way around: you load a low-resolution volume quickly to minimize “time to first image” (important quality factor of imaging software). Then, preferably in the background, you can load higher-resolution image.
In general, there is no way to safely probe the amount of memory available on a system by trying to allocate memory: your software may get immediately terminated if it tries to allocate more memory than it is permitted, or you may get virtual memory allocated that causes so much swapping that renders your entire computer unusable. It is better to allow the user to specify a maximum memory usage limit (maybe initialize that limit from value that you determine from system information; such as use maximum of 50% of total RAM by default).
Hi, Mykola,
It seems I’m missing something. Are you refering to main memory or graphics card memory? These are different resources. If you’re loading volume data to RAM, then there is nothing OpenGL can do to help you. You have to check the memory management techniques in your program.
Another reason for getting OoM errors is memory fragmentation. Sometimes you have many gigabytes free in your system, but there is no slot big enough for your big block.
regards,
Paulo
Sorry for the confusion, I was talking about graphics card memory.
I think to go the other way and load low-resolution first would be a much better approach, thanks. How would I know if the volume was loaded successfully?
I’m still confused why when having 5gb free mem, and after failing to load 20gb I’m also failing to load 3gb, even though if I load 3gb straight away, it works.
I’ve managed to fix the not loading 3gb after 23gb. If I trigger the rendering in between doing the loading (e.g. Modified() or ResetCamera() and then Render()), then it works and 3gb is loaded after the failure with 23gb. I think the opengl just needs time to deallocate the memory? idk…
is there any tools to detemine if there is GPU memory fragmentations problem in Linux? and then fix the problem and reclaim all the fragments/?
thanks!
Hello,
Please, take a look at this: Does OpenGL takes care of GPU memory fragmentation? - Stack Overflow .
best,
PC
OpenGL abstracts away the details of memory management. As you saw in the stackoverflow comment, you generally don’t need to worry about it.
A VTK/OpenGL developer does not have access to move or chunk up allocations made on the GPU. If you’re interested in alternatives, I suggest reading up vulkan’s memory story and implementing something like that in VTK using OpenGL. Although very contrived and difficult, it will fit your needs!
You can read about Sub-allocation and basics about memory allocation to understand how it works in Vulkan. Essentially, Vulkan encourages developers to create a single large allocation and borrow memory from that for individual allocations like vertex data, indices, etc. This way, an application developer is in complete control of memory management and (de)fragmentation
Following those concepts, you might be able to design a “main” large graphics memory area (conceptually similar to VkBuffer) in VTK and proxy all glBuffer* calls from VTK’s OpenGL classes through this singleton class to a single GL_ARRAY_BUFFER. Since this class has complete control and knowledge of all GPU allocations, it’s easy to write a defrag function to move all the free’d buffer blocks to the end.
int main(int argc, char* argv[])
{
const char* fileName = "xxx";
vtkMetaImageReader* reader = vtkMetaImageReader::New();
reader->SetFileName(fileName);
reader->Update();
const std::uint64_t LOOP_NUM = std::numeric_limits<uint64_t>::max();
for (uint64_t i = 0; i < LOOP_NUM; i++)
{
vtkRenderWindow* window = vtkRenderWindow::New();
vtkRenderer* render = vtkRenderer::New();
vtkVolume* volume = vtkVolume::New();
vtkVolumeProperty* volumeProperty = vtkVolumeProperty::New();
vtkOpenGLGPUVolumeRayCastMapper* mapper = vtkOpenGLGPUVolumeRayCastMapper::New();
mapper->SetInputConnection(reader->GetOutputPort());
mapper->SetBlendModeToComposite();
mapper->SetAutoAdjustSampleDistances(0);
mapper->SetLockSampleDistanceToInputSpacing(0);
mapper->SetSampleDistance(0.2);
mapper->SetUseJittering(0);
volumeProperty->SetInterpolationToNearest();
for (int c = 0; c < VTK_MAX_VRCOMP; c++)
{
volumeProperty->ShadeOn(c);
volumeProperty->SetAmbient(c, 0.2);
volumeProperty->SetDiffuse(c, 0.8);
volumeProperty->SetSpecular(c, 0.2);
volumeProperty->SetSpecularPower(c, 30.0);
volumeProperty->SetScalarOpacityUnitDistance(c, 0.2);
vtkSmartPointer<vtkColorTransferFunction> color = vtkSmartPointer<vtkColorTransferFunction>::New();
color->AddRGBPoint(100, 1.0, 0.0, 0.0);
color->AddRGBPoint(200, 1.0, 0.5, 0.5);
color->AddRGBPoint(300, 1.0, 1.0, 1.0);
vtkSmartPointer<vtkPiecewiseFunction> opacity = vtkSmartPointer<vtkPiecewiseFunction>::New();
opacity->AddPoint(100, 0.0);
opacity->AddPoint(500, 1.0);
vtkSmartPointer<vtkPiecewiseFunction> gradient = vtkSmartPointer<vtkPiecewiseFunction>::New();
gradient->AddPoint(-100, 0.0);
gradient->AddPoint(0, 0.5);
gradient->AddPoint(100, 1.0);
volumeProperty->SetColor(c, color);
volumeProperty->SetScalarOpacity(c, opacity);
volumeProperty->SetGradientOpacity(c, gradient);
}
volume->SetProperty(volumeProperty);
volume->SetMapper(mapper);
render->AddVolume(volume);
render->SetBackground(0.0, 0.0, 0.0);
window->SetOffScreenRendering(1);
window->AddRender(render);
window->SetSize(500, 500);
window->SetWindowName("VTK test");
window->Render();
mapper->Delete();
mapper = nullptr;
volumeProperty->Delete();
volumeProperty = nullptr;
volume->Delete();
volume = nullptr;
render->Delete();
render = nullptr;
window->Delete();
window = nullptr;
}
reader->Delete();
reader = nullptr;
return 0;
}
did a test like this: after about 65000 loops, crash, got OpenGL error(1285)“Out of memory” .
while the testing, also use nvidia-smi -l 1 -f to output the GPU memory info and the utilization of GPU memory seems OK all the time(eg. all around 1000MiB / 6000Mib).
That’s why i doubt the reason is GPU memory fragmentation. If it was, is there any way to avoid it? Thanks!
Could you provide code that we can use to reproduce the issue? The pseudocode above is not really useful because critical parts don’t even compile, e.g., it is not possible to compile delete mapper as VTK objects have private destructors.
add the fileName, and can be compiled now. Thanks!
by the way, may i ask does VTK support Vulkan now?
There is no such class as vtkRender. You are not supposed to instantiate backend-specific classes, such as vtkOpenGLGPUVolumeRayCastMapper, but use the generic version vtkGPUVolumeRayCastMapper instead and let VTK to decide what is appropriate for the current rendering backend. GPU mapper does not have SetAutoAdjustSampleDistance method. There are several other errors. How did you test this code if it does not compile?
How long does it take for you to run out of memory? If you recreate everything (render window, renderer, mapper, etc.) then each iteration can take several hundreds milliseconds. It really does not make sense to recreate all these objects for every rendering request. It may be interesting as a stress test but does not reflect real-world use.
vtkRenderer, SetAutoAdjustSampleDistances, sorry for typo error
yes, it’s a test, for a server app.
if all of this new and delete in the 1 main process, it crash with OpenGL error Out of memory after tens of thousands loops; if move all of them into another thread it crash with this error after hundreds of thousands loops; if move all of them into another process it seems ok to always running.
Thanks!
have VTK dev team ever made a similar test? Thanks!
and do you know the inside memory layout or management of OpenGL? different on various platforms and drivers?
very werid that after got the OpenGL Out of memory ERROR, not only nvidia-smi shows that there is still enough available VRAM, and test with cudaMalloc shows that there is enough large contigous VRAM…donot know why after many loops of new and delete window with other res, OpenGL in VTK crash and shows Out of memory, it seems that OpenGL is not able to alloc on the very large enough contigous VRAM…
by the way, may i ask does VTK support Vulkan now?
No, VTK does not use Vulkan.
In general, Vulkan lets us probe the amount of memory available and many interesting limits of the GPU. In fact, vkAllocateMemory returns VK_ERROR_TOO_MANY_OBJECTS when trying to allocate larger than the limit VkPhysicalDeviceMaintenance3Properties::maxMemoryAllocationSize. This way, applications can know the system configuration at run time and prefer lower memory code paths. As a result, systems with higher limits and processing power do not need to waste cycles loading low resolution and then switch up to high resolution.
Unfortunately, there is no concept of a physical device in OpenGL. So you’re left to emulate a physical device backed by an OpenGL buffer as it’s ‘memory’ and use that through out OpenGL. It’s easier to just use Vulkan or webgpu in VTK instead of resorting to such tricks.
If you’re interested in this solution for a project, or just discuss about it, feel free to contact us for professional support.
Creating tens of thousands of render windows is allowed, but a suboptimal and probably a not thoroughly tested use case anywhere. There may be some bugs in VTK but maybe in the driver that are not prepared for this very unusual stress testing.
I would suggest to try to remain on a more common, more reasonable usage pattern. For example, there should be absolutely no need to destroy your render window and renderer after each rendering request. It would also make sense to cache actors and mappers as much as possible, as it would not just decrease chances of exposing GPU driver or VTK errors but could also speed up rendering by 1-2 magnitudes.
from: Bindless Textures - resident textures Limits - OpenGL / OpenGL: Advanced Coding - Khronos Forums
On Windows, the limited imposed by the Microsoft WDDM on your graphics driver is 4096 maximum allocations. On Linux, this limit is 4 billion. You can query this limit under Vulkan as VkPhysicalDeviceLimits::maxMemoryAllocationCount . See:
yes, i also doubt the test reach the MAX number of allocation. BUT as you said, there is not way to query this in VTK and even OpenGL, right?
thanks! Not to mention new and delete Window is necessary at some situation. Consider the MaxNumber Limit of OpenGL for allocation, even not re-create the Window, and optimize to the MIN number of allocation, if it was running in a Server, it would reach the Limit and crash sooner or later, right?