While using 3D Slicer, I ran into an issue with VTK when using very large meshes on Apple Silicon OpenGL. I realize OpenGL is deprecated on apple silicon, but I think this is a relatively straightforward work-around for an issue with Apple’s OpenGL driver. Below is Claude’s summary and suggestions, let me know what you think and I’m happy to open a merge request.
(And thanks much for your help with the earlier issue!)
Summary
On macOS (Apple Silicon), rendering a polydata whose index or vertex buffer is larger than 2³¹ bytes crashes with EXC_BAD_ACCESS in _platform_memmove inside Apple’s OpenGL driver. The crash happens in glBufferSubData when a single call transfers more than 2³¹ bytes. Splitting the same upload into several smaller glBufferSubData calls works. I’d like to propose doing that in vtkOpenGLBufferObject::UploadRangeInternal, and to ask whether that’s an acceptable approach before opening a merge request.
I hit this in 3D Slicer with a segmentation surface of ~234M triangles (a 2.8 GB index buffer). The Slicer developers suggested fixing it in VTK.
Reproducer (pure VTK, stock 9.7.1 wheel from PyPI)
A 10000 × 10000 vtkPlaneSource is 100M quads, which the mapper triangulates into 600M indices, ~2.4 GB uploaded in one call. It needs ~13 GB of RAM.
import vtkmodules.vtkRenderingOpenGL2 # noqa: F401
from vtkmodules.vtkFiltersSources import vtkPlaneSource
from vtkmodules.vtkRenderingCore import (
vtkActor, vtkPolyDataMapper, vtkRenderer, vtkRenderWindow,
)
plane = vtkPlaneSource()
plane.SetResolution(10000, 10000)
mapper = vtkPolyDataMapper()
mapper.SetInputConnection(plane.GetOutputPort())
actor = vtkActor()
actor.SetMapper(mapper)
renderer = vtkRenderer()
renderer.AddActor(actor)
window = vtkRenderWindow()
window.AddRenderer(renderer)
window.SetSize(400, 400)
renderer.ResetCamera()
window.Render() # crashes in glBufferSubData on macOS
print("Rendered without crashing")
It crashes within a few seconds:
EXC_BAD_ACCESS (KERN_PROTECTION_FAILURE)
libsystem_platform.dylib _platform_memmove
AppleMetalOpenGLRenderer gldBufferSubData
GLEngine glBufferSubData_Exec
libvtkRenderingOpenGL2.dylib vtkOpenGLBufferObject::UploadInternal(...)
libvtkRenderingOpenGL2.dylib vtkOpenGLIndexBufferObject::CreateTriangleIndexBuffer(...)
libvtkRenderingOpenGL2.dylib vtkOpenGLPolyDataMapper::BuildIBO(...)
It’s the driver, not VTK
A standalone CGL / OpenGL 3.2 core program with no VTK shows:
glBufferDataof 2.4 GB succeeds:glGetError()returns 0 andGL_BUFFER_SIZEreports the full size.- A single
glBufferSubDataof exactly 2³¹ bytes works. Anything larger crashes (tested 2³¹ + 4096 and 2.4 GB). - The same 2.4 GB uploaded as
glBufferSubDatacalls of ≤1 GB works,glGetError()stays clean, and reading the buffer back withglMapBufferRangematches the source byte for byte.
It isn’t a real hardware limit: MTLDevice.maxBufferLength on this machine is ~86 GB. The driver also reports no error before crashing, so VTK has nothing to check for.
Standalone OpenGL test program
// Build: clang -w gl_buffersubdata_2gb.c -framework OpenGL -o gl_buffersubdata_2gb
// Usage: ./gl_buffersubdata_2gb <total bytes> [chunk bytes]
#define GL_SILENCE_DEPRECATION
#include <OpenGL/OpenGL.h>
#include <OpenGL/gl3.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main(int argc, char** argv)
{
setvbuf(stdout, NULL, _IONBF, 0);
CGLPixelFormatAttribute attrs[] = { kCGLPFAOpenGLProfile,
(CGLPixelFormatAttribute)kCGLOGLPVersion_3_2_Core, kCGLPFAAccelerated, 0 };
CGLPixelFormatObj pf;
GLint npf;
CGLChoosePixelFormat(attrs, &pf, &npf);
CGLContextObj ctx;
CGLCreateContext(pf, NULL, &ctx);
CGLSetCurrentContext(ctx);
size_t size = strtoull(argv[1], NULL, 10);
size_t chunk = argc > 2 ? strtoull(argv[2], NULL, 10) : size;
unsigned char* data = malloc(size);
for (size_t i = 0; i < size; i += 4096)
{
data[i] = (unsigned char)(i >> 12);
}
data[size - 1] = 0xAB;
GLuint buffer;
glGenBuffers(1, &buffer);
glBindBuffer(GL_ARRAY_BUFFER, buffer);
glBufferData(GL_ARRAY_BUFFER, (GLsizeiptr)size, NULL, GL_STATIC_DRAW);
GLint64 reported = 0;
glGetBufferParameteri64v(GL_ARRAY_BUFFER, GL_BUFFER_SIZE, &reported);
printf("glBufferData(%zu): error 0x%x, GL_BUFFER_SIZE %lld\n", size, glGetError(), reported);
for (size_t offset = 0; offset < size; offset += chunk)
{
size_t len = size - offset < chunk ? size - offset : chunk;
glBufferSubData(GL_ARRAY_BUFFER, (GLintptr)offset, (GLsizeiptr)len, data + offset);
}
printf("glBufferSubData in %zu-byte chunks: error 0x%x", chunk, glGetError());
unsigned char* mapped = glMapBufferRange(GL_ARRAY_BUFFER, 0, (GLsizeiptr)size, GL_MAP_READ_BIT);
printf(", read-back %s\n", mapped && memcmp(mapped, data, size) == 0 ? "matches" : "MISMATCH");
return 0;
}
Proposed change
All polydata VBO and IBO uploads go through UploadRangeInternal, so chunking there covers them:
--- a/Rendering/OpenGL2/vtkOpenGLBufferObject.cxx
+++ b/Rendering/OpenGL2/vtkOpenGLBufferObject.cxx
@@ -5,6 +5,8 @@
#include "vtk_glad.h"
+#include <algorithm>
+
@@ -260,8 +262,16 @@ bool vtkOpenGLBufferObject::UploadRangeInternal(
glBindBuffer(this->Internal->Type, this->Internal->Handle);
vtkDebugMacro(<< "glBufferSubData: "
<< "(offset: " << offset << ", size: " << size << ")");
- glBufferSubData(
- this->Internal->Type, static_cast<GLintptr>(offset), static_cast<GLsizeiptr>(size), buffer);
+ constexpr ptrdiff_t maxChunkSize = ptrdiff_t(1) << 30;
+ const char* data = static_cast<const char*>(buffer);
+ for (ptrdiff_t chunkOffset = 0; chunkOffset < size; chunkOffset += maxChunkSize)
+ {
+ const ptrdiff_t chunkSize = std::min(maxChunkSize, size - chunkOffset);
+ glBufferSubData(this->Internal->Type, static_cast<GLintptr>(offset + chunkOffset),
+ static_cast<GLsizeiptr>(chunkSize), data + chunkOffset);
+ }
this->Dirty = false;
Uploads of 1 GB or less are unchanged (one call). With this patch, the reproducer renders correctly: the 200M-triangle plane gives a pixel-identical image to the same plane at 100 × 100 resolution. I’ve been running it in my Slicer builds since September, and it fixes the original crash there.
In the interest of full disclosure: while retesting the standalone program, one run of the chunked 2.4 GB upload also crashed in gldBufferSubData. I couldn’t reproduce that in ~30 further runs (standalone and through VTK), including immediately after deliberate crashes, so I don’t know whether it’s a rare driver issue or something about that particular run.
Questions
- Is chunking in
UploadRangeInternalacceptable, or would you prefer it only on Apple (#ifdef __APPLE__)? It’s harmless elsewhere, since buffers up to 1 GB take the same single call as before. - Any preference for the chunk size? 1 GB is well under the 2³¹-byte limit; I didn’t find a performance difference.
- Should I also report it to Apple (Feedback Assistant)? Given OpenGL’s deprecated status on macOS, I’m not expecting a driver fix.
Environment: Apple M5 Max, 128 GB RAM, macOS 26.5 (25F71), VTK 9.7.1 (PyPI) and Slicer’s VTK fork (9.6.1).