If you are following rendering research, you certainly heard about 3D Gaussians splatting, a technology used to create a realistic 3D scene based on a video or multiple images and presented in Siggraph 2023.
I made some investigation to see what would be required to visualize these 3D Gaussians scenes in F3D, which uses VTK as a visualization backend.
Basically, the scenes are just a large number of points associated with:
An opacity value
A 3D scaling vector
A rotation (quaternion)
A view-dependent RGB color
I successfully managed to make it work with solid performance:
I’m planning to make several MRs to slowly bring the support to VTK:
Improve the current point Gaussian shader implementation (merged)
Very cool! It looks very realistic near the trees. It seems like shadows/lighting look more natural with this method. So, the only thing you upload from host to device(gpu) are point positions and those 4 attributes? That’s very neat!
Recently, I’ve had to wrestle with lots of hairy issues where the OpenGL polydata mapper would refuse to work in webassembly because it had some non-portable code. Since all of your work is in early development stages, do you plan for your code to run in webassembly as well? Are you making an active effort to ensure your OpenGL contributions to VTK are compatible with GLES 3.0 spec? I’m asking this because we are now taking webassembly, as a supported platform, more seriously than before
Geometry shaders are not supported in webgl2. So, it’d be nice to avoid them.
I’m currently using an extended version of the point gaussian mapper.
Unfortunately, the current mapper on VTK master uses the geometry shader but it should be fairly easy to migrate to an instancing approach (it could be more performant too).
Other than that I’m also planning to use a compute shader to sort the index buffer by depth.
It’s much more performant to do it on the GPU to modify the IBO in place and avoid a reupload.
Alternatively, this step could be done on the CPU (and there’s already a vtkDepthSortPolyData for that), but for cases with several millions of points (there’s 6M+ points in the video) that’s not an acceptable solution.
Sounds like you’re on a good path. As for the compute shader, it can be an alternate code path that runs when the OpenGL version is greater than 3.2 core or 3.1 ES.
Everything is ready in VTK master, but sorting splats and computing view-dependent colors (optional, it works fine without it) has to be implemented by the application (ideally using a compute shader if there’s a large number of Gaussians).
How did you solve the different type properties of from ply when it comes to Gaussian Splatting?
Reading the xyz seems quite straightforward but what about colours and its spherical harmonics components? Trying to test this myself with current VTK 9.3.0 and I would appreciate your help in this matter.
Thank you!
I’m not using the PLY reader in VTK and just implemented mine using Happly.
Regarding spherical harmonics, the RGB color of each point is computed (in a compute shader) when the camera direction changes.
Did you use any VTK component from version 9.3.0 or did you have to implement your own vertex shader for the splats with OpenGL? I have been trying to use VTK but it does not seem to support those typical Gaussian splats I see in the online renderers.
I would very much appreciate your guidance on how to implement this in VTK. Many thanks.
