# Differentiable rendering module with PyTorch integration for VTK
**URL:** https://discourse.vtk.org/t/differentiable-rendering-module-with-pytorch-integration-for-vtk/16401
**Category:** Development
**Tags:** code, proposal
**Created:** [April 30, 2026, 5:42pm UTC](https://discourse.vtk.org/t/differentiable-rendering-module-with-pytorch-integration-for-vtk/16401 "2026-04-30T17:42:18Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Juan\_C\_Prieto](https://discourse.vtk.org/u/Juan_C_Prieto)
#### Post date: [April 30, 2026, 5:42pm UTC](https://discourse.vtk.org/t/differentiable-rendering-module-with-pytorch-integration-for-vtk/16401/1 "2026-04-30T17:42:18Z")
## **Summary**
I would like to propose the addition of a differentiable rendering module to VTK — tentatively `VTK::RenderingDifferentiable` — that exposes gradients through VTK’s rendering pipeline and bridges them to PyTorch’s autograd system. The goal is to enable gradient-based optimization of scene parameters (camera pose, lighting, geometry) directly within VTK’s pipeline model.
I’m writing here first to validate the approach, gather feedback on architectural fit, and understand whether this belongs upstream or is better maintained as an out-of-tree module initially.
## **Motivation**
Differentiable rendering has become a core tool in scientific computing, inverse problems, and 3D machine learning — enabling tasks like:
- Camera calibration and pose estimation from rendered images
- Geometry reconstruction from 2D observations
- Physics-based parameter fitting (material properties, lighting conditions)
- Integration of 3D rendering into neural network training loops
Existing differentiable renderers (PyTorch3D, nvdiffrast, Mitsuba 3) are powerful but disconnected from VTK’s rich data pipeline — filters, readers, meshing, and visualization infrastructure. A researcher using VTK for preprocessing and visualization today has to serialize their scene out of VTK, re-import it into a separate renderer, and lose the pipeline context. This proposal aims to close that gap.
## **Proposed architecture**
The module would consist of three layers, with PyTorch as an _optional_ dependency (similar to how `RenderingRayTracing` treats OSPRay):
- **C++ core:** `vtkSceneParameterMap` for declaring differentiable parameters, `vtkGradientBuffer` for storing `∂L/∂param`, and a new `vtkRenderPass` subclass implementing soft rasterization
- **CUDA backend:** GPU-accelerated gradient computation with OpenGL–CUDA interop to avoid device round-trips
- **Python bridge:** A `torch.autograd.Function` wrapper exposing the render as a differentiable node in a PyTorch compute graph
## **Dependency strategy**
PyTorch and CUDA would be strictly optional. A CPU fallback using finite differences would be provided so the module is functional without a GPU and without PyTorch installed.
##
**Questions for the community**
- Is `Rendering/DifferentiableRendering` the right home, or would a standalone module be more appropriate?
- Is there appetite for a PyTorch dependency (even optional) in the VTK tree, or should the Python bridge live in a separate package like `vtk-torch`?
- Are there existing gradient buffer or parameter tracking abstractions in VTK I should build on rather than introduce new ones?
- Has anything like this been attempted before?
---
### Author:  [@jaswantp](https://discourse.vtk.org/u/jaswantp)
#### Post date: [May 2, 2026, 2:21pm UTC](https://discourse.vtk.org/t/differentiable-rendering-module-with-pytorch-integration-for-vtk/16401/2 "2026-05-02T14:21:07Z")
Hello @Juan_C_Prieto
Yes. Are you interested in making volume rendering differentiable, or just surfaces for now?
> Existing differentiable renderers (PyTorch3D, nvdiffrast, Mitsuba 3) are powerful but disconnected from VTK’s rich data pipeline — filters, readers, meshing, and visualization infrastructure. A researcher using VTK for preprocessing and visualization today has to serialize their scene out of VTK, re-import it into a separate renderer, and lose the pipeline context. This proposal aims to close that gap.
I was reading “Transferring transfer functions (TTF): A guided approach to transfer function optimization in volume visualization” ([https://www.sciencedirect.com/science/article/pii/S0097849324002024](https://www.sciencedirect.com/science/article/pii/S0097849324002024)) a while back and it implements DVR (differentiable volume rendering) in a python class derived from `torch.nn.Module`. That could be avoided if VTK had DVR out of the box.
@sankhesh @berk.geveci
---
### Author:  [@mwestphal](https://discourse.vtk.org/u/mwestphal)
#### Post date: [May 4, 2026, 6:47am UTC](https://discourse.vtk.org/t/differentiable-rendering-module-with-pytorch-integration-for-vtk/16401/3 "2026-05-04T06:47:05Z")
Adding am (optional) pytorch dependency in VTK is not a good idea imo.
Create VTK / pytorch / whatever dependent project and maintain it. No need for integrating this in VTK.
---
### Author:  [@Juan\_C\_Prieto](https://discourse.vtk.org/u/Juan_C_Prieto)
#### Post date: [May 26, 2026, 3:31pm UTC](https://discourse.vtk.org/t/differentiable-rendering-module-with-pytorch-integration-for-vtk/16401/4 "2026-05-26T15:31:26Z")
I’m interested in compatibility of frameworks that perform differentiable rendering such as pytorch3d. We’ve used this library in our research to train 3D Deep Learning models. We’ve faced several issues and setbacks when trying to deploy our models in 3DSlicer for example.
However, recently I made substantial progress in facilitating the deployment of pytorch3D within 3DSlicer. I have custom repo that builds wheels compatible with 3DSlicer [here](https://github.com/ImageMindAnalytics/pytorch3d-slicer-wheels)