OpenUSD Support

Development
1

I would like to understand the VTK community’s perspective on potential support for the OpenUSD standard. This thread was prompted by discussion at today’s VTK.js developers meeting.

Overview

Universal Scene Description is an open source project initially developed by Pixar Animation Studios. The project was released as open source in 2016. Earlier this year the Alliance for OpenUSD was announced by NVIDIA, Pixar, Adobe, Apple, Autodesk, and more. OpenUSD is used throughout NVIDIA’s Omniverse platform and has broad industry support for 3D application interoperability.

From the OpenUSD website:

USD is a high-performance extensible software platform for collaboratively constructing animated 3D scenes, designed to meet the needs of large-scale film and visual effects production.

USD enables robust interchange between digital content creation tools with its expanding set of schemas, covering domains like geometry, shading, lighting, and physics.

USD’s unique composition ability provides rich and varied ways to combine assets into larger assemblies, enables collaborative workflows so that many creators can work together with ease, and more.

OpenVDB is also mentioned for volumes, though less commonly.

Why It Matters

VTK interoperability with OpenUSD would enable scientists, developers, general community members, to more easily move data between various platforms for processing and visual representation.

Some possible application-driven examples:

  • Process spatial cloud data from multiple cameras in a ParaView scene, export as OpenUSD for advanced AI-driven rendering in NVIDIA Omniverse
  • Ingest a scene from NVIDIA Omniverse into ParaView as input data for a predefined scientific processing pipeline
  • Perform markups on medical research data in 3D Slicer, export as OpenUSD, animate for medical education with industry animation tools
  • Ingest a prosthetic scene from AutoCAD into 3D Slicer to compare with patient CT images

Past Support and Guiding Principles

In 2021 NVIDIA released the NVIDIA Omniverse ParaView Connector with USD connectivity. It doesn’t seem that connector is open source, or at least, I have not been able to track down the source repository.

My understanding is that historically VTK has avoided a scene graph representation, with the directive that scene graph implementation should be handled at the application level. For instance, 3D Slicer implements the MRML Scene. I would like to understand whether there are components that would make sense to be handled at the VTK library level for reuse, rather than requiring each application to figure out their own unique approach for supporting industry standards.

While initial discussion was in regards to VTK.js and related web applications, it would be good to first consolidate an approach from the VTK C++ level to guide that development.

For Discussion

I would like to better understand the VTK community’s vision on this topic and actionable next steps.

  • Is OpenUSD support applicable to VTK’s user base?
  • Is there existing support that I have overlooked, beyond the (seemingly closed-source) ParaView connector? Accompanying documentation?
  • Even without a scene graph, is there a reasonable way to provide in VTK a reader composing VTK objects from an OpenUSD or OpenVDB description?
  • Are there components of OpenUSD serialization that would make sense to handle at the VTK library level?
  • What components do not make sense to support in VTK and should be offloaded to individual applications such as 3D Slicer?

Thank you in advance.

cc @Forrest @Stephen_Aylward @jcfr

1 Like
2

First I would like to say that this is a very good initiative!

Is OpenUSD support applicable to VTK’s user base?

Most definitely, VTK user base overlap greatly with USD user base

Is there existing support that I have overlooked, beyond the (seemingly closed-source) ParaView connector? Accompanying documentation?

F3D has an ongoing implementation of a vtkUSDImporter. It is not full feature but is still a great starting point.

It is not merged yet because of issues with libusd OneAPI support that prevent F3D maintainers to ship with libusd.

Here is how it looks:

image
image1920×1280 171 KB

Even without a scene graph, is there a reasonable way to provide in VTK a reader composing VTK objects from an OpenUSD or OpenVDB description?

Well, you can look at the vtkUSDImporter (above) and the vtkOpenVDBImporter (in VTK) implementations

I dont know enough about the USD format to answer other questions.

1 Like
3

+1 for USD support in VTK at some level. We had some discussions a year or so ago about creating an OmniVerse connector for 3D Slicer but it seemed to require a little more effort than anyone had available. It would be great if VTK support worked something like glTF import/export.

3 Likes
4

Yes, USD support in VTK would be great. We discussed this a little bit 2 years ago.

Is OpenUSD support applicable to VTK’s user base?

Yes. It would be used similarly to glTF importer/exporter.

Are there components of OpenUSD serialization that would make sense to handle at the VTK library level?

VTK’s glTF exporter is very limited in exporting metadata but it is fine because we can add and modify metadata by a json reader/writer. If this can be achieved for OpenUSD as well then it should be enough if VTK provides mapping between VTK objects and OpenUSD scene object IDs.

What components do not make sense to support in VTK and should be offloaded to individual applications such as 3D Slicer?

It should be enough if VTK can read/write what is needed for rendering, as long as VTK provides mapping to OpenUSD scene objects. Then applications could read/write additional metadata.

2 Likes
5

F3D has an ongoing implementation of a vtkUSDImporter. It is not full feature but is still a great starting point.

This is great, thanks for sharing @mwestphal . Do you have thoughts on how difficult it may be to port the initial vtkUSDImporter to VTK proper?

Well, you can look at the vtkUSDImporter (above) and the vtkOpenVDBImporter (in VTK) implementations

Great, you’re right, it seems like the OpenVDB part is already addressed in VTK: https://gitlab.kitware.com/vtk/vtk/-/tree/master/IO/OpenVDB

6

Shouldn’t be too hard tbh, what would be hard would be to add libusd in the CI as they do not officially support OneAPI yet.

7

This is a good step!

How hard would it be to make a USD exporter? I can see folks wanting to use Hydra-based/custom in-house renderers(desktop/web) instead of VTK for the large-scale datasets that are typically targeted by OpenUSD.

1 Like
8

On a somewhat related note, UNC and others have begun discussing a “surgical” extension to DICOM, particularly in the context of surgical robotics. It would support describing scenes and actors (robots/actuators). I pointed them to ROS and ROS2, but perhaps OpenUSD is a better fit / source of inspiration.

If we consider OpenUSD and DICOM for surgery / surgical robotics together, this is starting to look like the basis for a very interesting Specific Aim (one of several) for the VTK Innovation grant renewal proposal (we will submit the renewal proposal in Nov). Perhaps this could also be done as an admin supplement to existing grants (e.g., for the low-cost surgery guidance system being built with Slicer and BWH)?

OpenUSD + DICOMSurgery = MedicalVerse :slight_smile:

1 Like
9

Very interesting idea Stephen - it would be interesting to look at how to have a standards-based back end and interoperability format for Slicer. People do use MRML (based on VTK and XML) and mrb files (basically a zip of all their scene data and MRML) for surgery plans a lot, so I think we know what would need to be included. Happy to be part of any discussions.

10