Integrating 3D Object with Medical Images in VTK.js

Web vtk.js
1

I am looking to replicate a workflow I currently use in 3D Slicer on a web-based platform using VTK.js or Cornerstone3D.js. Specifically, I need to view a 3D object file (e.g., OBJ) in conjunction with a medical image file (e.g., NIFTI) across three orthogonal views and see the intersections.

Current Workflow in 3D Slicer:

  1. Load Medical Image:
  • Add a NIFTI image as a volume.
  • Right-click on the image and select “Show in 3D views as volume rendering.”
  1. Load 3D Model:
  • Add an OBJ file as a model.
  • In the Data tab, right-click on the model and enable “2D Visibility.”
  1. Transform Model:
  • Create a new transform for the model object.
  • Right-click on the transform and go to “Edit Properties” to adjust the model’s position and rotation using sliders.
  1. View Intersections:
  • Move the model within the volume and observe intersections in Axial, Sagittal, and Coronal views.
  • Use the Model tab to switch the model’s visibility between intersection and surface modes.

Requirement:

I need to achieve similar functionality on the web with VTK.js or Cornerstone3D.js. Specifically:

  • Load a medical image file (e.g., NIFTI).
  • Load a 3D object file (e.g., OBJ).
  • Move the OBJ file within the NIFTI volume.
  • Navigate through the volume in Axial, Sagittal, and Coronal views to observe the position of the 3D model.
  • Display intersections of the 3D model with the medical image volume in the orthogonal views.

Questions:

  1. Which specific modules and components of VTK.js are suitable for this task?
  2. Are there any existing examples or tutorials that demonstrate similar functionalities?
  3. How can I implement interactive transformations (position and rotation) of the 3D model within the volume?
  4. What is the best approach to enable and visualize intersections of the 3D model in the orthogonal views?

Any guidance or examples to get started with this would be greatly appreciated.

Thank you!

2

Kitware’s VolView can do many of these. You can also use 3D Slicer in the web browser (via docker+novnc).

3

Thank you for the suggestion.

I need a custom solution with API integration, and VolView doesn’t support 3D model intersections with a volume. I’ve implemented most parts, but vtkIntersectionPolyDataFilter isn’t in VTK.js, which 3D Slicer uses for intersections.

If you know any alternatives to visualize intersections in VTK.js, please let me know. Thanks again!

4

Can you use VTK proper? As far as I know it is usable in JavaScript now (via webassembly).

5

Thanks, Andras. It seems that I should do that. I will still try to see which functionalities I need are possible with vtk.js, and if it doesn’t meet my requirements, I will read more about vtk.wasm.

6

You can do polydata intersections in VTK.js (hence VolView)

1 Like
7

Thanks for the information @finetjul, it seems that vtk.js contains what you need for polydata slice intersection visualization.

Note that if you work with closed surfaces then your users will expect to see the regions inside the contours filled, so displaying the raw intersectWithOBBTree result is not sufficient, but you also need contour triangulation.

1 Like
8

Thank you, Julien (@finetjul) and Andras (@lassoan), for the helpful information. I played with OBBTree and was able to get the intersection line. I’m also working with WebAssembly and found some interesting things. Thanks again!