Summary: We are considering adding orientation into vtkImageData to support rotations and inversions but not shears. We are looking for feedback on the idea and the two approaches listed below.
This document was created based on the Slicer wiki Labs page found here originally written by @lassoan with input from @jcfr and later updated based on comments from @ken-martin , @demarle , @berkgeveci , @lisa-avila and @jcfr. The intent of posting this is to get feedback on what the community feels about this idea and specific issues, ideas, etc. So please provide feedback to help direct this concept. As always remember that contributors time and effort is limited, so the goal is to get the most positive impact for the effort people put in. Consider scope and impact when suggesting directions (ha ha, directions – orientation) etc. Thanks!
In many fields we often have to work with images that are arbitrarily oriented in space. We’ll refer to images that can have arbitrary orientation as “oriented images”, and images that have voxel coordinate axes aligned with the object coordinate axes as “axis-aligned images”.
VTK’s lack of orientation in vtkImageData requires many workarounds in applications or conversion to a much larger data representation such as vtkStructuredGrid, which increases complexity, probability of errors, and decreases performance:
- Many file formats that we use (nrrd, mha, nifti, etc.) store the axis directions, but currently when we load these images, the direction information is lost.
- Most VTK filters could be easily extended to accept oriented images as input and/or generate oriented images as output, but currently they only work with axis-aligned images.
- When a filter takes multiple images as inputs then those cannot be used with oriented images. There is no workaround, other than resample the images or implement new filters. For example, vtkGridTransform or vtkBSplineTransform classes would need to work with oriented images (coefficients are stored in vtkImageData) because the DICOM standard requires transformation with arbitrarily oriented grids, but if the input is an oriented image as well, there is no common voxel coordinate system where all the input data could be transformed to.
- Some filters have additional inputs to be able to pass axis directions (such as Filters that are often used with oriented images (such as vtkImageReslice), but collecting the direction information separately and then injecting it into these filters correctly makes these filters difficult to use.
- While actors can make the volumes appear in arbitrary orientation, the orientation information has to be collected from the data reader and sent to the actors manually.
- Linear transforms cannot be efficiently applied on vtkImageData because the transformation result cannot be stored in a vtkImageData (when the transformation includes rotation).
Software applications deal with this in various ways – none of them are great solutions:
- 3D Slicer does not use vtkImageData origin and spacing fields and just stores the VoxelToObject transform (origin, spacing, direction) separately in a 4x4 matrix. This makes implementing VTK-based image processing pipelines quite complicated to implement.
- Paraview ignores this problem and assumes that all images are axis-aligned. This makes it challenging for users to use Paraview for their medical images that are often not axis-aligned.
- ITK, ITK-Snap, etc. use ITK-based classes for image storage, although they use VTK for visualization.
There are two main design options:
- Create a new image data class, such as vtkOrientedImageData and during the transition time use that to distinguish between oriented and axis-aligned data. Once the transition time is over (probably in a few years), the old vtkImageData class can be replaced by vtkOrientedImageData (an alias may be kept for backward compatibility). ITK used this approach, but maybe because ITK objects did not have vtkInformation-type generic description that can be passed through pipelines. The disadvantage of this approach that a new class has to be added, all filters and other classes have to be changed to use this new type, and once the transition is complete (after many years) the vtkOrientedImageData would need to be retired and changed back everywhere to vtkImageData.
- Keep using vtkImageData but add direction information to the class and vtkInformation object that already stores origin and spacing information. This approach would be less invasive, requiring less changes to existing code base, but probably it would require a special check in the pipeline for presence of image rotation for a couple of years, until VTK fully supports oriented image data.
To be investigated:
- How to manage vtkStructuredPoints (which is a specialization of vtkImageData) and vtkUniformGrid (which is an extension of vtkImageData)?
- If a new class is created, should it be child or parent of vtkImageData? All ImageData could be interpreted as a vtkOrientedImageData, so it seems to make sense to make vtkOrientedImageData the base class.
Implement oriented image class and filters can operate on oriented image data.
This phase is completed. We’ve found that oriented image data support greatly simplified data application code, but it was difficult to use filters that are not orientation-aware. It was not always trivial to update classes - had to spend time with completely understanding how the filter or transform worked (update how normals and derivatives, cutting plane positions and orientations are computed) - to add oriented image data support. However, the filters did not become slower or more complex by making them orientation-aware.
Related code is available here:
Simply implementing vtkOrientedImageData class did not help much because developers need to propagate direction information manually. It is important to make the pipeline accept vtkOrientedImageData.
Updated approach to implement mechanism that allows storing and passing direction data in vtkImageData. It would be nice to implement a mechanism to report error if a filter receives oriented image data as input but it can only work with axis-aligned image data.
Implementation is work in progress in this branch:
What is orientation specifically? Are we talking euler angles (3dof) or something more general that can include shears and inversions, lefthanded coords etc?
The current thinking is that orientation would be a 3x3 matrix limited to an orthonormal basis with inversions, while it would be more general to support shears there is concern that shears would complicate some filters and that they may not be that common compared to rotations and inversions. This is what ITK supports. Clearly this is an important decision so feel free to speak up.
What should be done for vtkStructuredPoints and vtkUniformGrid ?
Currently, vtkImageData is used as base class for vtkStructuredPoints and vtkUniformGrid. If we add orientation to vtkImageData, then these classes also get it. It would be probably generally useful, but it has to be checked what is the preference of heavy users of vtkStructuredPoints and vtkUniformGrid classes (do they want to change their base class and not even have the opportunity to orient the grids, or just disable it in those filters that cannot handle it yet).
Answer: TBD but I think we should have them handle orientation as well
What about vtkRectilinearGrid ?
It is very similar to vtkImageData, should we add orientation to it as well?
Yes. There are only a few filters that operate directly on rectilinear grids and many of the required changes will be similar to the changes made to vtkImageData.