The aim of this post is to summarize the current status of the VTKHDF File Format handles, indicate where supporting resources can be found, and provide a roadmap of what is planned for 2024 and beyond.
Additionally, a master issue has been opened here to track any improvements and clarifications of the VTKHDF specification itself.
This post will be updated as we augment the format specification and improve the implementation.
In short, VTKHDF is a file format to store VTK data types in a HDF5 file. The up-to-date specification about this file format is here¹.
What is currently supported with VTKHDF
In the VTKHDF file format.
This format¹ specifies how to store the following VTK data types:
- vtkPolyData
- vtkUnstructuredGrid*
- vtkImageData
- vtkOverlappingAMR
- vtkMultiBlockDataSet
- vtkPartitionedDataSetCollection
- vtkPartitionedDataSet
*: not all cell types are supported
The VTKHDF format also specifies how to store distributed and temporal data for most of these VTK data types.
In VTK
VTK has a dedicated module⁶ VTK::IOHDF to read and write VTKHDF file format, it contains a reader named vtkHDFReader and a writer, vtkHDFWriter.
| types | serial | distributed | Temporal (in one file) |
|---|---|---|---|
| vtkPolyData |  |
|
|
| vtkUnstructuredGrid | |
|
|
| vtkImageData | |
|
|
| vtkPartitionedDataSet | |
|
|
| vtkPartitionedDataSetCollection³ | |
|
 |
| vtkMultiBlockDataSet³ | |
|
|
| vtkOverlappingAMR² | |
|
|
Some types are specified and can be read, but some of them can not be written using vtkHDFWriter just yet. The types not listed here need to be written externally, using any library which supports writing HDF5 files.
| types | serial | Temporal (in one file) |
|---|---|---|
| vtkPolyData | |
|
| vtkUnstructuredGrid | |
|
| vtkPartitionedDataSetCollection³ | |
|
| vtkMultiBlockDataSet³ | |
|
Note that a part of the vtkHDFReader is available in VTK 9.3.0 but most of these features are available on VTK master only.
In ParaView
This VTK::IOHDF module is fully exposed in ParaView and an extractor has been added for the vtkHDFWriter, which is essential for in situ use cases with Catalyst.
Roadmap
Few other features are already under development, discussion or planned. Some of them will be data type dependent like in this table:
| PolyData | UnstructuredGrid | ImageData | OverlappingAMR | PartitionedDataSet | PartitionedDataSetCollection | MultiBlockDataSet | |
|---|---|---|---|---|---|---|---|
| vtkHDFReader: Distributed support |  |
|
|
|
|
 |
|
| vtkHDFReader: Static mesh support** |  |
|
 |
|
|
|
|
| vtkHDFWriter: non-distributed support | |
|
|
|
|
|
|
| vtkHDFWriter:Distributed support | |
|
|
|
|
|
|
| vtkHDFWriter: iterative temporal writing | |
|
|
|
|
|
|
| vtkHDFWriter: Static mesh support | |
|
|
|
|
|
|
Legends:
Future work
Looking for funding
To be implemented in 2024
Currently being implemented
Already done
**: The VTK pipeline can reuse the mesh from another timestep if the correct key information is passed through and a special cache system is used with vtkDataObjectMeshCache. For more details about this feature, see the following blog post about this topic⁷.
A few other features, not related to any specific data type, will be implemented this year:
- Add a
Miscgroup underVTKHDFgroup to contain user data ignored by the VTK implementation of the file format. - Support writing VTKHDF file in one file per block.
There are other features which may be implemented this year:
- Iterative temporal writing in vtkHDFWriter which means to append in the same file all requested timestep.
- vtkPolyhedron support for VTKHDF UnstructuredGrid⁵.
Last, we aim to create a benchmark this year to evaluate performance of the VTKHDF file format in an HPC environment.
How you can help
If you are interested in the VTKHDF file format or the current VTK implementation, please don’t hesitate to try it out. Feedback is always welcome!
Finally, if some features that currently lack funding in the roadmap interest you, please reach us via the Kitware contact page.
Resources
Official up-to-date documentation¹ : VTK File Formats - VTK documentation
Related discussions:
- About vtkOverlappingAMR²: link
- About composite support³ : link
- About OverlappingAMR with temporal data⁴ : link
- About Polyhedron cell support in VTKHDF⁵: link
- About temporal field data: link
- About splitted HDF5 files: link
HDF Module in the vtk repository⁶ : link
Kitware blog post about the StaticMeshPlugin⁷: link
Repository containing python scripts to generate VTKHDF file or to convert VTKXML file to VTKHDF file: https://gitlab.kitware.com/keu-public/vtkhdf/vtkhdf-scripts