vtkhdf reading error for unstructured transient dataset

Development
1

I am developing vtkhdf export module for FE solver. I have a problem in reading the data for the last step (in Paraview 5.12.0-RC2). I guess that Paraview uses vtkHDFReader, so I ask for help here. The dump of sample hdf file demonstrating the problem:

HDF5 "se.out.m0.hdf" {
GROUP "/" {
   GROUP "VTKHDF" {
      ATTRIBUTE "Type" {
         DATATYPE  H5T_STRING {
            STRSIZE 16;
            STRPAD H5T_STR_NULLTERM;
            CSET H5T_CSET_ASCII;
            CTYPE H5T_C_S1;
         }
         DATASPACE  SCALAR
         DATA {
         (0): "UnstructuredGrid"
         }
      }
      ATTRIBUTE "Version" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 2 ) / ( 2 ) }
         DATA {
         (0): 1, 0
         }
      }
      GROUP "CellData" {
      }
      DATASET "Connectivity" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 24 ) / ( H5S_UNLIMITED ) }
         DATA {
         (0): 0, 2, 3, 1, 2, 4, 5, 3, 0, 2, 3, 1, 2, 4, 5, 3, 0, 2, 3, 1, 2,
         (21): 4, 5, 3
         }
      }
      DATASET "NumberOfCells" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 3 ) / ( H5S_UNLIMITED ) }
         DATA {
         (0): 2, 2, 2
         }
      }
      DATASET "NumberOfConnectivityIds" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 3 ) / ( H5S_UNLIMITED ) }
         DATA {
         (0): 8, 8, 8
         }
      }
      DATASET "NumberOfPoints" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 3 ) / ( H5S_UNLIMITED ) }
         DATA {
         (0): 6, 6, 6
         }
      }
      DATASET "Offsets" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 9 ) / ( H5S_UNLIMITED ) }
         DATA {
         (0): 0, 4, 8, 0, 4, 8, 0, 4, 8
         }
      }
      GROUP "PointData" {
         DATASET "DisplacementVector" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 18, 3 ) / ( H5S_UNLIMITED, 3 ) }
            DATA {
            (0,0): 0, 0, 0,
            (1,0): 0, 0, 0,
            (2,0): 0.012391, 0.00160247, 0,
            (3,0): 0.012391, -0.00160247, 0,
            (4,0): 0.025, 0.00109012, 0,
            (5,0): 0.025, -0.00109012, 0,
            (6,0): 0, 0, 0,
            (7,0): 0, 0, 0,
            (8,0): 0.024782, 0.00320494, 0,
            (9,0): 0.024782, -0.00320494, 0,
            (10,0): 0.05, 0.00218023, 0,
            (11,0): 0.05, -0.00218023, 0,
            (12,0): 0, 0, 0,
            (13,0): 0, 0, 0,
            (14,0): 0.037173, 0.00480741, 0,
            (15,0): 0.037173, -0.00480741, 0,
            (16,0): 0.075, 0.00327035, 0,
            (17,0): 0.075, -0.00327035, 0
            }
         }
      }
      DATASET "Points" {
         DATATYPE  H5T_IEEE_F64LE
         DATASPACE  SIMPLE { ( 18, 3 ) / ( H5S_UNLIMITED, 3 ) }
         DATA {
         (0,0): 0, 0, 0,
         (1,0): 0, 1, 0,
         (2,0): 1, 0, 0,
         (3,0): 1, 1, 0,
         (4,0): 2, 0, 0,
         (5,0): 2, 1, 0,
         (6,0): 0, 0, 0,
         (7,0): 0, 1, 0,
         (8,0): 1, 0, 0,
         (9,0): 1, 1, 0,
         (10,0): 2, 0, 0,
         (11,0): 2, 1, 0,
         (12,0): 0, 0, 0,
         (13,0): 0, 1, 0,
         (14,0): 1, 0, 0,
         (15,0): 1, 1, 0,
         (16,0): 2, 0, 0,
         (17,0): 2, 1, 0
         }
      }
      GROUP "Steps" {
         ATTRIBUTE "NSteps" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SCALAR
            DATA {
            (0): 3
            }
         }
         DATASET "CellOffsets" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 3, 1 ) / ( H5S_UNLIMITED, 1 ) }
            DATA {
            (0,0): 0,
            (1,0): 3,
            (2,0): 6
            }
         }
         DATASET "ConnectivityIdOffsets" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 3, 1 ) / ( H5S_UNLIMITED, 1 ) }
            DATA {
            (0,0): 0,
            (1,0): 8,
            (2,0): 16
            }
         }
         DATASET "NumberOfParts" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 3 ) / ( H5S_UNLIMITED ) }
            DATA {
            (0): 1, 1, 1
            }
         }
         DATASET "PartOffsets" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 3 ) / ( H5S_UNLIMITED ) }
            DATA {
            (0): 0, 0, 0
            }
         }
         DATASET "PointOffsets" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 3 ) / ( H5S_UNLIMITED ) }
            DATA {
            (0): 0, 6, 12
            }
         }
         DATASET "Values" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 3 ) / ( H5S_UNLIMITED ) }
            DATA {
            (0): 1, 2, 3
            }
         }
      }
      DATASET "Types" {
         DATATYPE  H5T_STD_U8LE
         DATASPACE  SIMPLE { ( 6 ) / ( H5S_UNLIMITED ) }
         DATA {
         (0): 9, 9, 9, 9, 9, 9
         }
      }
   }
}
}

I got following error:

ERROR: In vtkHDFReaderImplementation.cxx, line 1025
vtkHDFReader (00000169353D4DA0): Error H5Dread start: 6, 9801279100561222763, 1552024982320 count: 2, 9801295593230199896, 0

ERROR: In vtkHDFReader.cxx, line 160
(nullptr): Cannot read the Types_0 array from file

ERROR: In vtkHDFReader.cxx, line 856
vtkHDFReader (00000169353D4DA0): Cannot read the Types array

Clearly, the problem is with types dataset, but in my opinion it contains data for 2 cell types for 3 time steps = 6 entries.

Any help very much appreciated. Borek

2

@lgivord @danlipsa

3

hello @Borek_Patzak,

is it possible to share us the data here? It would be easier for me and see if the issue is linked to your data or the reader.

Thanks

4

nevermind I see that your already shared the data in the related vtk issue : https://gitlab.kitware.com/vtk/vtk/-/issues/19249

I will take a look when I have some time

5

In this post I prepared simplified version with just 2 elements and 3 time steps. The example submitted to vtk issues was slightly complex. Please use following link to access hdf file: se.out.m0.hdf - Google Drive

Thank you

6

Hello @Borek_Patzak,

Sorry for the late anwser and thanks for this simpler example, in fact you need to update the version used. In this file the version was the 1.0, you need at least to use the 2.0 that’s why you have an overflow:

vtkHDFReader (00000169353D4DA0): Error H5Dread start: 6, 9801279100561222763, 1552024982320 count: 2, 9801295593230199896, 0

now we have something like that:

vtkHDFReader (0x559590378920): Error H5Dread start: 6, 0, 32 count: 2, 0, 32

Next, the root of your issue is the usage of CellOffset, ConnectivityIdOffsets.

In your file, only the ConnectivityIdOffsets needs to be updated like what you done.
However, you shouldn’t update the CellOffset and let each row to 0 as your cells don’t varying on time.

Here the corrected version of your file:
se.out.m0.hdf (107.7 KB)

And here is what I have in ParaView:
result

2 Likes
7

Dear Lucas,
thank you, I can confirm this works and the provided corrected hdf file was really helpful.

I have one follow up question: suppose that my domain (cells and points) is evolving with time. Actually the example I provided was prepared in this direction: Connectivity, NumberOfCells, NumberOfConnectivityIds, NumberOfPoints, Offsets, PointData, Types and Points attributes were appended by relevant data for each step (although the data were the same in all steps in this example). Also the Steps Group in my opinion contained data for each step.
The question is how to adapt the example to take evolving domain into account?

Thank you,
Borek

8

Dear @Borek_Patzak,

I’m glad to see that the corrected file helps you.

The question is how to adapt the example to take evolving domain into account?

As you already append at each step all of these data, for me your example is already ready to support evolving domain.

Regarding the Steps group, it contains several array offset (such as PointsOffset) to be sure to read at a specific step the correct data but it seems to me that you already update it correctly.

I believe you can try it with evolving domain. Let me know if it works or not.

Thanks

9

Dear Lucas,
thank you. I am still confused by the documentation and missing an example illustrating the data structure for transient unstructured grid with changing topology.
I prepared the simple example of transient hdf file with two steps, in the first one there are 2 elements, in the second one there is additional element added. Basically, I obtain reading error when switching to second time step (Offsets array). I believe I have properly appended all the arrays for both steps and set up the Steps group correctly, but obviously, there is an error that I can’t see.
You can find the dump below and you can access hdf file using this link: se2.out.m0.hdf - Google Drive

HDF5 "se2.out.m0.hdf" {
GROUP "/" {
   GROUP "VTKHDF" {
      ATTRIBUTE "Type" {
         DATATYPE  H5T_STRING {
            STRSIZE 16;
            STRPAD H5T_STR_NULLTERM;
            CSET H5T_CSET_ASCII;
            CTYPE H5T_C_S1;
         }
         DATASPACE  SCALAR
         DATA {
         (0): "UnstructuredGrid"
         }
      }
      ATTRIBUTE "Version" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 2 ) / ( 2 ) }
         DATA {
         (0): 2, 2
         }
      }
      GROUP "CellData" {
      }
      DATASET "Connectivity" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 20 ) / ( H5S_UNLIMITED ) }
         DATA {
         (0): 0, 2, 3, 1, 2, 4, 5, 3, 0, 2, 3, 1, 2, 4, 5, 3, 4, 6, 7, 5
         }
      }
      DATASET "NumberOfCells" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 2 ) / ( H5S_UNLIMITED ) }
         DATA {
         (0): 2, 3
         }
      }
      DATASET "NumberOfConnectivityIds" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 2 ) / ( H5S_UNLIMITED ) }
         DATA {
         (0): 8, 12
         }
      }
      DATASET "NumberOfPoints" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 2 ) / ( H5S_UNLIMITED ) }
         DATA {
         (0): 6, 8
         }
      }
      DATASET "Offsets" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SIMPLE { ( 7 ) / ( H5S_UNLIMITED ) }
         DATA {
         (0): 0, 4, 8, 0, 4, 8, 12
         }
      }
      GROUP "PointData" {
         DATASET "DisplacementVector" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 14, 3 ) / ( H5S_UNLIMITED, 3 ) }
            DATA {
            (0,0): 0, 0, 0,
            (1,0): 0, 0, 0,
            (2,0): 0.012328, 0.00199971, 0,
            (3,0): 0.012328, -0.00199971, 0,
            (4,0): 0.025, 0.00137615, 0,
            (5,0): 0.025, -0.00137615, 0,
            (6,0): 0, 0, 0,
            (7,0): 0, 0, 0,
            (8,0): 0.0247358, 0.00415912, 0,
            (9,0): 0.0247358, -0.00415912, 0,
            (10,0): 0.05, 0.00211351, 0,
            (11,0): 0.05, -0.00211351, 0,
            (12,0): 0.0250967, -0.000350552, 0,
            (13,0): 0.0250967, 0.000350552, 0
            }
         }
      }
      DATASET "Points" {
         DATATYPE  H5T_IEEE_F64LE
         DATASPACE  SIMPLE { ( 14, 3 ) / ( H5S_UNLIMITED, 3 ) }
         DATA {
         (0,0): 0, 0, 0,
         (1,0): 0, 1, 0,
         (2,0): 1, 0, 0,
         (3,0): 1, 1, 0,
         (4,0): 2, 0, 0,
         (5,0): 2, 1, 0,
         (6,0): 0, 0, 0,
         (7,0): 0, 1, 0,
         (8,0): 1, 0, 0,
         (9,0): 1, 1, 0,
         (10,0): 2, 0, 0,
         (11,0): 2, 1, 0,
         (12,0): 3, 0, 0,
         (13,0): 3, 1, 0
         }
      }
      GROUP "Steps" {
         ATTRIBUTE "NSteps" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SCALAR
            DATA {
            (0): 2
            }
         }
         DATASET "CellOffsets" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 2, 1 ) / ( H5S_UNLIMITED, 1 ) }
            DATA {
            (0,0): 0,
            (1,0): 2
            }
         }
         DATASET "ConnectivityIdOffsets" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 2, 1 ) / ( H5S_UNLIMITED, 1 ) }
            DATA {
            (0,0): 0,
            (1,0): 8
            }
         }
         DATASET "NumberOfParts" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 2 ) / ( H5S_UNLIMITED ) }
            DATA {
            (0): 1, 1
            }
         }
         DATASET "PartOffsets" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 2 ) / ( H5S_UNLIMITED ) }
            DATA {
            (0): 0, 0
            }
         }
         DATASET "PointOffsets" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 2 ) / ( H5S_UNLIMITED ) }
            DATA {
            (0): 0, 6
            }
         }
         DATASET "Values" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 2 ) / ( H5S_UNLIMITED ) }
            DATA {
            (0): 1, 2
            }
         }
      }
      DATASET "Types" {
         DATATYPE  H5T_STD_U8LE
         DATASPACE  SIMPLE { ( 5 ) / ( H5S_UNLIMITED ) }
         DATA {
         (0): 9, 9, 9, 9, 9
         }
      }
   }
}
}

Any help very much appreciated.
Borek

10

Dear @Borek_Patzak,

on my side I didn’t have an error when I load your data at step2, which version of vtk do you use? I try it in ParaView Master.

Regarding your example, everything is fine unless the Part and NumberOfParts which are not handled correctly.

To be honest the documentation regarding Part isn’t clear for me too, I will see how I can improve the documentation regarding your use case.

In the meantime, could you please try this corrected version of your data?
se2.out.m0.hdf (107.7 KB)

Here is what I have in paraview:

result

is it correct?

Best

11

Dear Lucas,
thank you, I can confirm that by fixing PartOffsets the problem is solved. Any update of the documentation would be more than welcome.

With regards,
Borek