How to create PartitionedDataSetCollection in an MPI context ?

Support
1

Hello,

There is a thing I do not understand about vtkPartitionedDataSetCollection construction in an MPI context.

I have a mesh in a specific data structure distributed over x MPI ranks. The mesh data structure stores the polyhedrons per type (one array for hexahdra, one array for tetrahedra, etc.).
So I have multiple arrays (one per polyhedron type per rank). All ranks have the same number of arrays.

I have to output these arrays in something compatible with paraview. I was thinking of vtkPartitionedDataSetCollection.

I understand that I can for each rank build a vtkUnstructuredGrid per polyhedron type and add them to a vtkPartitionedDataSet (so one per rank).

But how can I manage to assemble these vtkPartitionnedDataSets into a vtkPartitionedDataSetCollection ?
Is there a way to pass the information about these vtkPartitionnedDataSets to rank 0 so it can build a vtkPartitionedDataSetCollection ?

2

Each vtkPartitionedDataSetCollection (PDC) across all ranks should have the same vtkDataAssembly, and the same number of vtkPartitionedDataSets (PD). The number of vtkDataSets (DS), in each PD, can differ across each rank. For that reason, the compositeIndex to PD A in rank 1 might differ from rank 2. but using selectors (a.k.a paths described in the data assembly), should return you the same thing. But remember that the paths need to be unique. Also, the paths described in the assembly should be on PD and not on a DS.

To gather information from all ranks to rank 0 and then distribute them, you would probably need to perform a gather operation.

Also, you might wanna look into:

  1. bool vtkIOSSReader::vtkInternals::UpdateEntityAndFieldSelections(vtkIOSSReader* self)
  2. bool vtkIOSSReader::vtkInternals::UpdateAssembly(vtkIOSSReader* self, int* tag)
  3. bool vtkIOSSReader::vtkInternals::BuildAssembly(
    Ioss::Region* region, vtkDataAssembly* assembly, int root, bool add_leaves)

Let me know if you need more clarifications.

3

Thank you for your reply.
I understand the “across” such as each rank can have its own PDC.
The only thing I have to share between the ranks is an instance of vtkDataAssembly ? Or each can have its own instance provided that they have the same information inside ?
I have other questions:

  • If each rank can have and write its own PDC how can I gather a “meta” PDC ?
  • More importantly, is this process for each timestep ?
4

Each rank can have its own instance with identical information.

A meta PDC? Could you elaborate please?

If your data assembly does not change over time, then you do it once and just copy it over time steps, if it does change, then in the vtkIOSSReader we have an option to scan all time step files to make sure that we have a global vtkDataAssembly that describes all time steps.