Distributed I/O with vtkXMLPUnstructuredGridWriter

Support
1

Hi Everyone,

We have our own solver that uses .vtu output via a vtkXMLUnstructuredGridWriter and synthesizes timeseries .pvds so that our domain scientists can use ParaView for post-processing. We’ve recently gone from pure OpenMP to a distributed MPI + OpenMP implementation, which distributes our unstructured grid data via a 1D distribution across ranks.

I’m attempting to move to a parallel output format to avoid gathering the grid at rank 0. I thought I could use the vtkXMLPUnstructuredGridWriter to do this via a combination of

    writer->SetNumberOfPieces(n_ranks);
    writer->SetStartPiece(rank);
    writer->SetEndPiece(rank);
    writer->Update();

but this doesn’t work quite right because, while the .vtu files are generated properly, the .pvtu file appears to be overwritten by each rank (or at least the .pvtu file is generated with only one Piece that seems to be from a random rank each time, so I assume that’s what is happening).

As far as I can tell, the documentation implies that only the first process should be writing this summary file, and even if each rank contended on the write I would have expected that they would all try and write all of the pieces and it would be idempotent.

Does this behavior require VTK_GROUP_ENABLE_MPI=On? Can I force the behavior that I expect? Is there something else obvious I am missing?

Thanks,
Luke

2

Replying to myself. I have been unable to figure out how to get this to work. I’ve implemented a workaround where I force the Summary to be written only by rank 0, and then open it and manually add the missing pieces using tinyxml2 (a dependency we already have in order to produce .pvd timeseries data).

This still feels like something VTK should be able to do natively. I assume that I have some sort of user-error. I tried to track down the implementation of what actually happens when I call Update() but the tangled web of inheritance, virtualization, and executor abstraction was too much for me.

3

Hey Luke,

I’ve stumbled upon the same problem. I could fix it using the descriptions of Gerstrong:

https://www.steinzone.de/wordpress/hacking-vtk-for-parallelisation-mpi-and-c/

and the implementation of the libmesh team:

https://github.com/libMesh/libmesh/blob/484c8652977b6504e1613633b4c7d87297f94957/src/mesh/vtk_io.C

Essentially, every thread has to use the vtkXMLPUnstructuredGrid interface. The vtu files are then written out automagically. Now, in order for your Code’s MPI and VTKs MPI to work together properly, you have to share the MPI communicator:

    auto VtkComm = vtkSmartPointer<vtkMPICommunicator>::New();
    vtkMPICommunicatorOpaqueComm VtkOpaqueComm(MPI_COMM_WORLD);
    VtkComm->InitializeExternal(&VtkOpaqueComm);
    auto VtkMpiController = vtkSmartPointer<vtkMPIController>::New();
    VtkMpiController->SetCommunicator(VtkComm);

Here, I assume that your code uses the MPI_COMM_WORLD communicator. Hope this helps.

Best,
Christian

4

Ah, the missing magic must be the MPI controller code, which I’m not using at all. My VTK was built without VTK_GROUP_ENABLE_MPI, so the mpi libs aren’t available, which was my concern.

My current workaround of manually editing the .pvtu is working fine at the moment, but if I can convince someone to rebuild VTK I’ll try working with the MPI controller.

Thank you.

5

My VTK was built without VTK_GROUP_ENABLE_MPI, so the mpi libs aren’t available, which was my concern.

In that case, you might have to use the vtkDummyController, although I haven’t tested it myself.

My current workaround of manually editing the .pvtu is working fine at the moment, but if I can convince someone to rebuild VTK I’ll try working with the MPI controller.

Never change a running system :grin: But honestly, the fix was well worth the hussle as now, we only need to call the XMLPUnstructuredGridWriter and not to the others which got rid of some Code.

Anyhow, have a great one.