# Distributed I/O with vtkXMLPUnstructuredGridWriter
**URL:** https://discourse.vtk.org/t/distributed-i-o-with-vtkxmlpunstructuredgridwriter/7418
**Category:** Support
**Tags:** file-formats
**Created:** [December 16, 2021, 4:17pm UTC](https://discourse.vtk.org/t/distributed-i-o-with-vtkxmlpunstructuredgridwriter/7418 "2021-12-16T16:17:45Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Luke\_Dalessandro](https://discourse.vtk.org/u/Luke_Dalessandro)
#### Post date: [December 16, 2021, 4:17pm UTC](https://discourse.vtk.org/t/distributed-i-o-with-vtkxmlpunstructuredgridwriter/7418/1 "2021-12-16T16:17:45Z")
Hi Everyone,
We have our own solver that uses .vtu output via a `vtkXMLUnstructuredGridWriter` and synthesizes timeseries `.pvd`s so that our domain scientists can use ParaView for post-processing. We’ve recently gone from pure OpenMP to a distributed MPI + OpenMP implementation, which distributes our unstructured grid data via a 1D distribution across ranks.
I’m attempting to move to a parallel output format to avoid gathering the grid at rank 0. I thought I could use the `vtkXMLPUnstructuredGridWriter` to do this via a combination of
```auto
writer->SetNumberOfPieces(n_ranks);
writer->SetStartPiece(rank);
writer->SetEndPiece(rank);
writer->Update();
```
but this doesn’t work quite right because, while the `.vtu` files are generated properly, the `.pvtu` file appears to be overwritten by each rank (or at least the `.pvtu` file is generated with only one Piece that seems to be from a random rank each time, so I assume that’s what is happening).
As far as I can tell, the documentation implies that only the first process should be writing this summary file, and even if each rank contended on the write I would have expected that they would all try and write all of the pieces and it would be idempotent.
Does this behavior require `VTK_GROUP_ENABLE_MPI=On`? Can I force the behavior that I expect? Is there something else obvious I am missing?
Thanks,
Luke
---
### Author:  [@Luke\_Dalessandro](https://discourse.vtk.org/u/Luke_Dalessandro)
#### Post date: [December 16, 2021, 9:17pm UTC](https://discourse.vtk.org/t/distributed-i-o-with-vtkxmlpunstructuredgridwriter/7418/2 "2021-12-16T21:17:49Z")
Replying to myself. I have been unable to figure out how to get this to work. I’ve implemented a workaround where I force the Summary to be written _only_ by rank 0, and then open it and manually add the missing pieces using tinyxml2 (a dependency we already have in order to produce `.pvd` timeseries data).
This still feels like something VTK should be able to do natively. I assume that I have some sort of user-error. I tried to track down the implementation of what actually happens when I call `Update()` but the tangled web of inheritance, virtualization, and executor abstraction was too much for me.
---
### Author:  [@cheinigk](https://discourse.vtk.org/u/cheinigk)
#### Post date: [January 5, 2022, 10:01am UTC](https://discourse.vtk.org/t/distributed-i-o-with-vtkxmlpunstructuredgridwriter/7418/3 "2022-01-05T10:01:02Z")
Hey Luke,
I’ve stumbled upon the same problem. I could fix it using the descriptions of Gerstrong:
[https://www.steinzone.de/wordpress/hacking-vtk-for-parallelisation-mpi-and-c/](https://www.steinzone.de/wordpress/hacking-vtk-for-parallelisation-mpi-and-c/)
and the implementation of the libmesh team:
[https://github.com/libMesh/libmesh/blob/484c8652977b6504e1613633b4c7d87297f94957/src/mesh/vtk\_io.C](https://github.com/libMesh/libmesh/blob/484c8652977b6504e1613633b4c7d87297f94957/src/mesh/vtk_io.C)
Essentially, every thread has to use the `vtkXMLPUnstructuredGrid` interface. The `vtu` files are then written out automagically. Now, in order for your Code’s MPI and VTKs MPI to work together properly, you have to share the MPI communicator:
```cpp
auto VtkComm = vtkSmartPointer::New();
vtkMPICommunicatorOpaqueComm VtkOpaqueComm(MPI_COMM_WORLD);
VtkComm->InitializeExternal(&VtkOpaqueComm);
auto VtkMpiController = vtkSmartPointer::New();
VtkMpiController->SetCommunicator(VtkComm);
```
Here, I assume that your code uses the MPI\_COMM\_WORLD communicator. Hope this helps.
Best,
Christian
---
### Author:  [@Luke\_Dalessandro](https://discourse.vtk.org/u/Luke_Dalessandro)
#### Post date: [January 5, 2022, 7:31pm UTC](https://discourse.vtk.org/t/distributed-i-o-with-vtkxmlpunstructuredgridwriter/7418/4 "2022-01-05T19:31:14Z")
Ah, the missing magic must be the MPI controller code, which I’m not using at all. My VTK was built without VTK\_GROUP\_ENABLE\_MPI, so the mpi libs aren’t available, which was my concern.
My current workaround of manually editing the `.pvtu` is working fine at the moment, but if I can convince someone to rebuild VTK I’ll try working with the MPI controller.
Thank you.
---
### Author:  [@cheinigk](https://discourse.vtk.org/u/cheinigk)
#### Post date: [January 10, 2022, 7:33pm UTC](https://discourse.vtk.org/t/distributed-i-o-with-vtkxmlpunstructuredgridwriter/7418/5 "2022-01-10T19:33:03Z")
> My VTK was built without VTK\_GROUP\_ENABLE\_MPI, so the mpi libs aren’t available, which was my concern.
In that case, you might have to use the `vtkDummyController`, although I haven’t tested it myself.
> My current workaround of manually editing the `.pvtu` is working fine at the moment, but if I can convince someone to rebuild VTK I’ll try working with the MPI controller.
Never change a running system 😁 But honestly, the fix was well worth the hussle as now, we only need to call the `XMLPUnstructuredGridWriter` and not to the others which got rid of some Code.
Anyhow, have a great one.